Measuring Your Impact: Using Evaluation to Demonstrate Value

1

• Measuring Your Impact Using Evaluation to
  Demonstrate Value
• Betsy Kelly
• Assessment Evaluation Liaison, NN/LM MCR
• and
• Maryanne Blake
• Evaluation Communication Coordinator, NN/LM PNR

2
Instructors

• Betsy Kelly
• kellyb_at_msnotes.wustl.edu
• -314-362-2783
• Maryanne Blake
• blakema_at_u.washington.edu
• -206-221-3405

3
Why Demonstrate the Librarys Value?

• To show the impact of the library on the larger
  organizations mission and goals

• To show accountability for your resources the
  librarys contribution to the bottom line

• To demonstrate that the library provides
  value-added services to solve staff problems

• To use as an advocacy and marketing tool

4
Workshop Objectives

• Workshop participants
• Understand librarys value in terms of the
  mission of the larger organization
• Describe tools used to assess the library, its
  users and stakeholders
• Identify elements of a logic model

5
Workshop Objectives

• Workshop participants will
• Identify tools and methods used for data
  collection and analysis
• Design an evaluation plan for a service in a
  library
• Report evaluation results

6
Yesterdays Hospital Librarian
That was one of the delightful things about her
job. It brought her in touch with so many people,
allor almost allof whom made her not only feel
welcome but that she was doing something really
worthwhile. She gave a little sigh of pleasure
as she unlocked the library door. She loved
working here and she loved the work
itself. From Jan Marlowe, Hospital Librarian.
Margaret Malcolm. Toronto New York Harlequin,
1976, (c)1960. Harlequin's collection ed.
7
Todays Hospital Librarians

• Must be
• Proactive
• Political
• Practical
• Businesslike
• The library is a cost center in the hospital
• Must reinvent hospital librarianship

8
Reinventing Librarianship

• T. Scott Pluchak, JMLA July 2004 Editorial
• Align the librarys priorities and services with
  the vision, mission and values of the larger
  organization to show its value to the hospital
• Remember, the library is a tool not an end in
  itself
• It will not be supported because it has intrinsic
  value

9
Get Out Of The Library

• Talk to people
• Talk about problems they are trying to solve
• Show how can the library help them

10
Solving Problems Value-Added With Thanks to
Will Welton

• Medical Staff Support
• Evidence-based Clinical Decision-making
• Marketing
• Research
• Patient/Family Education and Support
• Community Support
• Legal/Risk Management
• Human Resources Support
• Staff Education and Development
• Planning Program Development
• Evidence-based Management Support (Competitive
  Intelligence)

11
The Bottom LineWill Welton, Again

• You are either generating revenue, or supporting
  those who doOR
• You are helping to control operating expenses or
  supporting those who doOR
• You are creating expenses, with no value-added,
  that must be controlled or eliminated to reduce
  overhead

12
So, ask yourself. What am I doing to support the
bottom line?
13
A Plan To Show Your Value Like Building A House
14
Before You Dig

• The Vision
• The Environment
• The Blueprint
• The Tools
• And then
• Build It
• The Housewarming

15
The Vision
16
The Vision

• Know what you want to build and why
• Align library mission and goals with organization
  mission and goals
• Librarys value librarys contribution to
  achieving organization mission and goals

Vision
17

• Mission Concept
• Clinical care
• Education
• Management of operations
• Service

• Organizational Goal
• Provide excellent clinical care
• Provide services, resources needed for teaching
  and learning
• Reduce corporate risk
• Increase profitability
• Improve the lives of patients and their families

Vision
Abels EG, Cogdill KW, Zach L. Identifying and
communicating the contributions of library and
information services in hospitals and academic
health sciences centers J Med Libr Assoc. 2004
Jan92(1)46-55.
18
The Environment
19
The Environment

• The Organization
• Your Library
• Clients/Users
• Stakeholders
• The Community

Environment
20
A Look at the Environment

• Understand needs, desires and problems in
  context
• Validate assumptions about your contributions and
  services
• Provide a baseline for future evaluation
• Helps you develop the blueprint to plan and
  evaluate your contributions and services

Environment
21
SWOT Analysis

• Strengths
• Internal
• Positive statements about your library

• Weaknesses
• Internal
• Statements about what is lacking in your library

• Opportunities
• External
• What do clients want you to do that you are not
  doing?

• Threats
• External
• Factors that can adversely impact your librarys
  goals

Environment
22
Environment
23
Assessing the Environment

• User/stakeholder input
• Surveys
• Focus groups
• Interviews
• Unsolicited Feedback
• Observation
• Assessment by walking around
• Library statistics and records

Environment
24
I Cant Believe I Have to Evaluatethe Whole
Thing!

• Use smallball evaluation
• What you decide to evaluate depends on
• What you need or want to know
• What your users feel is important
• What certain stakeholders want to have evaluated

Environment
25
What You Need Is A Blueprint!
26
The Blueprint

• What you will build comes from your analysis
• What do you want/need in the end?
• Another bedroom?
• A house?
• A bigger house?
• What do you already have?
• An apartment?
• A two bedroom house?

Blueprint
27
The Blueprint

• What do you have to create to get what you
  want/need?
• Another bedroom?
• A whole new house?
• What do you have to do and what resources will
  you need to get what you want/need?

Blueprint
28
The Blueprint Part I The Logic Model

• Framework for program and evaluation
• Fluid
• Changes over time
• Makes evaluation easier

Blueprint
29
Logic Models May Change Over Time
Blueprint
30
Plan BackwardImplement Forward

• Goals
• Outcomes
• Activities
• Resources

Blueprint
31
Goals

• Statement(s) of Purpose why we do what we do
  and for whom
• Example
• Make effective, efficient use of organization
  resources
• Make information resources available to
  (community) members

32
Outcomes

• Outcomes should be
• Goal-related
• Specific
• Measurable
• Action oriented
• Realistic
• Timed

Blueprint
33
Pieces of the Logic Model

• Inputs or Resources
• What will you need to carry out your activities?
• Activities
• What will you do?
• Outputs
• How many of what will your activities produce?
• Outcomes
• So what the difference your program makes, the
  benefits that accrue because of your program

Blueprint
34
Sample Logic Model
Goal Cost effectively acquire and manage
information resources
35
Logic Model Group Activity

• Choose a service to evaluate
• Articulate the goal of the service
• Identify outcome(s)
• List
• activity
• resources
• outputs

Blueprint
36
The Blueprint Part IIThe Evaluation Plan

• Based on the logic model
• Consider the purpose
• Who is your audience?
• Your users
• Administrators
• Fund raisers
• How will the information be used?
• Financial savings or justification
• Intangible or non-monetary value of program
  benefits to community

Blueprint
37
The Evaluation Plan, contd.

• Questions
• What do you want to know about your program?
• Indicators
• How will you know you have achieved the outcomes
• Data
• Sources
• Methods
• Frequency
• Resources
• Expertise or tools needed to collect and analyze
  data

Blueprint
38
What Do You Want to Measure?

• Savings statistics, surveys, budgets
• Attitudes written instruments
• Awareness proxy measures
• Behavior self-report, observation, proxy
  measures
• Knowledge written or oral tests
• Skills observations, skill tests

Blueprint
39
Methods Tools for Collecting Data

• What to measure and how
• What indicators and outputs need to be measured
• What type of data needs to be gathered
• When and from whom will it be gathered
• Focus on the users perspective

Blueprint
40
Resources for Doing the Evaluation

• Existing records and statistics
• Observation
• Obtain user and stakeholder input focus groups,
  interviews with key informants
• Surveys
• Formal
• Informal
• The literature
• Specialists or consultants

Blueprint
41
Evaluation Plan
Outcome Reduce hospital costs by buying shared
resources.
Blueprint
42
Evaluation Plan II
Outcome Reduce hospital costs by buying shared
resources.
Blueprint
43
Design Your Own Evaluation Plan
Blueprint
44
The Tools To Build It
45
Make Sense of the Data

• Data analysis should be thought about before
  collecting data
• Test your collection and analysis tools
• Be sure to leave enough time
• If needed, limit the evaluation goals and reduce
  the amount of data collected (smallball)

Build It
46
Hot Analysis Tools

• Benchmarking
• How you compare to others of similar size and
  circumstance
• A common measuring stick to evaluate process
  performance
• CBA is Cost/Benefit Analysis
• Use to evaluate the benefits of a program or
  service
• ROI is Return on Investment
• How much your investment in the program or
  project earned for the company

Build It
47
Benchmarking

• Benchmarking, conducted properly, can
• improve your library's performance
• help you to gain or improve upper management
  support and
• help to prove the value of your library
• http//www.mlahq.org/members/benchmark/
• (from the MLA Benchmarking Network web page)

Build It
48
Benchmarking Partners

• Your own library
• The competition
• Other hospitals in your system
• Other hospitals in your area
• MLA Benchmarking

49
CBA

• Direct vs indirect benefits
• Contingent valuation
• Willingness-to-pay (WTP)
• Willingness-to-accept (WTA)
• Value of time

Build It
50
Willingness to Pay Survey

• About how many medical different books do you
  borrow per month from the Library? ___/mo
  (BOOKS1)
• How many medical books do you buy per month or
  per year? ___/mo. ___yr (BOOKS2 BOOKS2.1)
• Suppose that that the Library was closed
  indefinitely due to storm, fire, or earthquake
  damage and could not provide the medical books
  that you want. Also suppose that similar books
  are available to purchase at a price of 220
  each. How many of the books that you borrow per
  month or per year from the Library would you
  replace by purchases at 220 per book? ____ /mo.
  (BOOKS3 BOOKPR)

51
Calculating Willingness to Pay
52
Example Value of Book Purchases

• Every book purchased in a year is used on average
  2 times that year.
• Without a library each person who used the book
  that year would buy it at a cost of 250 per
  purchase.
• The Library gets a 20 discount and spends 200
  for one copy of the book.
• Benefit to the hospital 500-200 300/book

Build It
53
Example Value of Reference Services

• Librarian performs 8 searches per week, at a cost
  of 50 (15 min/search 8 searches 25/hr)
• 8 Professionals perform 1 searches per week each
  at a cost of 300 45min/search 8 searches
  50/hr)
• Savings or benefit 31.25/search, 250/wk,
  13,000/yr

Build It
54
ROI

• Business tool to determine earnings on money
  spent a percentage value
• CFOs are charged with getting the most return
  from every dollar invested or spent
• ROI of a library must show that the expense
  provides the highest return of any potential use
  of the budget dollars

Build It
55
CBA vs ROI

• CBA ratio that shows dollar value of benefits
  gained for dollar value of expenditures to
  achieve the benefit
• ROI percentage that shows the return (increase
  in value) on dollars spent to achieve a benefit.
• IRR Companies want to invest in the project
  that returns the highest rate, compared to rate
  earned in the securities market.

Build It
56
Process

• Design data collection method
• Collect data
• Convert data to monetary values
• Isolate the effects of the program
• Costs of the program must be included
• Difficult to use with qualitative data

Build It
57
An Example

• Library book budget 40,000 per year
• Average savings off list price is 20 because of
  librarians expertise at selecting vendor,
  negotiating best price and discounts, for a cost
  avoidance of 10,000 per year.
• New books typically circulate an average of 6
  times over a 3 year period.

Build It
58
An Example, cont

• If there were no library, users would need to
  spend their own for the books they need.
• Users would buy 2 copies of each book per year at
  100,000 (market value of the books)
• Library spent 40,000 to make the same books
  available to all interested users over the three
  years.

Build It
59
Simple First Year Benefit/Cost Ratio

• 40,000 spent on books provides a benefit of
  100k.
• The ratio of benefits to costs is 100k/40k, or
  2.51 or 2.50 in benefits for every 1 spent.
• BC over three years 300k/40k 7.51

Build It
60
Net Benefits and Costs

• Net benefits - difference between discounted
  benefits and discounted costs.
• Discounting annual benefits and costs are
  recalculated to reflect todays dollars.
• 100 benefit every year for 3 years, and
  inflation of 5 - total benefits are 228.75 not
  240.

Build It
61
Net Benefits and Costs

• Add costs of carrying out the service or program
  to the cash outlay discount any costs
• 40K for books (one time expense, no discounting)
• 10K (1/10 FTE discounted over 3 years) for
  personnel to select, order, catalog, shelve books
  Net benefits over three years
• 228.75 (net benefits)-50 net costs) 178.75,
  not 240 (benefits) - 40 (book budget) 200
• NBC over three years 3.5751

Build It
62
ROI

• Essentially the NBC ratio expressed as a
  percentage.
• The ROI reflects the realized by making an
  investment.
• ROI should be larger than the likely return on a
  bank account or certificate of deposit.
• Great stock market return is 10-12.

Build It
63
Calculating ROI

• Add costs of carrying out the service or program
  to the cash outlay
• 40K for books
• 10K for personnel to select, order, catalog,
  shelve books (remember to discount over 3 years)

Net Benefits Net Costs
X 100
Net Costs
Build It
64
ROI

• 228,750 50,000

X 100
50,000
357.5
Build It
65
Calculations you can make it say whatever you
want!
Build It
66
The Housewarming
67
The Housewarming

• Invite your friends
• Share evaluation results with users, stakeholders
• Update the décor
• Introduce new products, services identified in
  the evaluation
• Enjoy your (new) house
• Continue to provide services knowing that you are
  giving your institution the highest value for its
  money

Housewarming
68
Communicate the Results

• Purpose or use of the report
• Audience
• Information needs
• Report format
• Dissemination strategies
• Timeline

Housewarming
69
Report Guidelines

• Have partners review the draft report
• May be oral or written
• Always have an Executive Summary
• Make it brief, understandable, descriptive
• Figures, tables, graphs should be able to stand
  alone
• Explain figures, etc in narrative

Housewarming
70
Report Guidelines

• Provide purpose, background, and the methodology
  of the evaluation
• Always give conclusions and recommendations
• Establish clear link between conclusions and the
  original evaluation question
• Be sure to have appendices to support your report
  findings

Housewarming
71
Disseminate Results

• Identify audiences (administration, users,
  targeted groups, professional colleagues)
• Look for other publishing, presentation, or
  promotion opportunities
• Hospital newsletter, journal article, poster
  session, presentation at professional meeting
• Share with professional colleagues

Housewarming
72
Enjoy your new home!
73
Questions?

• Resources List for this class
• The OERC Web Site
• http//nnlm.gov/evaluation/
• Web Resources for Evaluation
• http//nnlm.gov/evaluation/webresources.htm

Housewarming
74
Logic Model Worksheet
75
Evaluation Plan Worksheet - sheet 1
Outcome
76
Evaluation Plan Worksheet - sheet 2
Outcome