Assess for Success: Proving Library Value

1
Assess for Success Proving Library Value

• Steve Hiller
• Director, Planning and Assessment University of
  Washington Libraries

2
Assessment Basics

• What do we need to know about our communities and
  customers to make them successful?
• How do we measure the effectiveness of our
  services, programs and resources?
• What do our stakeholders need to know in order to
  provide the resources needed for a successful
  library?

3
Assessment Is More than Numbers

• Library assessment is a structured process
• To learn about our communities
• To respond to the needs of our users
• To improve our programs and services
• To contribute to the success of our communities

4
Assessment Process

• Focuses on understanding customer needs and
  offering services that meet those needs
• Collects, analyzes and uses quantitative and
  qualitative data for more effective management
  and decision-making
• Emphasizes ongoing communication with customers
  and stakeholders,
• Seeks opportunities for collaboration and
  comparisons within the organization, institution
  and beyond

5
What Do We Need to Know About Our Customers?

• Who are our customers (and potential customers)?
• What areas/fields/courses are they working in?
• How do they work? Whats important to them?
• How do they find information needed for their
  work?
• How do they use our services? What would they
  change?
• How do they differ from each other in use/needs?
• Outcomes
• How do we add value to their work?
• How do we contribute to their success?

6
The Value of Community Assessment

• Identify actual and potential customers
• Understand needs and use preferences
• Use funding and staff effectively
• Understand and address competition
• Encourage community involvement and ownership
• Target marketing, market penetration and outreach
• Measure, demonstrate, present the value of the
  library to the community and stakeholders

7
Thinking Strategically About Library Futures

• What is our central work and how can we do more,
  differently, and at less cost?
• What important services do we provide that others
  cant?
• What advantages do we possess?
• How is customer behavior changing?
• How do we add value to our customers work?
• What are the essential factors responsible for
  library success now and in the future?

8
The Challenge for Libraries

• Traditional statistics and data are no longer
  sufficient
• Emphasize inputs how big and how many
• Do not tell the librarys story
• May not align with organizational goals and plans
• Do not measure service quality
• Need data from the users perspective
• Need to provide organizational accountability to
  funders and other stakeholders
• Need to define where we want to be in 3/5/more
  years
• Need to answer What value do we provide our
  primary community and how do we show it?

9
Defining Success in a Digital Environment

• Crafting new indicators and measures of success
• Moving from measuring inputs to outputs and
  outcomes
• Understanding impact of library/partner roles and
  services on the community
• Agreeing on qualitative measures of success user
  perceptions, user success, creating value,
  advancing community goals.
• Reallocating resources and managing capabilities
  to achieve success.
• Demonstrating value of library contributions

10
Some New Metrics for Academic Libraries

• Uniqueness of collections
• Value of consortia
• Administrative and budgetary efficiencies
• Student outcomes/student learning/graduate
  success
• Contributions to faculty productivity
• Social frameworks/intellectual networks
• Generating new knowledge
• Creating the collective good with reusable assets
  such as an institutional repository or e-science

11
Good Assessment Starts Before You Begin . . .
Some Questions to Ask

• Define the question
• What do you need to know and why
• How will you use the information/results
• Where/how will you get the information
• What are appropriate methods
• Is there existing information
• New data (where or who will you get it from)
• Is it cost and resource effective
• How will you analyze the information
• Who will act upon the findings

12
How Do We Get Customer Information?

• Statistics/data mining (local, institutional)
• Surveys
• Focus groups
• Observation
• Usability
• Interviews
• Embedding
• Logged activities
• Comments, suggestions, over the counter, being
  there

13
Another View of Metrics
14
Getting The Right Data Isnt Enough

• but to suppose that the facts, once established
  in all their fullness, will speak for
  themselves is an illusion.
• Carl Becker
• Annual Address of the President of the American
  Historical Association, 1931
• Understanding what the data means and how to
  follow-up are crucial

15
Make Data Meaningful

• Summarize
• Compare
• Analyze
• Present
• Go below the surface to examine results by
• Demographic group
• Users and non-users
• Stakeholders vs non-stakeholders
• Compare current data with past look for trends
• How can we use the data for action?

16
Data Presentation 3 Key Questions

• Whats the message?
• Fewer messages means greater impact
• Whos the audience?
• Multiple audiences may need different
  presentations
• How do we present?
• Quantitative data - Be graphic , provide
  meaning/understanding
• Qualitative data - Be selective, provide
  corroborating data
• Keep it simple AND focused!

17
And Remember the Content!
18
Moving from Analysis to Action

• From all of the data, determine what can and
  should be addressed
• Prioritize the important/critical action items
• Make evidence-based decisions
• Align with mission, vision and goals of parent
  organization
• Address what is important to customers
• Establish action guidelines
• Who, how, when

19
What Makes it Hard to Be Evidence-Based?(From
Pfeffer and Sutton, 2006)

• Theres too much evidence
• Theres not enough good evidence
• The evidence doesnt quite apply
• People are trying to mislead you
• You are trying to mislead you
• The side effects outweigh the cure
• Stories are more persuasive anyways

20
When the Evidence Isnt Used
21
Collection Assessment The 4 Questions

• Whats important from the
• Library perspective?
• Customer perspective?
• Stakeholder perspective?
• Vendor/publisher perspective(s)?

22
Some Good Reasons to Assess Collections

• Meeting user needs
• Costs/Budget/Space
• Changes in the information discovery/use
  processes
• Accountability to stakeholders, funding agencies
• Breadth and depth comprehensiveness
• Environmental changes (social, information,
  technical)
• Changes in publisher/vendor marketing and
  packaging
• Return on Investment (ROI) contingent valuation

23
What Numerical/Cost Data Do We Need?

• Costs
• Collections (annual/over time)
• Titles
• Packages, consortia
• Operational
• Staff
• Facilities
• Vendors
• Usage
• Print (on-site, check-out, ILL)
• Electronic/online
• Type of use
• Packages, consortial, doc delivery
• Change over time

• Cost per use
• What type of use
• Downloads
• Searches
• Loans
• Page views
• Packages, individual titles
• Global information
• Collection comparisons
• Publishing output
• Bibliometric tools, citation analyses, collection
• Community demographics

24
So We Need Systems/Products That

• Enable us to track and validate usage
• Enable us to track item level and package level
  costs
• Provide comparative information
• Provide canned/custom reports easily
• Provide data in flexible formats we can
  manipulate/analyse
• Provide information that meets established
  standards
• Present snapshot and trend data
• Are cost-effective

25
E-Metrics Tools COUNTER, SUSHI and ERMI

• Counting Online Usage of NeTworked Electronic
  Resources
• Standardized way to count usage
• Defined reports for journals, databases, and
  eBooks
• Standardized Usage Statistics Harvesting
  Initiative
• Model for automation of statistics harvesting
• SUSHI feed usage data that magically appears!
• Electronic Resources Management Initiative
• Can accept SUSHI feeds
• Easy to generate COUNTER-style reports within ERM
  module
• Brings in data from order records to calculate
  cost per use

26
(No Transcript)
27
Genesis Statistics Detail
2007 payment / 2007 Use 2007 CPU
1950.71 / 649 3.00
28
Dont Forget the Qualitative

• Discovery pathways and user expertise
• Usability
• User needs
• User outcomes
• What did the information enable the user to
  accomplish?

29
Integrated Collection Assessment

• Collections-centered
• Size, growth, formats, authors, subjects,
  comparisons
• Cost-centered
• Expenditures, price, inflation, cost per use,
  return on investment
• Use-centered
• Loans, e-usage, purchase requests, ILL
• Customer centered (secondary)
• Community demographics, programs, research,
  bibliometric studies, consortial
• Customer centered (primary)
• Behavior, impact, outcomes, success, discovery

30
What Will We Do With the Information?

• Analyze and understand
• Present and share
• Make good decisions and choices
• Ensure we add value to the work of our community
• Customers and stakeholders make the ultimate
  determination of value!

31
Focus on the UW Libraries Contribution to the
Research Enterprise

• How does the library support sponsored research?
• What is the value of the library contribution?
• Some measures indicators of library
  contributions
• Collections budget and usage data
• Survey information
• Bibliometrics
• Frequency of use
• Time saved/contingent valuation
• Information seeking behavior
• Qualitative information
• Services used
• Library research projects/collaborations

32
The Importance of the Research Enterprise
University of Washington Operating Revenues 2.4
Billion in 2007-08
Research Grants 1.1 Billion Health and Human
Services 510 million National Science
Foundation 80 million Other federal
agencies 200 million Industry/Foundations 110
million Other non-federal 160 million
33
University of Washington Libraries Assessment
Methods Used

• Large scale user surveys every 3 years
  (triennial survey) beginning in 1992. In 2007
• 1500 faculty responses 600 grad 500 undergrad
• In-library use surveys every 3 years beginning
  1993
• 4000 surveys returned in 2008
• Focus groups/Interviews (annually since 1998)
• Observation (guided and non-obtrusive)
• Usability
• Usage statistics/data mining
• Information about assessment program available
  at
• http//www.lib.washington.edu/assessment/

34

The Changing Business Model Trends in Library
Use at the University of Washington

• UW Libraries Usage Data
• Items Used In-Library
• 800,000 in 2002-03
• 300,000 in 2007-08
• (6.0 million article downloads in 2007-08)
• Gate Counts
• 4.6 million in 2002-03
• 4.3 million in 2007-08
• (8.7 million Web site user sessions in 2007-08)
  
• In-Person Reference Questions
• 140,000 in 2002-03
• 90,000 in 2007-08
• (18 of all ref queries are virtual in 2007-08)

• Traditional Library Core Business (Usually
  in-person)
• Physical Collections
• Print (primarily)
• Microforms
• Other
• Facilities
• House collections
• Customer service work space
• Staff work space
• Services
• Reference
• Instruction
• Access

35
Qualitative Information Provides Context
BioScience Interview/Focus Groups (2006)

• Content is primary link to the library
• Identify library with ejournals want more titles
  backfiles
• Provide library-related services and resources in
  our space not yours
• Discovery begins primarily outside of library
  space with Google and Pub Med Web of Science
  also important
• Library services/tools seen as overly complex and
  fragmented
• Print is dead, really dead
• If not online want digital delivery
• Go to physical library only as last resort
• Personal connection is important

36
(No Transcript)
37
Off-Campus Remote Use 1998-2007(Percentage using
library services/collections at least 2x week)
76 of faculty (80 of those using federal
research funds) connect online at least 2x week
38
Primary Reasons for Faculty Use of Libraries Web
Sites by Academic Area (2007 Triennial Survey, at
least 2x per week)
39
Importance of Books, Journals, Databases
byAcademic Area Scale of 1 not important to 5
very important)
40
UW Libraries Serial Purchasing and Usage

• 85 of collections budget spent on
  serials/backfiles
• Ranked 4th in 2006-07 ARL serial expenditures
  (11.4 million)
• 6 million journal article downloads (Counter
  compliant)
• 30 increase over 3 years
• 75 of downloads from scholarly journals in
  science-engineering-health sciences
• Bibliographic database use has declined
• Most pronounced in subject specific databases
• Web of Science unchanged at 160,000 login
  sessions annually (13 of all log-in sessions)

41
UW Libraries Contribution to Sponsored Research

• Among faculty receiving federal research funding
• 97 rate Libraries as very important to their
  work
• 96 rate journals as most important info resource
  for their work
• 93 rate the Libraries as making a major
  contribution to keeping current in their field
• 93 rate the Libraries as making a major
  contribution to their research productivity
• 80 connect to the Libraries at least twice per
  week from office, lab or off-campus

42
Differences in Library Contribution by Funding
Source (Scale of 1 Low to 5 High)
43
Libraries Contribution to Your Being a More
Productive Researcher by Academic Area and
Funding Source
44
Closing the Loop Success with Assessment

• Assess what is important
• Keep expectations reasonable and achievable
• Use multiple assessment methods corroborate
• Mine/repurpose existing data
• Focus on users how they work, find use
  information
• Use the data to improve and add customer value
• Keep staff, customers and stakeholders involved
  and informed

45
Eye to the Future

• Measuring performance is an exercise in measuring
  the past. It is the use of that data to plan an
  improved future that is all important. Peter
  Brophy
• Data trends can inform the future
• Strategic planning can frame the future
• Organizational performance models can align
  ongoing operations with future aspirations
• Understanding how customers work and how that
  work is changing is key to our future