Effective, Sustainable

1
Effective, Sustainable Practical Assessment

• Steve Hiller, Director, Assessment and Planning,
  UWMartha Kyrillidou, Statistics and Service
  Quality Programs, ARLJim Self, Director,
  Management and Information Services, UVA
• Boston University
• September 08, 2008

2
Monday Morning Metrics With Steve

• Baseball
• Presentation 535
• 8 September 2008
• Steve Hiller, UW Libraries

3
Case Study Worst Trade Ever Made by the Seattle
Mariners

• When
• Who
• Why was it the worst
• Evaluating performance
• Defining success and value
• Statistics
• Intangibles

4
Library AssessmentMore than Numbers

• Library assessment is a structured process
• To learn about our communities
• To respond to the needs of our users
• To improve our programs and services
• To support the goals of the communities

5
Why Assess?

• Accountability and justification
• Improvement of services
• Comparisons with others
• Identification of changing patterns
• Marketing and promotion
• Opportunity to tell our own story
• Using data, not assumptions, to make decisions
• Assumicide!

6
The Challenge for Libraries

• Traditional statistics are no longer sufficient
• Emphasize inputs how big and how many
• Do not tell the librarys story
• May not align with organizational goals and plans
• Do not measure service quality
• Need measurements from the users perspective
• Need the organizational culture and the skills to
  answer a basic question
• What difference do we make to our communities?

7

• If you think you're too small to have an impact,
  try going to bed with a mosquito.
• Anita Roddick

8
Charting User Change
User Behavior
User Expectations
Findings
Wide Array of User Studies Now Available
Self-sufficiency control of information seeking
process
Customer Service (qualified helpful staff)
Library as a place, symbol, refuge
Ready access to wide range of content (e.g.
complete runs of journals)
Born Digital
9
Re-conceptualizing Library Facilities
Changing nature of library usage
Re-configuring library facilities
Library as physical place, intellectual space,
and community center
Learning commons
Collaborative study
Social and intellectual center
Secondary Storage
Showcase for recruitment
10
ARL Sponsored Assessment

• Tools - StatsQUAL
• ARL Statistics - descriptive, longitudinal,
  comparative
• LibQUAL, ClimateQUAL, DigiQUAL
• MINES for Libraries, E-metrics . Google
  Analytics
• Building a Community of Practice
• Library Assessment Conferences
• Service Quality Evaluation Academy (training
  events)
• Library Assessment blog
• Individual Library Consultation (Jim and Steve)
• Making Library Assessment Work (24 libraries in
  2005-06)
• Effective, Sustainable, Practical Library
  Assessment (14 in 2007-08)

11
ARL Tools for Library Assessment

• As a result of the work of the New Measures and
  Assessment Initiative (1999)

MINES for Libraries Since 2003
LibQUAL Since 2000
DigiQUAL Since 2003
ClimateQUAL Since 2007
ARL Statistics Since 1907-08
12
Survey Structure (Detail View)
13
22 Items and The Box.

• Why the Box is so Important
• About 40 of participants provide open-ended
  comments, and these are linked to demographics
  and quantitative data.
• Users elaborate the details of their concerns.
• Users feel the need to be constructive in their
  criticisms, and offer specific suggestions for
  action.

14
MINES for LibrariesTM

• MINES is a transaction-based research methodology
  consisting of a web-based survey form and a
  random moments sampling plan
• MINES typically measures who is using electronic
  resources, where users are located at the time of
  use, and their purpose of use in the least
  obtrusive way
• MINES was adopted by the Association of Research
  Libraries (ARL) as part of the New Measures
  toolkit in May, 2003.
• MINES is different from other electronic resource
  usage measures that quantify total usage (e.g.,
  Project COUNTER, E-Metrics) or measure how well a
  library makes electronic resources accessible
  (LibQUAL).

15
ESP Insights

• Strong interest in using assessment to improve
• Uncertainty on how to establish and sustain
  assessment
• Lack of assessment knowledge among staff
• More data collection than data utilization
• Effectiveness not dependent on library size or
  budget
• Each library has a unique culture and mission

16
Effective Assessment

• Focuses on the customer
• Is aligned with library and university goals
• Assesses what is important
• Is outcomes oriented
• Develops criteria for success
• Uses appropriate and multiple assessment methods
• Uses corroboration from other sources
• Provides results that can be used

17
Sustainable Assessment Needs . .

• Organizational leadership
• Sufficient resources
• Supportive organizational culture
• Identifiable organizational responsibility
• Connection to strategic planning and priorities
• Iterative process of data collection, analysis,
  and use
• Involvement of customers, staff and stakeholders

18
Practical Assessment

• Keep it simple and focused less is more
• Know when enough is enough
• Use assessment that adds value for customers
• Present results that are understandable
• Organize to act on results

19
The University of Virginia

• 14,000 undergraduates
• 66 in-state, 34 out
• Most notable for liberal arts
• Highly ranked by U.S. News
• 6,000 graduate students
• Prominent for humanities, law, business
• Plans expansion in sciences
• Located in Charlottesville
• Metro population of 160,000

20
Collecting the Data at the U.Va. Library

• Customer Surveys
• Staff Surveys
• Mining Existing Records
• Comparisons with peers
• Qualitative techniques
• Long involvement with ARL statistics

21
Management Information Services

• MIS committee formed in 1992
• Evolved into a department 1996-2000
• Currently three staff
• Coordinates collection of statistics
• Publishes annual statistical report
• Coordinates assessment
• Resource for management and staff

22
but to suppose that the facts, once established
in all their fullness, will speak for
themselves is an illusion.

• Carl Becker
• Annual Address of the President of the
• American Historical Association, 1931

www.arl.org
23
UVa Customer Surveys

• Faculty
• 1993, 1996, 2000, 2004
• Response rates 59 to 70
• Students
• 1994, 1998, 2001, 2005
• Separate analysis for grads and undergrads
• Response rates 43 to 63
• LibQual 2006
• Response rates 14 to 24
• Annual Surveys 2008
• Student samples
• One third of faculty
• Response rates 29 to 47

24
Corroboration

• Data are more credible if they are supported by
  other information
• John Le Carres two proofs

www.arl.org
25
Analyzing U.Va. Survey Results

• Two Scores for Resources, Services, Facilities
• Satisfaction Mean Rating (1 to 5)
• Visibility Percentage Answering the Question
• Permits comparison over time and among groups
• Identifies areas that need more attention

www.arl.org
26
U.Va. Reference Activity and Reference
Visibility in Student Surveys
www.arl.org
27
Data Mining

• Acquisitions
• Circulation
• Finance
• University Records

www.arl.org
28
Investment and Customer ActivityUniversity of
Virginia Library1993-2006
29
The Balanced Scorecardat the U.Va. Library

• Implemented in 2001
• Results tallied FY02 through FY07
• Tallying results for FY08
• Completing metrics for FY09
• Builds upon a rich history of collecting data
• A work in progress

30
The Balanced ScorecardManaging and Assessing Data

• The Balanced Scorecard is a layered and
  categorized instrument that
• Identifies the important statistics
• Ensures a proper balance
• Organizes multiple statistics into an
  intelligible framework

31
Metrics

• Specific targets indicating full success, partial
  success, and failure
• At the end of the year we know if we have met our
  target for each metric
• The metric may be a complex measure encompassing
  several elements

32
What Do We Measure?

• Customer survey ratings
• Staff survey ratings
• Timeliness and cost of service
• Usability testing of web resources
• Success in fundraising
• Comparisons with peers

33
Metric L.2.B Retention Rate of Employees

• Target1 Retain 95 of employees.
• Target2 Retain 90 of employees.
• Result FY08 Target1.
• 95 of employees retained.

34
Metric U.4.B Turnaround time for user requests

• Target1 75 of user requests for new books
  should be filled within 7 days.
• Target2 50 of user requests for new books
  should be filled within 7 days.
• Result FY07 Target1.
• 77 filled within 7 days.

35
Metric U.3.A Circulation of New Monographs

• Target1 60 of newly cataloged monographs should
  circulate within two years.
• Target2 50 of new monographs should circulate
  within two years.
• Result FY07 Target1.
• 63 circulated.

36
Balanced ScorecardU.Va. Library FY2007
37
Using Data for Results at UVa

• Additional resources for the science libraries
  (1994)
• Redefinition of collection development (1996)
• Initiative to improve shelving (1999)
• Undergraduate library open 24 hours (2000)
• Additional resources for the Fine Arts Library
  (2000)
• Support for transition from print to e-journals
  (2004)
• New and Improved Study Space (2005-06)
• Increased appreciation of the role of journals
  (2007)

38
University of Washington

• Located in Seattle metro population 3.2 million
• Comprehensive public research university
• 27,000 undergraduate students
• 12,000 graduate and professional students (80
  doctoral programs)
• 4,000 research and teaching faculty
• 800 million yearly in U.S. research funds (2)
• Large research library system
• 40 million annual budget
• 150 librarians on 3 campuses

39
The Basic Question How Does the Library
Contribute to the Success of our Researchers and
Students?

• Our assessment priorities
• Information seeking behavior and use
• Patterns of library use
• User needs
• Library contribution to learning and research
• User satisfaction with services, collections,
  overall
• What have we learned (short version)
• Faculty perceive success through collections
• Grad students through timely access to resources
  and services
• Undergrads through library as place for work and
  community

40
University of Washington Libraries Assessment
Methods Used

• Large scale user surveys every 3 years
  (triennial survey) 1992, 1995, 1998, 2001,
  2004, 2007
• In-library use surveys every 3 years beginning
  1993
• 4000 surveys returned in 2008
• Focus groups/Interviews (annually since 1998)
• Observation (guided and non-obtrusive)
• Usability
• Usage statistics/data mining
• Information about assessment program available
  at
• http//www.lib.washington.edu/assessment/

41
UW Triennial Library Survey Number of
Respondents and Response Rate 1992-2007
42
(No Transcript)
43
What Did You Do in the Library Today?(In-Library
Use Surveys 2008/2005)
44
Activity in the Library by Group2008 Users 73
UG, 22 Grad, 5 Faculty
45
Usefulness of New and/or Expanded Services for
Undergrads Library as Place (2007 Triennial
Survey)
46
Library As Place Using the Results

• Libraries are student places
• 350 computer lab installed in Undergrad Library
  Autumn 1998
• Hours extended to 24/5.5 in Undergrad Library
  2002
• Collection footprint reduced
• Diversified user spaces provided (group, quiet,
  presentation)
• Student advisory committee provides ongoing
  feedback
• Add other collaborative student support services
  into library
• Upgrade/renovate facilities to meet student needs
• Furniture that encourages collaboration
• More electrical outlets
• Better lighting and noise control
• Plan for major renovation of Undergraduate Library

47
What Faculty/Grad Students Told Us Bioscience
Interview/Focus Groups (2006)

• Content is primary link to the library
• Identify library with ejournals want more titles
  backfiles
• Provide library-related services and resources in
  our space not yours
• Discovery begins primarily outside of library
  space with Google and Pub Med Web of Science
  also important
• Library services/tools seen as overly complex and
  fragmented
• Print is dead, really dead
• If not online want digital delivery/too many
  libraries
• Go to physical library only as last resort
• Uneven awareness of library resources and services

48
Sources Consulted for Information on Research
Topics (Scale of 1 Not at All to 5 Usually)
49
Off-Campus Remote Use 1998-2007(Percentage using
library services/collections at least 2x week)
76 of faculty and 72 of graduate students use
library services online at least 2x week
50
Primary Reasons for Faculty Use of Libraries Web
Sites 2007 (at least 2x per week)
51
E-Journal Usage at UW

• Scholarly Stats 2007 5 million article requests
  from 19 vendors/platforms - 7 accounted for
  nearly 75
• High Wire Press 1,200,000 Biomedical
• Science Direct 975,000 Science-Medical
• JSTOR 750,000 Multidisciplinary
• Nature 300,000 Science-Medical
• Meta Press 250,000 Science-Medical
• Blackwell Synergy 225,000 Science-Medical
• Ovid 225,000 Biomedical

52
Importance of Books, Journals, DatabasesAcademic
Area (2007, Faculty, Scale of 1 not important
to 5 very important)
53
Bibliographic Database Use Login Sessions
54
Librarian Liaison Satisfaction VisibilityBy
Selected School (2007 Triennial Survey
Satisfaction on 1 to 5 scale visibility who
rated)
55
Usefulness of New/Expanded Services for Faculty
Grads Integrate into my space
56
Integrate Library Services Resources into User
Workflows Follow-Up Actions

• WorldCat Local became primary catalog in 2007
• Scan on demand pilot began 2008
• Increase chat and remote services staffing
• Increase ILL staffing (due to WorldCat Local)
• Integrate course reserves services into My UW
  portal
• Redesign UW Libraries Homepage
• Use qualitative methods to gain deeper
  understanding of user work and behavior
• Strengthen librarian liaison efforts to academic
  programs

57
How UW Libraries Has Used Assessment

• Extend hours in Undergraduate Library (24/5.5)
• Create more diversified student learning spaces
• Enhance usability of discovery tools and website
• Provide standardized service training for all
  staff
• Review and restructure librarian liaison program
• Consolidate and merge branch libraries
• Change/reallocate collections budget
• Change/reallocate staffing
• Support budget requests to University

58
Overall Satisfaction by Group 1995-2007
You guys and gals rock!!!!!! We need to invest
in our library system to keep it the best system
in America. The tops! My reputation is in large
part due to you. Professor, Forest Resources
59
Four Useful Assessment Assumptions

• Your problem/issue is not as unique as you think
• You have more data/information than you think
• You need less data/information than you think
• There are useful methods that are much simpler
  than you think
• Adapted from Douglas Hubbard, How to Measure
  Anything (2007)

60
in conclusion Assessment is not

• Free and easy
• A one-time effort
• A complete diagnosis
• A roadmap to the future

61
Assessment is

• A way to improve
• An opportunity to know our customers
• A chance to tell our own story
• A positive experience

62
Moving Forward

• Keep expectations reasonable and achievable
• Strive for accuracy and honestynot perfection
• Assess what is important
• Use the data to improve
• Keep everyone involved and informed
• Focus on the customer

63
For more information

• Steve Hiller
• hiller_at_u.washington.edu
• www.lib.washington.edu/assessment/
• Jim Self
• self_at_virginia.edu
• www.lib.virginia.edu/mis
• www.lib.virginia.edu/bsc
• ARL Assessment Service
• www.arl.org/stats/initiatives/esp/