uniwersytety

1

Library performance indicators does it really
make sense to measure them?
Data collection and analysis of library
performance a case study for Polish research
libraries
Blazej Feret The Main Libray of theTechnical
University of Lódz, Poland blazej.feret_at_sunlib.p.l
odz.pl Marzena Marcinek Cracow University of
Technology Library, Poland marcinek_at_biblos.pk.edu.
pl
CASLIN Seminar, June 2006
2
Ackowledgements
Special thanks go for Mrs Lidia Derfert-Wolf, a
member of the Task Group for Standardisation for
Polish Research Libraries, for her kind
assistance and advice.
3
Plan of the workshop

• 1. Introduction 15 minutes
• 2. A few general remarks on library quality and
  performance measurement
• Quality criteria from different
  perspectives 10 minutes
• 3. A Common Project of Polish Research Libraries
  on Comparable Measures (incl. examples) 40
  minutes
• Czech realities 20 minutes
• 4. Standardised terminology and descriptions of
  library performance indicators based on the ISO
  11620 Standard Information and documentation
  Library performance indicators 20
  minutes
• Conclusions 15 minutes

4

• A few general remarks
• on library quality and performance indicators

5

• Quality
• fitness for purpose
• fitness for use
• conformity to requirements
• absence of defects
• Quality of libraries (ISO 11620)
• totality of features and characteristics of a
  product or services that bear on the library's
  ability to satisfy stated or implied needs

6
Quality of research libraries from
the perspective of
Funding bodies
Users
Librarians
Please consider all these perspectives
7
Model of measuring quality of libraries

• inputs - the raw data such as finance,
  collection, equipment, users and staff, space,
  seats
• outputs - the work done, i.e. circulation,
  cataloguing, reference services, preservation,
  interlibrary lending, facilities usage and
  e-sources searches
• outcomes user satisfaction and the impact of
  library services on users at the local
  institution and society

• quantitative data
• qualitative data

• from the perspective of
• users
• funding bodies
• librarians

group work
8
Library quality assessment- what is required?

• set of quality criteria librarys goals and
  objectives
• set of performance indicators
• national library statistics system
• standardisation on local, national and
  international level
• but ...

9

• The level of quality cannot be defined once and
  for all, since both the criteria and evaluation
  methods, as well as the assessment of the results
  achieved, may change.
• This is caused by various factors, for example
  technological, political, economic, as well as
  the ones connected with the community in which
  and for whom information services work. It is
  crucial that these varying criteria and methods,
  as well as the dissimilarity in the level of the
  quality achieved by a given country, are taken
  into account when aiming at adopting the
  international standards on the quality of
  information work.
• Countries differ in living standards they
  achieve, in the level of education, and the way
  they introduce innovations. Setting the goal is
  of fundamental importance, since its
  accomplishment or failure to reach it may be
  treated as a basis for quality assessment.
• W. Pindlowa
•  

10
Activities conducted by the Group for
Standardisation for Polish Research Libraries,
results and plans for the future

CASLIN Seminar, June 2006
11
The Group for Standardisation for Polish Research
Libraries coordinated by the Adam Mickiewicz
University in Poznan

• Lidia Derfert-Wolf (ATR, Bydgoszcz)
• Ewa Dobrzynska-Lankosz (AGH, Krakow)
• Miroslaw Górny (Adam Mickiewicz University,
  Poznan)
• Elzbieta Górska (Warsaw Public Library)
• Marek Górski (Cracow University of Technology)
• Artur Jazdon (Adam Mickiewicz University,
  Poznan)
• Dariusz Pawelec (Silesian University, Katowice)
• Anna Sokolowska-Gogut (Academy of Economics,
  Krakow)
• Teresa Wildhardt (Pedagogical University,
  Krakow)

12

• Objectives

• to gather libraries' statistical data for a
  computer database
• to select a set of performance indicators and
  standards for library performance (quantity,
  quality and effectiveness)
• to conduct a comparative research
• to prepare and publish yearly reports
• to define methods for the assessment of
  Polish research libraries

13
Project for the Analysis of Polish Research
Libraries
14
Some facts as of May 2006

• Libraries registered 59
• 54 academic libraries (42 state-owned and 11 non
  state-owned)
• 3 public libraries
• 2 special libraries
• Questionnaires for 2004 completed 31 libraries
• Questionnaires for 2003 completed 29 libraries
• Questionnaires for 2002 completed 17 libraries

15
Software for the collection and analysis of
data - general requirements

• on-line access to the questionnaire
  (submission, modification)
• selected performance indicators automatically
  calculated and presented
• automatic control and verification of the
  accuracy of submitted data
• access to analysing functions for individual
  libraries
• possibility to conduct a multi-aspect
  comparative analysis of selected data and
  performance indicators

16

• Web-based Application
• for
• General users
• Library directors
• Analysts
• Administrators

17
General user

• General information about the Project
• Instructions
• Registration
• Rules for filling up questionnaires
• Data analyses
• Questionnaires (patterns in doc. format)
• Example results
• Useful links

18
http//ssk2.bu.amu.edu.pl/standaryzacja/index.htm
General information
Instructions
Questionnaires
Results
handouts 1
Useful links
19
(No Transcript)
20
Library Director

• Filling up questionnaires
• Analysis of the own library data sorted by years,
  question categories or indicators
• Access to calculated automatically indicators
• Comparison of the own library data with average
  results of other libraries in the same category
  or in the country (data published in the form of
  tables).
• Establishment and modification of access rights
  for the library staff

21
Login
22
For directors of registered libraries
Users
Questionnaires
Analyses
Send e-mail
Change password
Logout
23
handouts 2
Questionnaire

• 48 questions of various types
• refer to easily accessible or computable data
  (e.g. size of collection, number of users etc.)
• closed questions about the services offered
  (e.g. on-line reservation Yes/No)
• 88 performance indicators
• 19 calculated by librarians
• 69 calculated automatically

24
Questionnaire

• addresses all the elements of a library system,
  its environment, library processes and services
• divided into chapters
• Staff
• Collection
• Budget
• Infrastructure
• Circulation
• Information services
• Didactics
• Publications and data bases created by the
  library
• Library cooperation
• Organisation of library events
• Professional activity of library staff
• ,

25
Examples of performance indicators required to
complete the questionnaire

• librarys total expenditures
• expenditures for library materials/books
• ratio of library budget to the budget of its
  parent university
• time required for the technical processing of
  a document
• collection on the computer system as a of the
  whole collection of the library
• percent of catalogue descriptions acquired from
  outside resources

26
Examples of performance indicators calculated
automatically

• library expenditure per student/user
• expenditures for library materials/books per
  student/user
• library registered users as a percent of
  potental users
• space of the library per user
• collection on the computer system as a of the
  whole collection of the library
• number of user training hours per one staff
  member

27
Survey form
For university libraries
No of libraries in the university library and
information system (incl. the main library)
data concern
main library
university library and information system
Staff
commentary
No of library staff
categories
education
age
28
Collection
Acquisition
29
Budget
30
Infrastructure
Circulation
31
(No Transcript)
32
Standard lending period
Information services
Electronic sources usage
Didactics
Librarys own publications and databases
33
Librarys own publications and databases
Interlibrary cooperation, staff professional
activities
34
Selected performance indicators
35
Once the questionnaire is validated the
indicators are calcullated automaticly
Indicators calculated automatically
Admin can add other indicators based on the data
gathered - no action by director required
36
Adding and modifying users
37
statistical data
values
Director can analyse performance of his/her own
library for selected years
38
categories for the analysis
Tyle kategorii ile pytan i wskaznikow wpisanych i
automatycznych
Number of categories number of data and
indicators (entered and calculated automatically)
39
Total library space per student
No of loans per registered user
No of loans per staff membber
40
No of years analysed 3
Total library space per student
No of loans per one registered user
No of loans per staff member
41
Analyst

• Access to all the questionnaires
• No modification rights

Analysis
Questionnaires
42
Selection of the library categories and years
Type academic
Other types
Source of funding
Analyse the years
Sort by (years, types)
43
180 categories for the analysis found, please
select
Staff
Collection
44
Indicators calculated automaticaly
Total library space per one user
No of loans per one registered user
45
Research libraries performance in 2003
Total library space per user
No of loans per one registered user
46
Research libraries performance in 2004
No of loans per one registered user
No of loans per staff member
47
(No Transcript)
48
Administartors Module
Libraries
Users
Questionnaires
Forms
Outstanding questionnaires
Send e-mail
Change password
Database statistics
logout
49
Registration form
Send via snail mail to.
50
Add / modify library
Sortowanie
Sort by
Source of funding
Type
Name
51
Select the library
52
List of Libraries with outstanding questionnaires
Questionnaires
missing
not validated
validated by director
Validated by administrator
53
The analysis of data examples (no complete
data for 2005 yet)
54
(No Transcript)
55
(No Transcript)
56
Structure of budget in state university libraries
in the years 2002 and 2003
other
salaries
collection
administration
space
hardware software
57
Structure of budget in state technical
universities libraries in the years 2002 and 2003
other
salaries
collection
administration
space
hardware software
58
Selected indicators for state academic libraries
No of books per one user vols
all academic libraries
technical universities
universities
59
Selected indicators for state academic libraries
No of new documents (books) per user vols.
60
Selected indicators for state academic libraries
Expenditures for new collection per user
Expenditures per user
All academic libraries
Technical universities
universities
All academic libraries
Technical universities
Universities
61
Selected indicators for state academic libraries
Library budget as percent of parent university
budget
Budzet biblioteki jako procent budzetu uczelni -
rok 2002
rok 2003
62
Selected indicators for state academic libraries
Staff with MA degree in librarianship as a
percent of the whole library staff
Pracownicy z wyzszym wyksztalceniem
bibliotekarskim jako odsetek pracowników
dzialalnosci podstawowej
63

• Feedback loop

64
Problems arisen when receiving data

• lack of statistical data required to complete
  the questionnaire
• lack of comparable data on the use of
  electronic resources (incl. differences in usage
  statistics generated by various providers)
• differences in library structure and budgeting
  within university
• difficulties with validation mistakes (e.g.
  wrong ratio) need correction, misunderstanding of
  data requirements, wrong interpretation of
  questions

65
...and solving them

• Verivfication of data required
• Verification of indicators
• Software tools to control data in the fields
• Detailed commentaries

problems noticed by librarians and analysts
product
improvements
testing
66
Selection of performance indicators the reasons
for measurement What? Why? At what cost?

• Part of reporting mechanism (statistics)
• Part of internal assessment
• Support for decision making (locally, within
  parent organisation, at the national level)
• External requirements

67
Library performance indicators in the ISO
116201998 Standard
68
The purpose of library performance indicators

• To function as tools to assess the quality and
  effectiveness of activities and services provided
  by a library
• To assess the efficiency of resources allocated
  by the library to its activities and services

69
Required features of performance indicators

• Informative content
• Reliability
• Validity
• Appropriateness
• Practicality
• Comparability

70
Informative content

• The indicator has to be informative as a tool for
  measuring activity, for identyfying achievements
  and problems in the performance of the library so
  that action can be taken to remedy this.
• It should provide information for
  decision-making, e.g. goalsetting, budget
  allocation, prioritizing services and activities.

71
Reliability

• A performance indicator should be reliable in the
  sense that it consistently produces the same
  results when used repeatedly under the same
  circumstances.

72
Validity

• A performance indicator must be valid, that is,
  it must measure what it is intended to measure.

73
Appropriateness

• A performance indicator must be appropriate for
  the purpose to which it is to be put. That is,
  the units and scale must be suitable and the
  operations necessary to implement the process of
  measurement should be compatible with the
  librarys procedures, physical layout, etc.

74
Practicality

• A performance indicator has to be practical in
  the sense that it uses data that can be made
  available by the library with a resonable amount
  of effort in terms of staff time, staff
  qualifications, operational costs and users time
  and patience.

75
Comparability

• A performance indicator allows comparison between
  libraries if the same score, making allowance for
  the accuracy of the score, means the same level
  of quality of services or the same level of
  efficiency in the libraries to be compared.

76
Some definitions group work

• Performance
• Indicator
• Evaluation
• Accessability

• Effectiveness
• Efficiency
• Reliability
• Validity

• Loan
• Recurrent expenditure
• Resources
• Target population

Handouts 3 and 4 afterwards
77
Performance indicators a descriptive framework
Handout 5

• Name
• Objective
• Scope
• Definition
• Method
• Interpretation
• Factors affecting the indicator
• Related indicators

78
Uses of performance indicators

• The quality and effectiveness of the services of
  the library as well as the efficiency of the uses
  of the resources are evaluated against the
  mission, goal and objectives of the library
  itself.
• Performance indicators should be linked to
  systematic library planning and evaluation.
• Indicators are useful for comparison over time
  within the same library.
• Comparison between libraries is possible but
  careful interpretation is required.

79
Comparability of performance indicator data

• Purposes of using library performance indicators
• self-diagnosis (within the same library) e.g.
  comparisons of one years performance with
  another
• comparison accross different libraries in full
  recognition of the limitations of such
  comparisons and with respect for each librarys
• mission, goals and objectives
• resources
• user groups
• governance / funding structure
• procedures

80
Limitations

• It is impossible to achieve optimum scores
  simultaneously on all performance indicators (eg.
  user satisfaction vs. expenditure per user)
• The scores on performance indicators must be
  interpreted in the light of what the library
  intends to accomplish, not simply in terms of
  optimizing scores on particular indicators.
• Degree of accuracy
• Sampling errors
• Subjective aspects of the measuring process
• Inadequate time or resource for measuring process
• Users skills vs. library performance perception
• Linking resources to services, management
  approaches, staff skills etc.

81
What factors would you consider when making
decision for registering your library to the
library project for comparable measures? - group
work





82
Consider

• Relevancy to to the services being analysed
• Strategic purpose of the measurement activity
• Staff involved in the measurement process and
  their awarness of the fact that measurment
  process is part of their normal flow of work
• Accuracy
• Reliability
• Consistency over time
• Practical aspect

83
Success conditions

• Simplicity versus complexity (goals!)
• The effort involved in data collection versus the
  expected outcomes
• Possibility to integrate data collection with the
  already carried work
• Trends are (usually) more important than figures
• For some measures (e.g. time taken to handle
  information queries) setting aside certain
  periods throughout the year and extrapolating
  findings to a full year
• Representative samples
• Do not collect data for the sake of it

84
Thank you!