The ARL EMetrics Project Measures for Electronic Resources

1
The ARL E-Metrics ProjectMeasures for Electronic
Resources

• ACRL/NEC
• Information Technology Interest Group
• May 17, 2002
• Brinley Franklin
• Director, University of Connecticut Libraries

2
In the Beginning...

• ARL Libraries all of a sudden found themselves
  spending more than 100 annually on electronic
  services and wanted to know who was using them,
  if the expenditures were justified by use
  statistics, and what was the value derived from
  our investment in electronic resources
• In January 1999, the ARL Statistics and
  Measurement and Leadership Committees held a
  retreat to consider new measures for research
  libraries that were output, not input related
• Five New Measures projects evolved from that
  retreat (1) Learning and Research Outcomes (2)
  Measurement of Service Quality (LibQual) (3)
  Cost Studies (4) Interlibrary Loan/Document
  Delivery and (5) Measures for Networked
  Statistics and Electronic Resources (E-Metrics)

3
The E-Metrics Project is Born

• Representatives from 36 ARL Libraries attended a
  retreat in February 2000.
• Rush Miller (Pittsburgh) and Sherrie Schmidt
  (Arizona State) agreed to serve as project
  co-chairs
• Dr. Charles McClure, Francis Eppes Professor and
  Director of the Information Management Use and
  Policy Institute at Florida State Universitys
  School of Information Studies was hired as
  project consultant
• Four areas were included in the project scope
  (1) Study of users and uses (2) Cost and benefit
  analyses (3) Study of staff impact and needs and
  (4) Engaging information providers in a
  discussion of their usage data provided to
  libraries

4
24 Participating ARL Libraries

• Pennsylvania
• Penn State
• Pittsburgh
• Purdue
• Southern California
• Texas AM
• Virginia Tech
• Western Ontario
• Wisconsin
• Yale
• Library of Congress
• New York Public Library

• Alberta
• Arizona State
• Auburn
• Chicago
• Connecticut
• Cornell
• Illinois-Chicago
• Manitoba
• Maryland
• Massachusetts
• Nebraska
• Notre Dame

5
Measuring Electronic Usage - Some Challenges

• Lack of a clear and consistent definition of data
  elements
• Vendors supply the data
• Vendors count different things different ways
• Consortium membership can skew individual library
  usage totals
• Data elements are shifting as services evolve
• Many libraries are not staffed to coordinate
  various data provided

6
The E-Metrics Project (May 2000 - December 2001)

• Phase One Report (delivered in November 2000)
  inventoried the state of the art within
  participating ARL libraries for measuring
  electronic information resources and services
  using questionnaires and site visits. A Working
  Group on Database Vendor Statistics was organized
  and met with 11 major vendors
• Academic Press/Ideal netLibrary
• Elsevier/Science Direct SilverPlatter
• Lexis-Nexis EBSCO
• Ovid JSTOR
• Bell Howell OCLC/FirstSearch
• Gale Group

7
The E-Metrics Project (May 2000 - December 2001)

• Phase Two - Defining and Testing Data Elements
• Other similar projects monitored included
• European Commission Equinox Project
• Publishing and Library Solutions Committee (PALS)
  Working Group on Online Vendor Usage Statistics
  (UK)
• ICOLC Guidelines for Statistical Measures of
  Usage of Web-based Indexed, Abstracted, and
  Full-Text Resources
• National Commission on Libraries and Information
  Science (NCLIS) project to standardize online
  database usage statistics
• Institute of Museum and Library Services (IMLS)
  project to develop national network online
  statistics for public libraries
• Council on Library and Information Resources
  (CLIR) report by Judy Luther on Network
  Statistics
• NISO Forum on Performance Measures and Statistics

8
The E-Metrics Project (May 2000 - December 2001)

• Phase Two - Defining and Testing Data Elements
• 18 Measures were field tested in three
  categories
• (1) information content - including the number of
  full-text journals or reference sources to which
  a library subscribes, virtual visits to the
  librarys electronic resources and the percentage
  of monographs represented by electronic books
• (2) information services - including measuring
  the usage of library digital collections and the
  percentage of reference and other transactions
  that are digitally based
• (3) technical infrastructure - cost of digital
  collections along with support costs,
  expenditures for electronic journals, etc.

9
E-Metrics Recommended Statistics

• Patron Accessible Electronic Resources
• R1 Number of electronic full-text journals
• R2 Number of electronic reference sources
• R3 Number of electronic books
• Use of Networked Resources and Services
• U1 Number of electronic reference transactions
• U2 Number of logins (sessions) to electronic
  databases
• U3 Number of queries (searches) in electronic
  databases
• U4 Items requested in electronic databases
• U5 Virtual visits to librarys website and catalog

10
E-Metrics Recommended Statistics (continued)

• Expenditures for Networked Resources and Related
  Infrastructure
• C1 Cost of electronic full-text journals
• C2 Cost of electronic reference sources
• C3 Cost of electronic books
• C4 Library expenditures for bibliographic
  utilities, networks, and consortia
• C5 External expenditures for bibliographic
  utilities, networks, and consortia
• Library Digitization Activities
• D1 Size of library digital collection
• D2 Use of library digital collection
• D3 Cost of digital collection construction and
  management

11
The E-Metrics Project (May 2000 - December 2001)

• Phase Two Final Report and a Proposal for Phase
  Three- Measuring Outcomes was delivered to ARL in
  December 2001 and presented at ALA-Midwinter in
  January, 2002
• ARL will not be engaging the Information
  Management Use and Policy Institute at the
  Florida State University School of Information
  Studies to continue work on Phase Three,

12
Post Mortem - Whats Next for E-Metrics?

• Interested ARL libraries will begin collecting
  the proposed data and statistics in FY 2003. The
  24 E-Metrics Libraries that funded the original
  project participate at no additional cost - new
  participants will be asked to contribute 2000.
  Gordon Fretwell, University of Massachusetts, and
  former editor of ARL Statistics, will support
  this data collecting effort.
• We hope that by acting in concert we will exert
  pressure on publishers to be responsive to our
  requests for more uniform and consistent usage
  data.
• We also hope that libraries will continue to
  shift their focus from measuring inputs and
  traditional services to measuring all library
  services, including electronic services, and to
  measuring outputs and outcomes.

13
ARL E-Metrics Project Homepage

• http//www.arl.org/stats/newmeas/emetrics/index.ht
  ml