Usage Statistics Further Developments

1
Usage Statistics Further Developments

• James Mouw
• The University of Chicago Library
• mouw_at_uchicago.edu

2
Or, are we just putting lipstick on insert your
favorite print resource
3
The brief

• Usage statistics, if gathered and analyzed
  correctly, can provide a wealth of information to
  librarians and publishers for a variety of
  purposes collection development, new products,
  marketing, customer support, etc.  What more can
  be expected?  How will such data be used in the
  future to help organizations make informed
  decisions, especially as the first generation of
  born-digital information users take on roles of
  faculty, researchers, and scholars?  What does
  the future hold?

4
Chris Anderson The end of Theory The data deluge
makes scientific method obsolete. Wired, 6/23/08

• All models are wrong, but some are useful
• So proclaimed statistician George Box 30 years
  ago, and he was right. But what choice did we
  have? Only models, from cosmological equations to
  theories of human behavior, seemed to be able to
  consistently, if imperfectly, explain the world
  around us. Until now.
• Today companies like Google, which have grown up
  in an era of massively abundant data, don't have
  to settle for wrong models. Indeed, they don't
  have to settle for models at all.

5
The reality
6
A means, not an end

• Building on the foundation
• COUNTER counts
• SUSHI delivers
• We assess
• A culture that values assessment
• Outcomes based assessment
• Focus on quality
• Trying to understand our scholars
• Trying to capture our added value

7
A culture of Assessment

• Assessment directors being hired
• Accountability
• Return on Investment
• Quantification

8
Outcomes Based Assessment

• Old measures largely focused on inputs
• Many still do
• Focus on the user experience
• Usage statistics being used to measure, but
  really only one more step on the path
• Disjuncture between library quality and
  traditional measures

9
Challenges
10
Inputs and outcomes far removed

• Say we add 25 new databases and install new
  interfaces
• How do we measure whether or not this results in
• More PhDs granted
• More faculty publications
• BETTER faculty publications
• Increased student retention
• Better grades
• Happier students
• More Nobel prizes

11
Few benchmarks

• If library A spends 15 bazillion dollars on
  e-stuff and this stuff gets used 30 bazillion
  times a year, is this good or bad?
• Tool A cost 0.6 cents per download, and tool B
  costs 35.00. What does this mean?
• I am approaching 1,000,000 hits per year on
  JSTOR. What is the benchmark?

12
How to count use in a linked environment?
13
Masses of data, few tools

• Sushi
• COUNTER
• Scholarly Stats
• Journal Use Reports (JUR)
• Etc.
• Mostly aimed at better COUNTING, not analysis.

14
More granularity

• Usage log analysis
• Research pathways
• Value of individual tools
• Differences among disciplines
• Draw connections between scholars and resources
  used

15
Alternative publication means

• Institutional (and other) repositories
• PrePrint files
• Self-Publishing
• Many times multiple outlets for the same work
• Resulting in fugitive usage

16
Whos performance are we measuring?

• Institutional focus is on Research, Teaching and
  Learning
• Our focus has been on what are we adding to the
  process need to refocus on our impact on the
  process

17
Things we wish we knew more about

• Faculty publication patterns
• Identification
• Gathering
• Analysis
• Faculty citation patterns
• Same as above
• How are tenure decisions really made

18
Going beyond the sciences

• Concentration has been on STM publications
• Humanists and Social Scientists are different
• Citation count less important
• Scholarly monographic publishing bypassing peer
  review
• Performance is as important as accomplishment
• Fear of knowledge getting loose

19
The need for best practices

• Research into usage benchmarking
• Tools that easily deduplicate stats coming from
  various stages of the same research process
• Assurance that uses are accurately measured and
  comparable
• Means of evaluating various uses that seem the
  same statistically
• How do our resources actually contribute to
  teaching and scholarship