Impact Measurement: introduction and issues

1
Impact Measurement introduction and issues

• Professor Peter Brophy

SCONUL Autumn Conference 2006
2
Perspectives on library evaluation

• Collection
• Building
• Service
• Access
• Embedding
• Competencies

3
Its never easy!

• What you measure - you distort
• Just because you cannot measure something does
  not mean that it is not important
• David Gurteen

4
The three Es

• Economy

5
Evaluation - foci
We have 2 million records
As a result of our service, humankind has beaten
malaria
We have the technology to transform data into
information-rich products
26 students have read this paper
We have delivered one million documents
Increasing difficulty
Increasing usefulness
6
Approaching impact - satisfaction

• The customer is always right Selfridges
  motto
• (originally The customer is never wrong)
• A belief that the customer is best placed to
  judge service effectiveness (fitness for purpose
  / conformance to requirements)
• Basis for SERVQUAL. LibQUAL etc.

7
The Kano model
8
Attributes of satisfaction
Brophy after Garvin
9
Attributes of satisfaction
Brophy after Garvin
10
What is Impact?

• An effect on an individual or group which
• may be positive or negative
• may be what was intended or unintentional
• may result in changed
• attitudes
• behaviours
• products
• may be short or long term
• may be critical or trivial
• .and so on
• Expectations together with perceptions of impact
  drive satisfaction

11
Why do we measure impact?

• To assess which services / aspects of service are
  worthwhile to our community

12
Impact on learning

• Concerns Based Adoption Model (CBAM)
• Stages of concern
• How engaged are subjects in the service
  /innovation?
• How much interest do they show?
• Levels of use
• How much use do they make?
• Avoid dichotomy of user / non-user
• Levels of Impact
• Draw together concern and use
• Assess effects on behaviour

13
Levels of Impact
14
Levels of Impact
15
Levels of Impact
16
Impact and Learning

• Surface learning
• Skim, memorise, regurgitate for tests
• Little long-term impact
• Deep learning
• Involved, questioning, interactive, integration
  with own knowledge
• Significant long-term impact
• A habit of critical thinking

17
Measuring Impact

• Who is making judgements?
• the customer?
• the library?
• the government?
• society?
• What does each of them mean by good?
• What does each of them think a library is for?

18
Measuring impact

• Usually necessary to employ a mixed-methods
  approach
• Quantitative data tells us the extent of service
  take-up
• Qualitative data provides the rich picture which
  includes effects and thus impacts

19
Surrogate measures of impact

• How big the library is
• How much it cost
• How many books it contains
• How much they cost
• How many staff you employ
• How much they cost
• How many letters the library staff have after
  their names

20
Qualitative data

• Can appear anecdotal and therefore needs
  careful data collection and dispassionate analysis

Phil worked for several years as a country park
ranger. His job mainly involved manual work but
he wanted to progress into managing a park
himself. He realised that he would need computer
skills in order to develop his employment
potential and decided to take the first step by
attending the taster sessions at Wellingborough
library. Phil attended Introduction to Computers
and Introduction to the Internet sessions. He has
since got a job as the manager of a heritage site
in Essex.
21
Some useful measures

• What the customers say about the service
• How the customers react to threats of withdrawing
  service
• What the customers actually do
• What the customers actually produce

22
Pitfalls of impact measurement

• Loading too much onto measures
• What gets measured gets done
• Some actions may be OBE (overtaken by events)
• What about the opportunity costs what might
  have produced even more impact than this?
• There is a tendency to accentuate the positive .

23
Accentuate the positive?
Advocacy has a habit of catching up with reality!
24
Other issues

• Economic impact
• Current interest in contingent valuation
• Social impact
• Social accounting and audit
• Hidden services with high impact
• Many ICT-based services e.g. middleware, such as
  resolver services

25
And finally If you think you're too small to
have an impact, try going to bed with a
mosquito. Anita Roddick