Evaluation of the State Library of North Carolina

1
NCknows Evaluation Overview
Jeffrey Pomerantz, Lili Luo School of Information
Library Science UNC Chapel Hill ltpomerantz,
luogt_at_unc.edu
Charles McClure School of Information
Studies Florida State University cmcclure_at_lis.fsu.
edu
Tapping the vast reservoir of human knowledge
--Louis Round Wilson, founder, 1931
2
Evaluation question
Is collaborative virtual reference an effective
way to meet the information needs of North
Carolinians?
3
Secondary evaluation questions
What is required of a library that wishes to
offer virtual reference? What is the value added
if different types of institutions work
together? What is the impact on libraries that
provide virtual reference service? Is virtual
reference expandable to the whole state? How will
this project increase our knowledge of effective
organizational models? How can the quality of the
reference service provided be measured? Can staff
from different types of libraries provide quality
reference service to users from other types of
libraries? How will the project further greater
use of existing resources such as NC LIVE? What
partnering or leveraging opportunities exist?
4
Stakeholders
NCknows users The individual participating
libraries The entire collaborative effort The
State Library of North Carolina Evaluation from
all of these perspectives
5
Data collection methods

1. Service statistical analysis
2. Chat sessions peer review of transcripts
3. Patrons exit surveys follow-up phone
   interviews
4. Librarians phone interviews

6
Statistical analysis
What patterns are emerging in the volume of
questions received by NCknows? Usage over
time NCknows librarians vs. 24/7 staff Net asker
/ Net answerer
7
Sessions per month
8
Sessions per day
9
Sessions per hour weekdays
10
Sessions per hour weekends
11
Net asker / Net answerer
12
Transcript peer review
What is the quality of reference service being
provided to users by NCknows? Overall quality of
sessions NCknows librarians vs. 24/7
staff Academic vs. public librarians
13
NCknows vs. 24/7Evidence of user satisfaction
of responses of responses
NCknows librarian 24/7 staff
Yes 32.0 17.6
Indirect evidence 32.3 35.9
No 3.0 2.3
Indeterminate 32.7 44.3
14
Academic vs. Public librariansComplete and
correct answer
of responses of responses of responses of responses
Via academic library Via academic library Via public library Via public library
Academic librarian Public librarian Academic librarian Public librarian
Complete correct 40.0 62.0 43.1 69.6
Incomplete but correct 36.4 22.5 22.4 23.2
Incomplete and incorrect 3.6 1.4 3.4 2.9
Incorrect 0.0 0.0 5.2 0.0
No answer provided 20.0 14.1 25.9 4.3
15
Patron exit surveys Follow-up interviews
What are the demographics of the user
population? How do users find out about
NCknows? What is the users level of satisfaction
with the reference service being provided by
NCknows? What motivates users to use NCknows? How
do users use the information provided to them?
16
Role
17
Answer completeness
18
Motivation for the question
For a work-related task (49) business-related
and school-related (50/50 split) For Personal
Reasons (28) Known-item search (23)
19
Use of the information provided
Use patron had used the information provided and
found it useful (61) Partial use patron had
used the information provided and had found it
partially useful, or had partially used the
information provided (24) No use patron had not
used the information provided at all (15).
20
Discovery of NCknows
Teacher/professor (10) Search engine
(20) Library materials (70)
21
Phone interviews with librarians
How has involvement in the NCknows service
impacted the participating libraries and
librarians?
22
Training
Most useful Patron-librarian role playing Most
in need of review Co-browsing
23
Policies Procedures
Scheduling Handling email follow-ups Quality
control
24
Tech support

• Support
• NCknows good
• 24/7 ok

• Infrastructure
• Adequate or better in university large public
  libraries
• Sometimes less than adequate in community college
  small public libraries

25
Thoughts on chat reference

• Inferior to desk reference
• Lacks non-verbal cues
• Conveys less information in more time
• More difficult to conduct an interview

• Good for
• Quick answers to well-defined questions

Will continue to develop evolve as an aspect of
traditional reference.
26
Additional future work
Cost/benefit analysis Sustainability Scalability
to the entire state? Situational and contextual
factors unique to specific libraries that affect
quality of chat reference MISs databases to
relate reference statistics to other library
statistics Longitudinal data
27
Key evaluation issues
Understanding the importance of
evaluation Ongoing funding/support for
evaluation a culture of assessment How will
evaluation data be used? Quality of data both
the data reported here, and in other data
collection activities
28
The big picture
The need to have ongoing evaluation
data Importance of a statewide initiative in
digital reference Understanding impacts and
applications The context of digital reference
efforts elsewhere Ultimately the question is Do
the benefits and outcomes outweigh the
costs? Congratulations!
29
Reports online
ils.unc.edu/jpom/ncknows/
30
THANKS!
Questions or Comments? Jeffrey Pomerantz, Lili
Luo School of Information and Library Science UNC
Chapel Hill ltpomerantz, luogt_at_unc.edu Charles
McClure School of Information Studies Florida
State University cmcclure_at_lis.fsu.edu