DEVELOPING BESTFIT EVALUATION STRATEGIES

1
DEVELOPING BEST-FIT EVALUATION STRATEGIES

• John T. Snead, Charles R. McClure, John Carlo
  Bertot Paul T. Jaeger
• Information Use Management and Policy Institute
• College of Information, Florida State University
• http//www.ii.fsu.edu

2
Introduction

• Little research to date provides comprehensive
  assistance in determination of what specific
  evaluation strategies serve libraries best
  relative to specific library situational factors
  and contexts, data needs, and a host of other
  considerations.
• With so many evaluation options available, there
  is a need to bridge information-need issues
  (i.e., situational factors, data needs,
  stakeholder questions, etc.) with evaluation
  approaches.
• Understanding information needs and linking these
  needs to evaluation approaches requires
  evaluation strategies capable of providing data
  that library decision makers can use to address
  specific problems.
• Identifying issues and factors that affect
  evaluation will provide researchers and
  practitioners with understanding and guidance in
  the selection of best practice assessment
  techniques that best meet their needs.

3
Best Fit Evaluation Approaches

• Best-fit evaluation strategies involve matching
  the data needs of a library within a specific
  situational context to the evaluation approaches
  that are most appropriate to that particular
  situational context.

4
The Problem to Address

• Identify which evaluation strategies supply the
  best data within specific library organizational
  and situational contexts for use to provide the
  greatest impact to improve library services, or
  enable libraries to better advocate the value of
  libraries to their institutions or the
  communities they serve.

5
Best-Fit Evaluation Strategy Considerations

• What evaluation approaches are available
• Which evaluation approaches might best meet a
  librarys data needs, either library developed or
  imposed by external funders/organizations/etc.
• How to develop an overall evaluation plan that
  makes effective and efficient use of limited
  library resources
• How to implement an evaluation strategy and
• How to use evaluation findings to advocate for
  local library support.

6
Information Need Perspectives

• Stakeholder type (i.e., who is requesting certain
  data)
• Data or information need (i.e., what data are
  necessary or sought) or
• Evaluation approach selected to meet stakeholder
  data/information needs (i.e., outputs evaluation,
  outcomes-based evaluation, etc.).

7
Stakeholder Perspective

• Provide data necessary to
• Answer a range of questions asked by various
  stakeholders groups (e.g., library boards, county
  or city executives, funding agencies, library
  patrons, state library agencies) regarding
  library services and resources or
• Make informed decisions regarding a librarys
  range or availability of services and resources.

8
Data Perspective

• Provide the data necessary to
• Answer a range of questions asked by various
  stakeholders groups (user-centered evaluation
  perspective)
• Make informed decisions regarding a librarys
  range or availability of services and resources
  (library-centered evaluation perspective)
• Demonstrate value and effectiveness of the
  library to the community that it serves
  (community-centered evaluation perspective)
• Frame the perceptions of the library in the local
  political environment (political context-centered
  evaluation perspective) and
• Support the notion of the library as serving as a
  public good (customer-centered evaluation
  perspective).

9
Evaluation Perspective

• Select most appropriate evaluation approach.
  Initial research indicates that most prevalent
  among evaluation strategies currently developed
  and used by researchers include
• Output assessment,
• Performance measures,
• Outcomes assessment,
• Quality assessment, and
• An emerging evaluation known as the Balanced
  Scorecard.

10
Development of Best-Fit Evaluation Strategies

• Library decision makers are often faced with
  difficulties matching their data needs with the
  appropriate evaluation approaches.
• There are many different kinds of evaluation data
  that a library may need and evaluation approaches
  that a library might employ.
• As a result, many libraries struggle with the
  problem of choosing the best evaluation
  approaches to effectively and efficiently
  demonstrate the value they provide.

11
Issues in the Determination of Best-fit Evaluation

• Success with which libraries are currently
  employing a number of different evaluation
  approaches
• Problematic evaluation efforts in libraries
  (i.e., historical but outdated efforts,
  mismatched evaluation efforts to data needs)
• How library situational factors (organizational,
  community, other) affect the successful use (or
  unsuccessful use) of leading evaluation
  approaches
• Library preparedness to engage in evaluation
  efforts and
• Types of evaluations available for use, including
  the data each evaluation approach might produce,
  the strengths and weaknesses of each evaluation
  approach, and potential applications of each
  evaluation approach within varying library
  situational settings and contexts.

12
Factors in Determination of Best-Fit Evaluation

• Purpose of the evaluation
• Type of data needed
• Knowledge and skills of library staff related to
  evaluation, data collection, data analysis, and
  data reporting
• Degree of difficulty associated to
  understanding/implementing a particular
  evaluation approach and
• Organizational and situational factors related to
  the library (such as available resources), its
  community, and its political/governing context.

13
Selection of Best-Fit Evaluation Strategies

• Library decision makers need be able to select
  the best evaluation strategy given the
• Specific program, service, resource use, or other
  item being evaluated
• Situational factors unique to that library and
  its setting
• Evaluation goals to be accomplished
• Motivation for the evaluation
• Availability of various data sources
• Availability of staff and other resources for the
  evaluation
• Diverse populations represented within the
  communities served by the library
• Governance factors
• Extent and availability of library resources to
  support the strategy and
• Intended audience of the evaluation.

14
Best-Fit Evaluation Strategies

• To understand the impacts, benefits, and value of
  library services and resources, library decision
  makers must select evaluation strategies
  appropriate to targeted data needs within
  specific situational contexts.

15
Future Research Directions

• Resource-based Process assessment of the use
  and allocation of resources
• Problem-based Process link specific problems to
  data needs and best-fit evaluation approaches
• Multiple Evaluation Process - need to engage in
  multiple evaluation techniques to yield a
  comprehensive picture
• Unanticipated Factors unplanned resource
  expenditures and assistance (i.e., aftermath of
  hurricanes, other natural disasters, etc.) and
• Other Factors understand impacts of poor
  evaluation, or when no evaluation may be the best
  approach.

16
Conclusions

• Libraries continue to engage in a wide range of
  evaluation efforts to assess services and
  resources provided to the communities libraries
  serve.
• The evaluation environment is increasingly
  complex, and requires knowledge of multiple
  evaluation frameworks, methodologies, data
  analysis techniques, and communication skills.
• The issue is not that libraries face a lack of
  available evaluation approaches.
• The issue is selecting the approach or approaches
  that best meet data needs of a library from the
  many evaluation techniques that exist.

17
Thank You

• Questions/Comments?
• Contact information
• John T. Snead jsnead_at_garnet.acns.fsu.edu
• Charles R. McClure cmcclure_at_ci.fsu.edu
• Information Institute http//www.ii.fsu.edu