Locating, Hiring, and Managing an Evaluator

1
Locating, Hiring, and Managing an Evaluator
Wayne Harding, Ed.M., Ph.D. Social Science
Research Evaluation, Inc. 21-C Cambridge
Street Burlington, Massachusetts
01803 781-270-6613 wharding_at_ssre.org
Based on a training developed by CSAPs
Northeast CAPT ? 2000 Educational Development
Center, Inc.
2
Workshop Objectives

• Increase understanding of the value of evaluating
  programs (reasons for doing and not doing an
  evaluation)
• Learn ways to find and hire an evaluator
• Increase understanding of how to work with an
  evaluator
• Identify ways to reduce the costs of an
  evaluation
• Stimulate interest in evaluating programs

3
What is Evaluation?
Evaluation is a systematic attempt to determine
whether (or to what degree) a program (policy,
procedure, activity) produces its stated outcomes
(short and/or long term).
4
Four Evaluation Designs
1. One-group posttest design X posttest 2.
One-group pretest-posttest design pretest
X posttest 3. Pretest-posttest comparison-group
design pretest X posttest pretest O posttest 4
. Pretest-posttest control-group
design pretest X posttest random
assignment pretest O posttest
5
What Is a Quark?

• The sound a duck makes.
• A subatomic particle
• A dean made by Toyota
• A candy bar
• A unit of currency in Peru

6
Five Issues

• More complex evaluation designs are more
  difficult to implement.
• There are many more evaluation designs than the
  four we just reviewed.
• No evaluation design is immune from threats to
  its validity.
• The list of possible complications is long.
• More complex evaluations increase certainty about
  the results but they also cost more.

7
Reasons for Doing Evaluation Whats In It for
Me?

• To learn how well the program has worked (outcome
  evaluation)
• To learn if the program was implemented as
  planned (process evaluation)
• To meet funders requirement
• To make the program more appealing to funders
• To test/improve the efficacy or efficiency of
  program components
• To address political controversy or public concern

8
Reasons for Not Doing Evaluation

• Channels scarce resources away from program
  activities
• Might get bad news about program effectiveness
• Difficulties in locating and managing an evaluator

9
How to Find an Evaluator

• Other programs
• Evaluation literature (published and unpublished
  studies)
• Funding agencies
• Contract research firms
• Colleges and universities
• Conference presenters

10
What to Ask About and Assess

• Biases about the type of program to be evaluated
• Experience evaluating similar programs
• Willingness to have program staff participate in
  the evaluation
• Who controls and owns the data, findings, and
  publications?
• How will the evaluator balance responsibilities
  to program staff, funder, and other costs?
• Budget, including indirect costs
• Can you and will others understand the
  evaluators writing and presentations
• References

11
Content of the Evaluation Plan (1 of 2)

• The evaluation questions (process and outcome)
  that will be answered
• The design of the evaluation (e.g., whether a
  control group will be used)
• The major independent and dependent variables and
  how they will be measured
• The types of data to be collected and how they
  will be collected
• The sampling plan
• Data analysis (exploratory and likely or known
  procedures)

12
Content of the Evaluation Plan (2 of 2)

• Protection of human subjects (e.g., informed
  consent, confidentiality)
• Reporting (when feedback will occur and in what
  forms)
• Staffing for the study
• Timetable for major evaluation activities and due
  dates for all deliverables
• Budget for the evaluation

13
Traditional vs. Collaborative Evaluation
COLLABORATIVE Done with the program Evaluator
operates in concert with the program Evaluator
advises Program staff are participants in
planning and carrying out the study Evaluator
interacts regularly with programs staff and other
stakeholders
TRADITIONAL Done to the program Evaluator
operates apart from the program Evaluator
decides Evaluator retrieves information from
program staff as needed to plan and carry out the
study Evaluator interacts (relatively
infrequently) through the program director
14
The Collaborative Model

• The primary mechanism is an evaluation
• committee made up of
• Evaluator
• Program staff
• Other stakeholders (e.g., in a school-based
  program, stakeholders may include curriculum
  designers, school-board members, teachers,
  parents)

15
Advantages of Collaborative Evaluation

• Promotes common expectations
• Staffs personal involvement improves quality
• Lowers cost
• Flexible model
• Helps staff better understand the ingredients of
  good research
• Increases likelihood that results will be applied
• Improves sensitivity to political and cultural
  issues
• Allows program to track evaluators performance

16
Disadvantages of Collaborative Evaluation

• Potential disagreements about staff versus
  evaluator responsibilities
• Resistance by program staff
• Takes longer to complete evaluation tasks
• May be more difficult to find suitable evaluator
• Staff participation may bias the study

17
Strategies for Minimizing Cost

• Search for a qualified but inexpensive evaluator.
• Search for an evaluator who can get funding for
  the evaluation.
• Search for an evaluator who has other than
  monetary interests.
• Search for an evaluator experienced in evaluating
  programs like yours.
• Start small limit the focus of the evaluation.
• Use a collaborative model.
• Ask the evaluator to price components of the
  study.
• Estimate cost before specifying an amount in an
  RFP.