Evaluation Basics

1
Evaluation Basics

• Sandy Bailey, Ph.D., CFLE
• Associate Professor Extension Specialist
• MSU Extension

2
Why do we conduct evaluations?

• Gain knowledge about our program
• Assess the programs strengths and weaknesses
• Look at program improvement
• Examine program outcomes
• Justify the programs existence

3
Two Major Types of Evaluation

• Process or formative evaluations
• look at the program processes
• give knowledge about the program
• look for strengths/limitations of program
• help determine program improvements
• Outcome or summative evaluations
• examine change in participant knowledge/behavior
• justify program existence

4
Where does evaluation fit on the research
continuum?
Evaluation
Applied Research
Basic Research
All have limitations
5
Why do practitioners dread program evaluations?

• Time consuming
• Measure our programs worth
• Worry we will look bad
• Worry it wont really measure what we do
• We dont feel we have the expertise or resources
  to do them

6
When is the best time to do an evaluation?

• Should be part of the program planning
• Should become a continuous part of the program
  rather than an add-on at the end
• When and how to measure program progress or
  outcomes varies according to the design of the
  evaluation

7
The Evaluation Cycle
Plan
Implement/ Operate
Decide
Assess/Evaluate
Specify/ Decode
Compare
Measure
8
How do I get started?The Elements of a Quality
Evaluation(Joint Committee on Standards for
Educational Evaluation, 1994)

• Utility
• Accuracy
• Feasibility
• Propriety

9
Simple to complex The Five-Tiered Model of
Program Evaluation(Jacobs, 1988)

• Pre-implementation
• Accountability
• Program clarification
• Progress toward objectives
• Impact tier
• ( formative evaluation summative evaluation)

10
Logic Model
Inputs
Outputs
Short term/med term/long term
Outcomes
11
Challenges in Evaluation

• Distance
• Practitioner-Researcher relationship
• Adhering to utilization-focused evaluation and
  feminist approach
• Feasibility
• Accuracy
• Competing evaluations

12
Challenges continued

• Wariness from communities that have been mined
  for data
• Over-reliance on written survey forms exclusive
  of other methods
• Language obstacles (e.g., i.e. etc.)
• Forced choiceoften no category for dont know
  or no opinion
• Readability of survey documentscover letter,
  forms, instructions

13
Benefits of Evaluation

• Keep your program
• Report to funding sources
• Data to support additional funding
• Find out what works and what doesnt
• Provides needed program data
• Helps in future planning

14
Evaluation Methods

• Interviews
• Pre and posttests
• Posttests only
• Retrospective pre/posttests

15
Measures

• Develop own
• Use measures already tested
• Surveys
• Interviews (phone or in person)
• Focus groups
• Observations

16
A Case for Retrospective Pre/Posttests

• No need to match pre to post
• Easier to administer
• May show change not captured by regular pre/post
• Research data supports the use
• Easier for participants
• Cost effective

17
Other Considerations

• 10 Steps to Successful Program Evaluation
• Suggestions for Getting Participants to Complete
  and Return Evaluations
• Tips for Writing Survey Questions