Evaluation 101 Planning an Evaluation

1
Evaluation 101Planning an Evaluation

• Paul Pope Extension Program Specialist
  Evaluation Extension Education 979-845-7202
  ppope_at_tamu.edu

2
Outline of Course Topics

• Concept of evaluation
• Why we conduct evaluations
• Checklist of major decisions (planning an
  evaluation)
• Resources

3
What is Evaluation?

• Systematic approach to assess the design,
  implementation, and utility of programs.
• Utility usefulness, worth, or quality.

4
Why Do We Evaluate?

• To determine if a program achieved its
  objectives (effectiveness).
• To know how lives were changed (impact).
• To modify programs that are not working according
  to plan or take advantage of something that is
  working exceptionally well (improvement).
• To help develop new efforts.
• Report to stakeholders (accountability).

5
Planning an Evaluation

• What ?
• When ?
• How ?
• Who ?

6
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation? (output or outcome)
• If an outcome evaluation, what client change
  will you be measuring?
• 2. When to collect data? (evaluation strategies).
  
• 3. How will you collect data? (data collection
  methods)
• 4. Who to include in the evaluation? (census or
  sample).

7
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation? (output or outcome)
• If an outcome evaluation, what client change
  will you be measuring?
• 2. When to collect data? (evaluation strategies).
  
• 3. How will you collect data? (data collection
  methods)
• 4. Who to include in the evaluation? (census or
  sample).

8
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation?
• - output
• - outcome

9
1. What type of evaluation? - Output -

• measures satisfaction general impressions
• does not measure client change
• can use customer satisfaction survey

10
1. What type of evaluation? - Output -

• Can find current customer satisfaction form and
  cover sheet at
• http//extensioneducation.tamu.edu/forms.htm

11
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation?
• - output
• - outcome

12
1. What Type of Evaluation? - Outcome -

• Primary purpose is to determine whether a program
  had its intended effect (measure client change)
• Some Extension programs already have evaluations
  built-in (ready to use)
• Can have customer satisfaction section too

13
1. What Type of Evaluation? - Outcome Four
Types of Client Change (Learning vs. Application)

• Application
• Behavior change /
• adoption of best practice or new
• technology

• Learning
• Knowledge
• Skills
• Attitude / Belief

14
1. What Type of Evaluation? - Outcome -

• Your program should have at least one type of
  client change as a goal.
• Your measures should have validity. Want to
  measure the appropriate client change resulting
  from your intervention.
• Dont try to measure behavior change if your
  program was designed solely to increase
  knowledge.

15
1. What Type of Evaluation? - Outcome
Anticipated Timing of Client Change

• Short-term
• Knowledge, attitudes, skills, some behavior
  changes.
• Long-term
• Most behavior changes, economic changes, adoption
  of best practices or new technology.

16
Outcome Evaluation Example (Knowledge gain)
17
Outcome Evaluation Example (Behavior)
18
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation? (output or outcome)
• 2. When to collect data? In other words, what
  evaluation strategy to use? (post-only, pre-post,
  retrospective post).
• 3. How will you collect data? (questionnaire,
  focus group, observation, individual measurement,
  etc.)
• 4. Who to include in the evaluation? (census or
  sample).

19
2. When to Collect Data? (Evaluation Strategies)

• There are four evaluation strategies typically
  used in Extension- Post-only (at and/or after
  conclusion of a program)- Pre-post (before and
  at / after conclusion of a program ID)-
  Retrospective post (at the conclusion of a
  program)
• - Pre-post series (at the conclusion of an
  individual lesson in a series post for
  current less pre for next lesson ID)

20
2. When to Collect Data? (Example of Post-only)
Program
21
2. When to Collect Data? (Example of
Pre-Post)pre program
post
22
2. When to Collect Data? (Example of
Retrospective Post)
Program
23
2. When to Collect Data? (Evaluation Strategies)

• How do you know when to collect data? What
  evaluation strategy to use?
• Answer to a large extent, this is dictated by
  the client change.

24
2. When to Collect Data?

• Customer Satisfaction (output programs)
• Preferred Strategy
• Post only(customer satisfaction form available)

25
2. When to Collect Data? (Matching Evaluation
Strategies with Client Change)

• Knowledge Gained
• Preferred Strategy
• Pre-Post of actual knowledge (immediate and/or
  longer-term)
• 1st Alternative
• Retrospective post (perceived understanding)
• 2nd Alternative
• Post only (perceived understanding)

26
2. When to Collect Data? (Matching Evaluation
Strategies with Client Change)

• Skills Acquisition, Attitudes/Beliefs, Behavior
  Change
• Preferred Strategy
• Pre-Post (immediate and/or longer-term)
• Retrospective post
• 1st Alternative
• Post only

27
2. When to Collect Data? (Matching Evaluation
Strategies with Client Change)

• Behavior Change, Adoption of Best Practices
• or New Technology
• Preferred Strategy
• Pre-Post (longer-term)
• Retrospective post
• Post only

28
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation? (output, outcome,
  needs assessment)
• 2. When to collect data? In other words, what
  evaluation strategy to use? (post-only, pre-post,
  retrospective post).
• 3. How will you collect data? (questionnaire,
  focus group, observation, individual measurement,
  etc.)
• 4. Who to include in the evaluation? (census or
  sample).

29
3. How to Collect Data?

• Surveys/Questionnaires
• Most common method in Extension
• Mail, in person, electronic
• Simple or complex
• Response Rates
• Short, but get what you need

30
3. How to Collect Data?

• Individual measurement
• - blood pressure reading
• - glucose level
• - weight

31
3. How to Collect Data?

• Observation
• Record reviews
• Focus Groups
• Case Studies

32
Planning an EvaluationChecklist of Major
Decisions

• 1. What type of evaluation? (output, outcome,
  needs assessment)
• 2. When to collect data? In other words, what
  evaluation strategy to use? (post-only, pre-post,
  retrospective post).
• 3. How will you collect data? (questionnaire,
  focus group, observation, individual measurement,
  etc.)
• 4. Who to include in the evaluation? (census or
  sample).

33
4. Who to Include an Evaluation? Census vs.
Sample

• Census All participants
• Sample Some portion of participants
• Random select for representative sample
• consider using a sample if 1,000 on list
• can use Excel, SPSS, SAS

34

• Once the four decisions have been made . . .
• . . . you are ready to implement the evaluation.
  Start collecting data!

35
Misc. Topics Types of Data/Evidence

• Quantitative numbers, more objective
• Qualitative words, more subjective
• Good evaluations use both

36
RESOURCES
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
Example Wheat Tour

• An agent conducts a Wheat Tour Day for wheat
  producers in the county. About 50 producers
  attend to hear the agent and specialists present
  information on a variety of topics including the
  advantages of no-till wheat, wheat variety
  demonstration, and wheat weed control. The
  agent wants know participants overall views and
  satisfaction with the event.
• What type of evaluation?
• When to collect data (evaluation strategy)?
• How to collect data?
• Who to include in the evaluation?

42
Example Seatbelt

• Non-compliance with seatbelt laws was identified
  as an issue in the county. In response, an agent
  conducts an extensive passenger safety campaign
  that targets all auto drivers in the county.
  Through newspaper articles, public service
  announcements, and other mass media, drivers are
  reminded of the benefits of using a seatbelt, the
  risks of not using one, and are urged to always
  use a seatbelt. The ultimate aim of the campaign
  was to get more drivers to actually use a
  seatbelt.
• Did the campaign have a positive effect?
• What type of evaluation?
• When to collect data (evaluation strategy)?
• How to collect data?
• Who to include in the evaluation?

43
Example Evaluation

• A specialist wants to increase agents knowledge
  of basic evaluation concepts. To accomplish this,
  the specialist conducts a one-hour educational
  program at the district center. About 40 agents
  participate in the training.
• The specialist wants to know if the program was
  effective in increasing agents knowledge of
  evaluation.
• What type of evaluation?
• When to collect data (evaluation strategy)?
• How to collect data?
• Who to include in the evaluation?

44
Results of Pre-Post (District 5)
45
Implementing an Evaluation

• Develop questionnaire
• Collect data
• Enter data
• Data analysis
• Interpretation / Report

46
Recommended Sections of a Questionnaire

• Program impact and/or customer satisfaction
• Program Improvement
• Demographics