Evaluation Done Right

1
Evaluation Done Right

• Dick Krueger and Mike Schmitt

2
Evaluation Done Right Workshop

• Program Success
• Evaluation
• Market Research
• End-of-Meeting Evaluation
• After-the-Meeting Evaluation

3
How do You Define Program Success?
4
How do You Define Program Success?

• Attendance?
• Income?
• Net income?
• Sponsors?
• Quotas/Benchmarks?

5
Program Success

• Creating and showing program impact within a
  financially sustainable model. It is desired
  (but not guaranteed) that the audience is
  satisfied, the stakeholders are pleased, and the
  program team is rewarded.

6
Program (Business) Plan Components
Impact
Evaluation
Delivery
Logistics
Marketing
Budget
Curriculum
Feasibility
Vision/Objectives
Needs Assessment
Start
Finish
7
Program (Business) Plan Components
Impact
Evaluation
Delivery
Logistics
Marketing
Budget
Curriculum
Feasibility
Vision/Objectives
Needs Assessment
Start
Finish
8
Evaluation Done Right Workshop

• Program Success
• Evaluation

9

• What are your initial perceptions or thoughts
  about Evaluation?

10
Why is Evaluation Important?

• Documents accountability of public resources
• Helps leverage additional resources
• Helps improve programs
• Allows for feedback from customers
• Helps us administer and coordinate programs
• Gives feedback to staff and volunteers

11
Why is Evaluation Important?

• Can provide data for personal performance factors
• Provides data and documentation for scholarly
  projects

12
BUT, Evaluation Requires Resources

• Evaluate programs when you are proud of them
• Evaluate programs when something isnt working
• Evaluate programs of concern to the public
• Evaluate before investments are made
• Evaluate to discover effective efficient
  delivery
• Evaluate to learn about your target audience

13
Whats Happening Now?

• Every evaluation is different
• Comparison is difficult or impossible
• Our customer base is changing
• The University Extension environments are
  changing
• Our competition is changing
• Information technology is changing

14
Future Trend

• More consistency in tracking results
• Discover and document longer term consequences
• More attention to market research
• Learning about our customers

15
Evaluation Done Right Workshop

• Program Success
• Evaluation
• Market Research

16

• What IS market research?

17
What makes Market Research Difficult?

• It is different from academic research
• We are unsure of our target audience
• We listen to the wrong people
• Important questions are difficult to ask
• Results arent always dependable

18
Key Steps in Market Research

• Get a clear fix on your target audience
• Use secondary data
• Learn from past experience and history

19
Key Steps in Market Research

• Get a clear fix on your target audience
• Use secondary data
• Learn from past experience and history
• Listen to your target audience
• Remember that people are complex
• Find out what draws people to participate

20
Key Steps in Market Research

• Get a clear fix on your target audience
• Use secondary data
• Learn from past experience and history
• Listen to your target audience
• Remember that people are complex
• Find out what draws people to participate
• Find out the barriers to participation
• Seek local verification
• Learn about the competition

21
Answers from Market Research

• What events/products are desired or economically
  or politically valued?
• Which audiences should we target?
• What educational medium is most desired and
  effective?
• Evaluation then can refine market research

22
Program (Business) Plan Components
Impact
Evaluation
Delivery
Logistics
Marketing
Budget
Curriculum
Feasibility
Vision/Objectives
Needs Assessment
Start
Finish
23
Evaluation vs. Market Research

• Measures
• Impact
• Refines
• Curriculum
• Logistics
• Marketing
• Delivery
• Program-based

• Describes
• Needs
• Feasibility
• Influences
• Logistics
• Marketing
• Budget
• Curriculum
• Delivery
• No Program needed

24
Evaluation Done Right Workshop

• Program Success
• Evaluation
• Market Research
• End-of-Meeting Evaluation

25
End-of-Meeting Strategy

• Just a few questions
• Easy to tabulate on Excel
• Emphasis on learner outcomes
• Consistent questions
• Optional questions as needed

26
Key Questions on End-of-Meeting Form

• Did you learn more?
• Did you acquire more skill?
• Do you feel you can use the ideas or skills?
• Were you satisfied?
• Are you satisfied with Extension in the past
  year?
• Age, ZIP, Gender

27
Evaluation Done Right Workshop

• Program Success
• Evaluation
• Market Research
• End-of-Meeting Evaluation
• After-the-Meeting Evaluation

28
Tips for Evaluating After the Event

• Decide how long to wait
• Decide how to get back in touch
• Decide on a sample
• Be able to identify your target audience
• Let people know youll be back in touch
• Consider retrospective or reflective questions

29
Reflective Questions or Retrospective Pre-Test
Questions

• Indicate knowledge, skill, or use NOW
• Indicate knowledge, skill, or use BEFORE the
  event
• Use a 5 or 10 point scale and anchor the ends
• Subtract the before from the after to
  calculate a gain score
• Ask the extent to which the program caused the
  gain or difference

30
Example of a Reflective Question

• My knowledge of pesticide safety.

• Today
• An exceptional amount
• Very much
• Much
• Some
• Little
• None

• One Year Ago
• An exceptional amount
• Very much
• Much
• Some
• Little
• None

31
Example of a Reflective Question

• 1A. If there is a difference between one year ago
  and today, to what extent was the change a result
  of the Pesticide Safety Session?

• Completely
• To a great extent
• To a moderate extent
• To a slight extent
• Not at all

32
Example of a Reflective Question

• 1. Rate your knowledge of pesticide safety on
  the scale below. Place the letter B on the
  scale where you would rate your knowledge BEFORE
  you attended the Pesticide Safety Session. Place
  the letter N on the scale where you would rate
  your knowledge NOW.

1 2 3 4 5
6 7 8 9
10
Low High
33
Evaluation Done Right

• Discussion Questions

34

• What key point stuck with you?
• What might your team do with this information?
• What help or information do you now need?