Dr' Dallas L' Holmes

1
Begin with the End in Mind Program Planning and
Evaluation

• Dr. Dallas L. Holmes
• Specialist Institutional Research
• Utah State University Extension

2
Program Planning Process
Identify Problem of Issue
Redesign Program based on Evaluation
BUGS Formula
Tell the Story of What Was Important
Assess Needs and Assets
Report Results to Stakeholders
VISION - start with the end in mind
Success markers toward IMPACTS
Write Goals
Assess Outcomes short and long term
SMART
What are your outputs?
Write Measurable Objectives
Choose and Use Interventions
Ask - So What and Who Cares?
Plan Evaluation
3
Avoid Frustration Ask these Questions Before
Beginning Your Program Not After It Has Begun

• What will you evaluate?
• What will you evaluate immediately?
• Less intense skill and knowledge development
• Pre-Post test or Retrospective Tests
• What will you evaluate via follow-up assessment?
• More intense long-term effects, behavior changes,
  skills and knowledge
• Surveys or telephone interviews

4
Key Factors to Planning

• Keep the main thing (Objectives and Goals) the
  main thing.
• Is it worthwhile to evaluate a one-shot brief
  program?
• Two hour rule.
• Evaluate larger efforts, i.e. program series over
  a year or five years, multi-dimensional efforts
  to teach, change behavior which lead to
  measurable IMPACTS.
• Will follow-up Outputs be necessary to achieve
  the desired outcomes? Have you planned for these?

5
Identification of the Problem or Issue

• 1st Clear vision of needs and assets of your
  target
• Create a statement page of the problem or issue.
• 2nd Is there
• clientele consensus
• leader and agency buy in
• 3rd Facilitate public awareness of problem or
  issue
• Foundation of support

• 4th Put together statements to create public
  awareness, seek program and financial support
• Contact stakeholders at beginning of program for
  their support not at the end of the program.

6
Needs Assessment

• Who should be assessed or asked about needs?
• BUGS Formula
• How will assessment be completed?
• focus groups
• interviews
• written or telephone interviews
• self-report surveys
• Review available social and scientific indicators
  surrounding the issue or problem.
• Demographic changes, percentages, increases and
  decreases in

7
Writing Goals

• Long term goals - one to five years to accomplish
• WHO will DO WHAT for WHAT PURPOSE?
• Verbs which usually accompany goals include
• change, promote, offer, increase, decrease, train
  or enable.
• Reported Utah bankruptcies in Utah will decrease
  by 15 in 2000 as a result of the Financial
  Management, Budgeting Awareness program conducted
  by Utah State University Extension

Goals Written on
Macro Level
Micro Level
Individual, family, ranch, farm, club - levels
Community/neighborhood,region/state - levels
8
Writing Objectives - The Basics

• Short Term Objectives - One year or less to
  accomplish
• They Target
• knowledge, attitudes, skills, aspirations, tax
  dollar support, behaviors and other program
  characteristics which will change as a result of
  the intervention.

SMART S - Specific M - Measurable A -
Attainable R- Relevant T- Time-based
The Formula By (DATE) (WHO) will (DO WHAT
IMPLEMENTING/ACTION VERB) for (WHAT OUTCOME)?
9
Evaluation Plan

• Overall purpose for the evaluation, and specific
  questions the evaluation needs to answer. Short
  summary of what the evaluation is to accomplish

• Methods to be used
• evaluation strategy
• mail survey, telephone survey, etc.
• target audience
• from which data is collected
• time frames
• persons responsible
• How Results will be used
• Budgetary Needs

10
Model for Evaluation Planning
Program Goals Integrated Weed Management Program
Outcomes
Specific Information Needed- Indicators
Data Collection method(s)
Date(s) for data collection
Data source(s)
Ecologically based weed management
Nov-Feb statistical collection from
groups Measures for pre-herbicidal June On-going
IPM measures
Reduced herbicide use Utilization of
IPM Utilization of other weed control methods
cultural,biological, GMOc
Data searches USDA statistical baseline Dealer
Coop records Commodity group data records
USDA commodity groups Special interest
groups Commercial applicators Scientists involved
in specialized research
11
Plan Evaluation

• Begin by asking the BIG question
• What outcomes do I hope to show from this
  program?
• Social
• Environmental
• Economic
• Begin with the end in mind!
• Outcomes and Success Markers
• We expect to see
• We would like to see
• We would love to see
• Success Markers Identified actions/behaviors
  which indicate successful accomplishment of the
  outcome

?
1.Who 2.Clause which describes the successful
attainment of identified change 3.Complete
statement with a behavioral intention that
represents a significant attainment for the
person targeted
Outcome 1
SM2
SM3
SM1
12
Assess Outcomes Measuring Impacts

• Techniques to consider include
• Interviews
• Focus Groups
• Post-test only, post-test then pre-test,
  pre-test-post-tests, pre-test-post-test follow-up
  evaluations, experimental-control groups,
  quasi-experimental or comparison groups
• Using Measures that determine changes in (KASA)
• Knowledge
• Attitude
• Skills
• Aspirations
• Measuring for change in Practices
• Interventions -That which has changed peoples
  lives

SEE

• Social
• Economic
• Environmental

13
Reporting Results to Stakeholders

• Remember who the stakeholders are you previously
  identified in the initial program planning.
  REPORTS MAY BE DIFFERENT FOR VARIOUS
  STAKEHOLDERS. Address the stakeholders needs!
• - One or two basic demographics about the problem
• - Your basic objectives/ intention of your
  program
• - What you did
• - What results/outcomes you found. What happened
  as a result of the program?
• Generally no longer than one page, concise,
  organized and clearly presented. Consider an
  executive summary approach.

A Basic Outline to Consider
14
Redesign the Program Based on Evaluative Results

• The most important use of your evaluation results
  is to use the results to redesign your program so
  that it will be more effective.
• ASK
• What did you learn?
• What worked?
• What did not work?
• What can you learn about your approach or
  teaching style?
• What did you learn about your audience?
• What additional services should/could be added?
• Loop back to the beginning phase of program
  planning and apply the learnings to each step in
  the planning process.

1
Redesign Program
2
Program Planning Process
7
3
6
4
5