Program Evaluation

1
Program Evaluation

• CHIC
• 2008
• Ann Dozier, RN, PhD

2

• What is a program?
• Program Intervention

3

• what is program evaluation?
• did the program DO
• what we SAID it was
• going to do?

4
why evaluate?

• evaluation can tell us
• to continue
• to change
• to stop

5
types of evaluation

• formative
• process
• outcome
• impact
• economic
• these are sequential and build on each other
  i.e. cannot do impact evaluation without evidence
  of positive outcome evaluation

6
types of evaluation

• formative evaluation
• conducted before the evaluation is established
• helps define problem to be addressed by the
  program
• determines opportunities for intervention
• better define target population
• may provide a baseline for later comparison
• involves target community/population

7
process evaluation

• assessment of the activities being undertaken
• cannot assess program outcome without process
  evaluation
• most commonly undertaken type of evaluation
  because it is perceived as the easiest
• done by program staff (not outside evaluator)
• weekly, monthly, quarterly

8
process evaluation

• assessment of the activities being undertaken by
  the program staff or staff of partner
  organizations
• represents a count or tally of what the program
  is actually doing
• (typically used to compare what it said it was
  going to do)

9
outcome evaluation

• measures overall effectiveness of the program
  based on a specific objective
• ex. was there a desired change among
  participants?
• may include changes in behavior, attitudes or
  knowledge
• commonly are annual measures
• represents if your activities are cumulatively
  having an effect

10
types of evaluation

• impact evaluation
• measures a longer term program impact in target
  population
• economic evaluation
• compares programs with similar long term program
  impacts to determine which one is more cost
  effective

11
(No Transcript)
12
Program Plan
Planned Intervention
Expected Outcome or Result
13
Program Implementation
Planned Program
Actual Program
Expected Outcome or Result
Process Evaluation Compares actual with planned
14
Implementation
Actual Program
Planned Program
Actual Outcome or Result
Outcome Monitoring Shows actual results
15
Implementation
Actual Program
Planned Program
Actual Outcome or Result
Expected Outcome or Result
Outcome Evaluation Shows how actual results
compared with expected results
16
Program Plan
Actual Program
Actual Outcome or Result
Outcome Evaluation Show what resulted from the
intervention
17
Implementation
Actual Program
Shorter Term Outcome or Result
Longer Term Outcome or Result
Outcome Evaluation Show shorter term outcomes
that result from the intervention.
18
Program Evaluation

• Why Evaluate?
• Verify if the program is really running as
  originally planned.
• Understand, verify or enhance the outcomes of the
  program.
• Identify strengths, weaknesses and areas for
  program improvement

19

• Evaluation is a measure of
• accountability
• progress
• program learning
• Evaluation is provided to
• agency head/directors
• funder
• community at large
• target population

20
The Health Policy Wheel
Adapted from Richmond-Kotelchuck Health Policy
Model, Oxford Textbook of Public Health 1991
21
project development process

• problem identification
• needs assessment
• data gathering (qualitative/quantitative)
• confirm/clarify problem
• prioritizing establish goal/purpose
• develop program objectives, interventions/activiti
  es and corresponding budget/personnel needs and
  evaluation

22
steps in the evaluation process

• planning the evaluation
• defining purpose
• involving stakeholders
• formulating goals/objectives/indicators
• what/who to target
• (what do you want to change or count or assess)
• what to measure to assess change

23
(No Transcript)
24
steps in the evaluation process

• selecting a design/method
• how and when to measure (timeframe, tools,
  approach)
• conducting the evaluation
• piloting/phased in implementation
• monitoring established processes
• gathering existing/new data

25
steps in the evaluation process

• data analysis and interpretation
• assessing data quality, changes in measures
• provide feedback
• conduct analyses
• creating information from data

26
Program Evaluation Decisions

• What will be evaluated?
• what is the program and its context?
• What aspects will be considered when judging
  program performance?
• What standards (i.e., type or level of
  performance) will determine if the program is to
  be considered successful?

Framework for Program Evaluation in Public
Health. MMWR. September 17, 1999 / 48(RR11)1-40
27
Program Evaluation Decisions

• What evidence will indicate how the program has
  performed?
• what information is relevant
• Using the evidence, what conclusions can we make?
  
• compare with the selected standards?
• value, quality, utility, and effectiveness
• How will the lessons learned from the evaluation
  be used to improve the program?

Framework for Program Evaluation in Public
Health. MMWR. September 17, 1999 / 48(RR11)1-40
28

• so why is evaluation not done?

29
evaluation barriers

• too hard poor planning
• no time mandated
• no money threatening
• we know we are doing a good job
• no one reads the reports anyway

30
evaluation barriers

• unrealistic targets/goals
• objectives not linked to program
• not meaningful to the project
• measures poorly defined not useful

31
overcoming barriers

• include evaluation from outset during project
  planning/proposal writing
• make it useful to program/agency not just to meet
  funder requirements
• involved key stakeholders from outset
• establish realistic goals/objectives and time
  frames

32
overcoming barriers

• establish clear and appropriate link between
  objectives/goals and program activities
• establish appropriate well defined evaluation
  measures
• provide training and/or technical assistance

33
overcoming barriers

• build in feedback loops to program (quality
  improvement) establish baseline
• build into existing work processes
• use/adapt existing methods/tools

34
formulating goals and objectives

• so what is a goal?
• statement of a desired long term impact of your
  program
• in the ideal world
• if everything works perfectly
• broad in nature
• ambitious

35
goals and objectives

• give some examples
• reduce perinatal HIV transmission
• reduce injection drug use
• increase high school graduation rates
• decrease number of pregnant teenagers
• reduce unintentional injury among children

36
goals and objectives

• objectives
• impact
• outcome
• process
• note depending on the funding agency impact and
  outcome may be switched

37
goals and objectives

• objectives
• impact
• more specific than the goal
• reasonable yet challenging
• long term
• typically a change in the rate or occurrence of
  some condition or situation

38
goals and objectives

• objectives
• outcome several for each impact objective
• very specific
• reasonable
• short term (1 year or less)
• program outcome
• measurable change

39
goals and objectives

• objectives
• process usually conducted by program staff
• very specific
• reasonable
• short term (weeks/months/quarters)
• activity

40
Goals and Objectives
goal
impact objective
outcome objective
process objective
41
goals and objectives

• elements of objectives
• when timeline for achieving
• who target group
• where geographic location
• what action or intended
• activity
• how much extent or degree of
• achievement

42
Goals and Objectives

• Elements Of Objectives
• By ________, __________,
• When? Who?
• _________, will _____________
• Where? What?
• by _______________?
• How Much
• (suggested format only)

43
SMART Objectives

• Specific - Be precise about what you are going to
  achieve
• Measurable - Quantify your objectives
• Achievable - Are you attempting too much?
• Realistic - Do you have the resources to make the
  objective happen (personnel, equipment,
  materials)?
• Timed - State when you will achieve the objective
  (within a month? By February 2010?)

44
examples of process

• products (e.g. media, curriculum)
• processes (e.g. referral agreements,
  dissemination of information, systems of care)
• services (e.g. classes held, visits, assessments
  completed)
• people (e.g. staff, consultants)

45
examples of process

• program or site level
• what are the components of your program?
• interventions
• casefinding, breastfeeding, nutrition education
• outreach
• target audience
• multiple sites, hours of operation

46
examples of outputs (process)

• Staff level
• Number of staff providing services?
• type of staff/service, hours available
• Size of staff caseloads? duration of involvement?
• what types/frequencies of encounters
  (number/type)?
• what types of clients are workers seeing?

47
examples of outputs (process)

• client level
• who are you serving? changed overtime? how does
  this compare with program objective or goal?
• demographics, bf status
• what services are they using?
• program enrollment, services/encounters
• how frequently/encounter type?
• duration of participation

48
measuring outcomes

• program/client level
• measure client participation
• enrollment, participation, retention
• measure client changes
• knowledge, attitudes, beliefs
• awareness, readiness, intentions
• skills
• competence

49
measuring outcomes

• program/client level
• self-efficacy
• confidence
• behaviors
• appointment adherence
• referral follow-up
• risk reduction practices

50
Summary Steps in Program Evaluation

• engage stakeholders
• describe the program
• focus the evaluation design
• gather credible evidence
• justify conclusion
• ensure use and share lessons learned

51
Summary

• Evaluation
• provides information on program design for
  existing programs and/or new program
• describes/confirms what was actually implemented
• demonstrates changes
• provides feedback
• may be seen as not value added