Measuring Outcomes: Potential Approaches in Evaluating Public Health

1
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• William M. Sappenfield, MD, MPH
• MCH EPI Program Team Leader
• Division of Reproductive Health, CDC
• Carolyn B. Slack, MS, RN
• Director, MCH Division
• Columbus Health Department

2
Framework for Program Evaluation in Public Health

• Adopted MMWR 48(RR11) Sept 99
• Summarizes the essential elements
• Provides framework
• Clarifies the steps
• Reviews standards
• Addresses misconceptions

3
Essential Shifts in Public Health Planning and
Policy
EFFORT RESULTS RESULTS
ACCOUNTABILITY ACCOUNTABILITY BUDGETS
4
Premises of Reinventing Government

• What gets measured gets done.
• If you don't measure results, you can't tell
  success from failure.
• If you can't see success, you can't reward it.
• If you can't reward success, you're probably
  rewarding failure.
• If you can't see success, you can't learn from
  it.
• If you can't recognize failure, you can't correct
  it.
• If you can demonstrate success, you can win
  public support.

Osborne and Gaebler (1992 chapter 5,
"Results-Oriented Government")
5
Being Effective in Public Health
6
Program Evaluation Steps
Engage Stakeholders
Ensure Use and Share Lessons Learned
Describe the Program
Standards Utility Feasibility Propriety
Accuracy
Focus the Evaluation Design
Justify Conclusions
Gather Credible Evidence
7
Step 1 Engage Stakeholders

• Fostering input, participation, and power-sharing
  among those persons who have an investment
• Those involved in program operations
• Those served or affected by the program
• Primary users of the evaluation

8
Step 2 Describing the Program
Scrutinizing the features of the program being
evaluated, including its purpose and place in a
larger context.

• Need
• Expected effects
• Activities
• Resources

• Stage of development
• Context
• Logic model

9
Columbus Health Department Perinatal Program
Columbus, OH
10
Program Map
Columbus Health Department Perinatal Program
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Waiting List
Referral
Service Plan
F/U Visit
No Show
Outreach
Delivery
F/U Visit
11
Program Map
Columbus Health Department Perinatal Program
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Counseling
F/U Visit
12
Program Map
Columbus Health Department Perinatal Program
Prenatal Care Entry
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Reduced Risk
Counseling
F/U Visit
Reduced Stress
Weight Gain
Quit Rates
13
Program Map
Columbus Health Department Perinatal Program
Prenatal Care Entry
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Reduced Risk
Counseling
F/U Visit
Reduced Stress
Weight Gain
Reduced LBW
Quit Rates
14
Step 3 Focusing the Evaluation Design
Planning in advance where the evaluation is
headed and what steps will be taken iterative
process.

• Purpose
• Gain insight
• Change practice
• Assess impact
• Inquiry affects participants

• Components
• Users
• Uses
• Questions
• Methods
• Agreements

15
Program Map
Columbus Health Department Perinatal Program
Prenatal Care Entry
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Reduced Risk
Counseling
F/U Visit
Reduced Stress
Weight Gain
Reduced LBW
Quit Rates
16
Step 4 Gathering Credible Evidence

• Compiling information that stakeholders perceive
  as trustworthy and relevant for answering their
  questions.
• Indicators
• Sources of evidence
• Quality of information
• Quantity of evidence
• Logistics

17
Step 5 Justifying Conclusions

• Making claims regarding the program that are
  warranted on the basis of the data.
• Stakeholder standards
• Analysis and synthesis of findings
• Interpretation of the facts
• Judgments
• Recommendations

18
Step 6 Ensuring Use and Sharing Lessons
Learned

• Ensuring that stakeholders are aware, findings
  are used in decisions, and participants have a
  beneficial experience.
• Designed for use
• Preparation for findings
• Feedback from all parties
• Follow-up of users
• Dissemination process
• Additional Uses

19
Program Evaluation Steps
Engage Stakeholders
Ensure Use and Share Lessons Learned
Describe the Program
Standards Utility Feasibility Propriety
Accuracy
Focus the Evaluation Design
Justify Conclusions
Gather Credible Evidence
20
Measuring Outcomes Potential Approaches in
Evaluating Public Health
21
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• Understand the Theory of Action
• Choose the Outcome(s) to Measure
• Choose the Standard or Comparison Group

22
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• Understand the Theory of Action
• Choose the Outcome(s) to Measure
• Choose the Standard or Comparison Group

23
What is the Intervention?
We are generally asked to evaluate an outcome of
an undefined program having unspecified
objectives on an often vaguely delineated
recipient group whose level or variety of
pathology is virtually impossible to assess,
either before or after their exposure to the
program.

• Bloom, 1968. The evaluation of primary prevention
  programs in Greenfield and Miller, Comprehensive
  Mental Health The Challenge of Evaluation,
  University of Wisconsin Press.

24
What is the Intervention?
25
What is the Intervention?
What is the Theory of Action?
Domino Approach What happens in the sequence to
make the whole chain of events occur?
26
Program Map
Columbus Health Department Perinatal Program
Prenatal Care Entry
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Reduced Risk
Counseling
F/U Visit
Reduced Stress
Weight Gain
Reduced LBW
Quit Rates
27
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• Understand the Theory of Action
• Choose the Outcome(s) to Measure
• Choose the Standard or Comparison Group

28
Choose the Outcome(s) to Measure Example
Smoking Cessation

• Ask
• Advise
• Assess
• Assist
• Arrange
• Stages of Change
• Cessation Attempts
• Cessation During Pregnancy
• Improved Low Birth Weight

29
Choose the Outcome(s) to Measure

• Relates to the Theory of Action
• Sufficient Evidence
• Measurable
• Accurate and Reliable
• Feasible in a Public Health Setting
• Affordable

30
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• Understand the Theory of Action
• Choose the Outcome(s) to Measure
• Choose the Standard or Comparison Group

31
Choose the Standard or Comparison Group
5 As
32
Choose the Standard or Comparison Group

• Standard
• Internal Comparison
• Randomized
• Systematic
• External Comparison

33
Choose a Comparison GroupInternal
Starting Event
Pretest
Posttest

• Randomize
• Select

• Individual
• Groups

34
Choose a Comparison GroupExternal
Starting Event
Pretest
Posttest
Inter vention
35
Measuring Outcomes Potential Approaches in
Evaluating Public Health

• Understand the Theory of Action
• Choose the Outcome(s) to Measure
• Choose the Standard or Comparison Group

36
CityMatCH Urban MCH Leadership Confluence

• Measuring Outcomes Potential Approaches in
  Evaluating Public Health
• Case Study Show Me the Results!

37
Program Map
Midstate Health Department Perinatal Program
Prenatal Care Entry
Self Referral
Outreach Visit
Pregnancy Test
Risk Assessment
Substance Abuse
Nursing
Medical
Social
Nutrition
Service Plan
Service Plan
F/U Visit
Coordination
Treatment
WIC
Intervention
Counseling
Food Stamps
Delivery
Reduced Risk
Counseling
F/U Visit
Reduced Stress
Weight Gain
Reduced LBW
Quit Rates