Health Impact Evaluation: An Introduction

1
Health Impact EvaluationAn Introduction

• Temina Madon, PhD
• Executive Director
• Center of Evaluation for Global Action
• University of California, Berkeley

2
What is Impact Evaluation?
Identify a health need or target. Proportion of children under 5 sleeping under ITN is lt 6 and target is 60. Health worker absenteeism is reducing patients use of clinics (unpredictable access to care is a deterrent)
What must be done to meet needs or reach targets? Increase access to and proper use of ITNs Decrease health worker absenteeism
What interventions or services could be used? Cost-free distribution of ITNs through schools, or door-to-door distribution. Incentive-based payments to health workers conditioned on health coverage or on attendance.
Which is the most robust program to use? Pilot a few programs and measure their impacts. Is program A able to reliably produce desired results? What about program B? Use treatment and control groups to strip away alternative explanations for the results.
3
Why Impact Evaluation?

• If you need evidence that a program works
• Does the program actually improve health
  outcomes?
• Does it reach the people in need?
• Accountability to civil society
• Accountability to funders
• Ability to target a limited budget
• If you want to improve the program over time
• Results-based management
• Demonstrate cost-effectiveness
• If you want to scale up but need proof of concept

4
Objectives of this Workshop

• Understand how IE is related to ME
• Identify questions suitable for impact evaluation
• Determine which indicators and designs to use
• Understand impact evaluation methods
• Develop an impact evaluation concept note for an
  intervention of your choice

5
ME vs. IE
6
ME vs. IE

• Are deworming treatments being delivered as
  planned?
• Does school-based delivery of deworming
  medication increase school attendance?
• What is the correlation between access to health
  clinics and the proportion of children under 5
  receiving routine vaccinations?
• Do house-to-house immunizations lead to an
  increase in the proportion of children under 5
  who are immunized, relative to the level in
  communities with annual vaccination campaigns?
  Which costs less per child vaccinated?

• ME
• IE
• ME
• IE

7
ME vs. IE

• ME can often be started well after the program
  has been designed and implemented
• IE generally needs to be considered at the outset
  of a programs implementation or scale-up,
  because you need to build the evaluation into the
  design of the program

8
ME vs. IE

• ME allows you to guess how the program may have
  hurt or helped people.
• IE allows you to measure the programs effects
  directly. You will rule out every other
  explanation for the observed effects.

9
What to Evaluate?

• A health service program
• ITN distribution at pre-natal clinics
• Oral rehydration packets and education for new
  mothers
• Deworming medicines in schools
• Health systems project
• Health insurance plans
• Clinic staff retention
• Local or federal health policy
• Seatbelts law
• Tobacco tarriffs

• Behavior Change
• Patient adherence to treatment
• Improved care provided by health care provider
• Community-wide adoption of prevention practices
• Services Systems
• Quality of Services
• Cost-Effectiveness
• Health Outcomes
• Maternal mortality
• Child survival

10
When to use IE?

• ME can be used to fine tune a programs design,
  but IE will
• Tell you whether or not the program is worth the
  money (Does it work?)
• Guide results-based management
• Improve communication with the public civil
  society