Impact Evaluation Designs for Male Circumcision

1
Impact Evaluation Designs for Male Circumcision

• Sandi McCoy
• University of California, Berkeley

Male Circumcision Evaluation Workshop and
Operations Meeting
2
Our Objective

• Estimate the CAUSAL effect (impact) of
• intervention P (male circumcision)
• on
• outcome Y (HIV incidence)

3
Our Objective

• Estimate the CAUSAL effect (impact) of
• intervention P (male circumcision)
• on
• outcome Y (HIV incidence)
• Since we can never actually know what would have
  happened, comparison groups allow us to estimate
  the counterfactual

4
Evaluation Designs for MC IE
Study Design
Cluster
Stepped wedge
Selective promotion DoseResponse
5
Evaluation Designs for MC IE
Study Design
Cluster
Stepped wedge
Selective promotion DoseResponse
Not everyone has access to the intervention at
the same time (supply variation)
The program is available to everyone (universal
access or already rolled out)
6
Cluster Evaluation Designs

• Unit of analysis is a group (e.g., communities,
  districts)
• Usually prospective

Intervention
Comparison
7
Cluster Evaluation Designs

• Case Study Progresa/Oportunidades Program
• National anti-poverty program in Mexico
• Eligibility based on poverty index
• Cash transfers
• conditional on school and health care attendance
• 506 communities
• 320 randomly allocated to receive the program
• 185 randomly allocated to serve as controls
• Program evaluated for effects on health and
  welfare

8
Stepped Wedge or Phased-In
Clusters Time Period
Clusters 1
1
2
3
4
5
6
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
9
Stepped Wedge or Phased-In
Clusters Time Period Time Period
Clusters 1 2
1 Program
2 Program
3
4
5
6
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
10
Stepped Wedge or Phased-In
Clusters Time Period Time Period Time Period
Clusters 1 2 3
1 Program Program
2 Program Program
3 Program
4 Program
5
6
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
11
Stepped Wedge or Phased-In
Clusters Time Period Time Period Time Period Time Period
Clusters 1 2 3 4
1 Program Program Program
2 Program Program Program
3 Program Program
4 Program Program
5 Program
6 Program
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
12
Stepped Wedge or Phased-In

• Case Study Rwanda Pay-for-Performance
• Performance based health care financing
• Increase quantity quality of health services
  provided
• Increase health worker motivation
• Financial incentives to providers to see more
  patients and provide higher quality of care
• Phased rollout at the district level
• 8 randomly allocated to receive the program
  immediately
• 8 randomly allocated to receive the program
  later

13
Selective Promotion

• Common scenarios
• National program with universal eligibility
• Voluntary inscription in program
• Comparing enrolled to not enrolled introduces
  selection bias
• One solution provide additional promotion,
  encouragement or incentives to a sub-sample
• Information
• Encouragement (small gift or prize)
• Transport

14
Selective Promotion
Universal eligibility
15
Selective Promotion
Universal eligibility
Selectively promote
NoPromotion
Promotion
16
Selective Promotion
Enrollment
Universal eligibility
Selectively promote
NoPromotion
Promotion
17
Selective Promotion
Not Encouraged 4 incidence
Never Enroll
Enroll if Encouraged
Always Enroll
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
18
Selective Promotion
Not Encouraged 4 incidence Encouraged 3.5 incidence
Never Enroll
Enroll if Encouraged
Always Enroll
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
19
Selective Promotion
Not Encouraged 4 incidence Encouraged 3.5 incidence ? Effect 0.5
Never Enroll
Enroll if Encouraged
Always Enroll
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
20
Selective Promotion
Not Encouraged 4 incidence Encouraged 3.5 incidence ? Effect 0.5 POPULATIONIMPACT 2 incidence reduction
Never Enroll
Enroll if Encouraged
Always Enroll
Brown CA, Lilford RJ. BMC Medical Research
Methodology, 2006.
21
Selective Promotion

• Necessary conditions
• Promoted and non-promoted groups are comparable
• Promotion not correlated with population
  characteristics
• Guaranteed by randomization
• Promoted group has higher enrollment in the
  program
• Promotion does not affect outcomes directly

22
Selective Promotion

• Case Study Malawi VCT
• Respondents in rural Malawi were offered a free
  door-to-door HIV test
• Some were given randomly assigned vouchers
  between zero and three dollars, redeemable upon
  obtaining their results at a nearby VCT center

23
Dose Response Evaluations

• Suitable when a program is already in place
  everywhere
• Examine differences in exposures (doses) or
  intensity across program areas
• Compare the impact of the program across varying
  levels of program intensity

Hypothetical map of program implementation levels
24
Dose Response Evaluations

• Example for MC
• All clinics in a region offer MC, but their
  capacity is limited and there are queues
• Some towns are visited by mobile clinics that
  help the fixed clinic rapidly increase MC
  coverage

25
Design Variations for MC IE
Study Design Allocation Method Allocation Method Allocation Method
Study Design Randomization Matching Enrolled vs. not Enrolled
Cluster ? ? ?
Stepped wedge ? ? ?
Selective promotion DoseResponse ? ? ? ? ? ?
26
Random Allocation

• Each unit has the same probability of selection
• for who receives the benefit, or
• who receives the benefit first
• Helps obtain comparability between those who did
  and did not receive the intervention
• On observed and unobserved factors
• Ensures transparency and fairness

27
Unit of Randomization

• Individuals, groups, communities, districts, etc

28
Matching

• Pick a comparison group that matches the
  treatment group based on similarities in observed
  characteristics

29
Matching
Region A - Treatment
Region B - Comparison
30
Matching
Region A - Treatment
Region B - Comparison
31
Matching

• Matching helps control for observable
  heterogeneity
• Cannot control for factors that are unobserved
• Matching can be done at baseline (more efficient)
  OR in the analysis

32
Enrolled versus Not Enrolled
Consider a school-based pregnancy prevention
program
10 schools in the district are asked if they
would like to participate
33
Enrolled versus Not Enrolled


5 schools decline participation
No intervention
5 schools elect to participate in the program
Pregnancy Prevention Program
34
Enrolled versus Not Enrolled


No intervention
Pregnancy rate 3 per 100 student years
2 per 100 student years
Pregnancy Prevention Program
35
Enrolled versus Not Enrolled
Schools in the program had fewer adolescent
pregnancies Can we attribute this difference to
the program?


No intervention
Pregnancy rate 3 per 100 student years
2 per 100 student years
Pregnancy Prevention Program
36
Enrolled versus Not Enrolled
Observed effect might be due to differences in
unobservable factors which led to differential
selection into the program(selection bias)


No intervention
Pregnancy rate 3 per 100 student years
2 per 100 student years
Pregnancy Prevention Program
37
Enrolled versus Not Enrolled

• This selection method compares apples to
  oranges
• The reason for not enrolling might be correlated
  with the outcome
• You can statistically control for observed
  factors
• But you cannot control for factors that are
  unobserved
• Estimated impact erroneously mixes the effect of
  different factors

38
Choosing Your Methods

• Two decisions to decide the design

Study Design Allocation Method Allocation Method Allocation Method
Study Design Randomization Matching Enrolled vs. not Enrolled
Cluster ? ? ?
Stepped wedge ? ? ?
Selective promotion DoseEffect ? ? ? ? ? ?
39
Choosing Your Methods

• Identify the best possible design given the
  context
• Best design fewest risks for error
• Have we controlled for everything?
• Internal validity
• Is the result valid for everyone?
• External validity
• Local versus global treatment effect

40
Consider Randomization First

• Minimizes selection bias
• Balances known and unknown confounders
• Most efficient (smaller Ns)
• Simpler analyses
• Transparency
• Decision makers understand (and believe) the
  results

41
Choosing Your Methods

• To identify an IE design for your program,
  consider
• Prospective/retrospective
• Eligibility rules
• Roll-out plan (pipeline)
• Is universe of eligibles larger than available
  resources at a given point in time?
• Who controls implementation?
• Budget and capacity constraints?
• Excess demand for program?
• Eligibility criteria?
• Geographic targeting?

42
Thank you
43
Dose Response Evaluations

• Case Study Global Fund Evaluation
• 18 countries categorized on magnitude of Global
  Fund disbursements, duration of programming

Country Global Fund HIV Grants Global Fund HIV Grants
Country Funds (US M) Time elapsed (yrs)
Benin 25 4.2
Cambodia 50 4.6
Ethiopia 270 3.9
Malawi 129 4.6
Mozambique 50 3.4
Rwanda 58 3.4
Source The Global Fund 5 Year Evaluation