Operational Aspects of Impact Evaluation

1
Operational Aspects of Impact Evaluation

• Arianna Legovini
• Impact Evaluation Cluster, AFTRL

2
Operational guidance Key steps
3. Prepare data collection instruments 4.
Collect baseline
1. Identify policy questions 2. Design evaluation
5. Implement program in treatment areas 6.
Collect follow up
8. Improve program and implement new 7. Analyze
and feedback
3
Operational guidanceKeep it useful!

• Interesting is what the government wants
• Questioning program helps strengthen design
• Provide answers to questions early
• Change program as a result
• Get better results for client and Bank!

4
Keep it relevant!

• 1. Right questions
• 2. Right team
• 3. Develop client capacity

5
Manage trade-offsMethod Data

• Experimental
• Least costly
• Best estimates
• Not always politically feasible
• Non-experimental
• More expensive
• Good second best estimates
• Always feasible

• Administrative data (expanded to control)
• Most cost effective
• Piggy back on ongoing data collection
• Cost effective
• Coordination intensive
• Collect special surveys
• Tailored and timely
• Expensive

6
Budget

• Data collection
• Consultants
• Staff time
• Include baseline costs in PPF
• Include follow up costs in project costs
• Use Trust Funds to finance consultants
• Initiate product IE-code and obtain BB
  supervision funds

Financing
7
ME Operational messages

• Plan ME while planning project
• Build an explicit team
• Develop budget and financing plan
• Influence roll-out to obtain control groups
• Use quantitative public allocation criteria
• Randomization is ethical
• Strengthen monitoring systems