National Solidarity Program Randomized Impact Evaluation Methodology

1
National Solidarity Program Randomized Impact
Evaluation Methodology Baseline Survey
Andrew Beath (Department of Government, Harvard
University) Fotini Christia (Public Policy,
Harvard University) Ruben Enikolopov (Department
of Economics, Harvard University) Shahim Ahmad
Kabuli (World Bank)
2
The Big Picture

• Evaluation Imperative
• Volume of development assistance is limited
• Rigorous evaluation can improve efficacy of
  resource allocation, both between within
  programs
• Important where accumulated knowledge is limited
  and/or program success is critical
• What type of programs should be funded?
• Research Imperative
• Macro-research concludes institutions rule
• But . . . little is known about efficacy of
  institutional change
• Questions remain about how to best deliver
  development in isolated and/or insecure
  environments
• How does development work?

3
Roadmap

• Overview of NSP Program
• CDD Program in Afghanistan
• Goals Hypotheses
• Assess program sub-program effects for donors,
  GoA NSP
• Sample Selection
• 500 villages in 10 districts in 6 provinces in W,
  N, C, E Afghanistan
• Methodology
• NSP randomly assigned to 250 of 500 evaluation
  communities
• Two election and sub-project selection forms
  randomly assigned to 250
• Methods of Data Collection
• 2 year longitudinal survey structure interim
  program monitoring
• Interim Baseline Survey Data

4
NSP Goals

• Four Core Program Elements
• Create CDC through secret-ballot election
• Size of CDC proportional to size of community
• ½ male, ½ female
• Build Capacity of CDC Members
• Community Development Plan (CDP) is drafted
• Sub-projects are proposed to NSP for financing
• Disburse Block Grants to Fund Sub-Projects
• 200 per household (community max 60,000)
• Expand Role of CDCs
• Form linkages with other agencies and development
  programs
• Governance role of CDCs is yet to be decided

5
NSP History

• NSP-I 2003 2007
• 431 million (63 in block grants)
• 17,300 communities (279 / 398 districts)
• NSP-II 2007
• 811 million (120 million IDA)
• 17,540 communities
• New Districts 74 districts (40 communities /
  district)
• On-Going Districts 69 districts

6
NSP Progress
Disbursements through end-2007
7
IE Motivation

• Opportunity for knowledge creation (academically
  interesting)
• Efficacy of externally-imposed institutional
  change
• Economic impacts of institutional change
• Efficacy of CDD as service delivery mechanism in
  post-conflict, low-capacity environment
• Cost of failure is huge (politically necessary)
• 1.2 billion program
• Development needs are critical
• Volatile political environment
• Evidence of impact is mostly anecdotal
  (innovative)
• Conditions facilitate rigorous evaluation
  (possible)
• Rationing renders randomized allocation of
  program feasible
• Heterogeneity in implementation renders
  experimentation in implementation feasible and
  desirable

8
IE Goals

• Assess impact of program
• Does program change governance structures, gender
  roles etc.?
• Does program improve livelihoods, access to
  essential services etc.?
• Assess effectiveness of alternative strategies
  for implementation
• Build knowledge for development
• Efficacy of externally-imposed institutional
  change
• Economic impacts of institutional change
• Efficacy of CDD as service delivery mechanism in
  post-conflict, low-capacity environment

9
IE Effects
250 Villages
1
250 Villages
NSP
No NSP
Vs.
125 Villages
125 Villages
2
Vs.
Ward CDC Election
At-Large CDC Election
3
125 Villages
125 Villages
Vs.
Referendum
Village Meeting
10
IE STIs

• STI-1 Election
• 125 / 250 villages elect CDC with ward election
• Ensures representation for all wards in village
• 125 / 250 villages elect CDC with at-large
  election
• Allows village to elect desired candidates
• STI-2 Sub-Project Selection
• 125 / 250 select sub-projects by village meeting
• Allows for discussion and debate of best
  alternatives
• 125 / 250 select sub-projects by secret-ballot
  referendum
• Directly democratic

11
IE Outcomes

• Social Economic Welfare
• Consumption
• Production
• Assets
• Capital Markets
• Infrastructure
• Access to Services
• Governance Institutions
• Governance Structures
• Activities of Elites
• Dispute Incidence Resolution
• Community Trust
• Political Participation
• Attitudes towards Authority, Tax etc.

12
IE Hypotheses 1

• NSP ? Social Economic Welfare
• Higher levels of consumption, production, assets
• Improved access to services and infrastructure
• Lowered borrowing for consumption, increased
  borrowing for investment
• NSP ? Governance Institutions
• Increased engagement in community activities and
  improved trust
• Local institutions more representative of
  community and more responsive to community
  preferences
• Improved satisfaction with dispute resolution
  mechanisms
• Women more involved in community governance
• Improved perception of central government (tax)
• Reduced economic exploitation of villagers by
  elites through monopsony arrangements, excessive
  interest etc.

13
IE Hypotheses 2

• CDC Election Type
• Ward elections produce CDCs less reflective of
  existing elites
• At-large elections produce CDCs with higher
  levels of competency
• CDCs selected by at-large elections more likely
  to select projects beneficial to whole community
• Sub-Project Selection Procedure
• Meetings more likely to produce projects captured
  elites
• Referendums will improve community satisfaction
• Contributions to projects will be higher under
  referendum
• Awareness of project will be higher under
  referendum

14
SS 10 Districts

• Number of villages
• Minimum of 65 villages 25 villages for control
  group 40 villages to receive NSP
• Security
• Enumerator teams to spend 1 month in district
• Regional Diversity
• Selected Districts
• Balkh Balkh Baghlan Khost Wa Firing Daykundi
  Sang Takht Ghor Daulina Herat Adraskan,
  Chisht-e Sharif, Farsi, Gulran Nangarhar
  Hisarak, Sherzad
• Participating NGOs
• AfghanAid (UK) CHA (Afg.) InterCooperation
  (Pak.) IRC (U.S.) NPO/RRAA (Afg.) Oxfam (UK)
  People In Need (Czech)

15
SS 10 Districts
Balkh
Khost Wa Firing
Gulran
Chist-e Sharif
Sherzad
Sang Takht
Daulina
Hisarak
Farsi
Adraskan
16
SS 500 Villages

• NGOs issued list of villages in district
• NGOs and authorities select 50 evaluation
  villages
• 50 evaluation villages to be included in baseline
  and follow-up surveys
• 25 of 50 randomly selected to receive NSP
• Necessary to ensure evaluation did not create
  logistical difficulties for participating NGOs
• NGOs and authorities select 15 priority villages
  to receive NSP, but to be excluded from
  evaluation
• Necessary to ensured preferences for targeting
  could be met in the 10 sample districts
• Evaluation team vetted lists of 15 priority
  villages to ensure no overlap with 50 evaluation
  villages

17
M. Randomization

• 25 matched-pairs of villages formed in each
  district using multivariate matching
• Protects integrity of inferences from partial
  non-compliance
• Enables inferences over interaction of program
  effects with underlying conditions (e.g. poorer
  vs. richer communities)
• Treated assigned to one unit in matched-pair
  using random number generator
• Minimal and arbitrary differences between units
  selected to receive program and those not
  selected to receive program
• Identification of program effects is a simple,
  transparent exercise of comparing outcomes in 250
  NSP evaluation communities to 250 non-NSP
  evaluation communities

18
M. TG/CG Balance
19
DC Two Fronts

• 1. Household Focus Group Surveys
• 500 evaluation communities (½ NSP, ½ non-NSP)
  in 10 districts
• 2. Monitoring Exercises
• 120 NSP evaluation communities in 10 survey
  districts per FP schedule

20
DC Surveys

• 2 Year longitudinal study of 500 villages
• Baseline Survey in August / September 2007
• 1st Follow-Up Survey Autumn 2008
• 2nd Follow-Up Survey Summer 2009
• Four Survey Instruments
• Male Head-of-Household Questionnaire (10 /
  village)
• Male Focus Group (Shura) Questionnaire (6 9
  participants / village)
• Female Focus Group Questionnaire (6 9
  participants / village)
• Female Individual Questionnaire (6 9 / village)

21
DC Monitoring

• Collect information documenting STI
  implementation
• Assess community involvement and CDC performance
  during various phases of NSP
• Exercises
• Election Monitoring (½ CDC elections)
• Post-Vote Interviews, Polling Station Reports,
  Election Reports Vote Tallies
• Sub-Project Selection Procedure Monitoring (½
  SPSPs)
• Post-Vote / Post-Meeting Interviews, Polling
  Station Reports, Referendum / Consultation
  Meeting Reports Vote Tallies
• Project Management Monitoring (proposed)
• Project Completion Assessment (proposed)

22
BS Instruments

• Male Head-of-Household Questionnaire
• 10 randomly sampled men individually interviewed
  in each village
• 5,007 total interviewees
• Male Focus Group (Shura) Questionnaire
• 6 9 members of village shura or village leaders
  interviewed together
• 3,962 total interviewees
• Female Focus Group Questionnaire
• 6 9 females interviewed together
• 3,407 total interviewees
• Female Individual Questionnaire
• Participants of Female Focus Group interviewed
  individually
• 3,515 total interviewees

23
BS Education
Level of education of household-heads
No Education
Madrassa
Primary School
gt Primary School
24
BS Services
Access to electricity by district . . .
25
BS Food Security
26
BS Finance
Household-heads who have taken loan in past year
No
Yes
27
BS Finance
Reason for taking loan . . .
28
BS Gender Roles
Women on village shura acceptable?
No
Yes
29
BS Devt Priorities
30
BS Devt Priorities
31
BS Governance
How happy are you with the work of your village
council?
32
BS Governance
Have your village leaders done anything you dont
agree with?
33
BS Governance
In whose interests do the following people act?
34
BS Governance
Should people pay taxes?
35
BS Governance
Who should people pay taxes to?
36
BS Governance
Unjust act by village leaders? (women)
37
BS Governance
Unjust act by village leaders (women)
38
BS Governance
39
BS Governance
Those who have asked someone else in village to
collect money on their behalf
40
BS Governance
Villagers attending shura meetings
41
BS Happiness
42
BS NSP Awareness
43
EM Election
What is the purpose of the election?
44
EM Election
Who organized the election?
45
EM Election
What was the most important consideration in
deciding who to vote for?
46
EM Election
Can you vote for anyone in the village or just
people who live in your ward?
47
EM CDC
What is the purpose of the CDC?
48
EM CDC
Is the CDC a part of the government?