OVE

1
OVEs Experience with Impact (Treatment)
Evaluations

• Presentation prepared for DAC, 15th November 2006

2
Policy

• The general evaluative questions proposed by the
  IDBs ex post policy, approved in 2003, are (i)
  the extent to which the development objectives
  of IDB-financed projects have been attained. and
  (ii) the efficiency with which those
  objectives have been attained (para1.1 )
• Policy left for practice sampling, methodology,
  organizational framework, and the forum for the
  presentation of results.
• Note the task is to evaluate already approved
  and/or closed projects ( average project time is
  about six years).

3
Implementation decisions

• Project Sampling Strategy Option random or
  meta-evaluation. Decision meta-evaluation.
• Method and Project types (i) process cum naïve
  or treatment (impact) evaluations. Decision
  Treatment effect evaluations and (ii) projects
  with partial or national coverage. Decision
  partial coverage models
• Organizational. Decisions (I) separate activity
  within the office (ii) evaluations to be
  carried out both in-house and outsourced.Therefore
  (I) hired staff with appropriate expertise and
  (ii) created EVALNET, a register of evaluators
• Forum for presenting results. Decision overall
  report (sent to the Board) with
  background-working papers (discussed in ad hoc
  seminars).

4
Evaluative questions

• what were the problems that the program was
  designed to tackle?
• what was the policy response, i.e. the design
  features of the program? (theory based
  evaluation)
• was the program of a sufficient size given the
  size of the problem(s)?
• were the programs deliverables provided in a
  cost efficient (and cost effective manner)?
• What was the incidence and was the program well
  targeted?
• what was the impact on welfare outcomes of the
  program?and
• what were the benefits relative to the cost of
  the program ?

5
What was the impact on welfare outcomes of the
program?

• To answer the question OVE normally use three
  approaches in the same evaluation
• Naïve evaluation
• Regression based (cross-section and panel)
• Treatment effects

6
Social Investment Fund (naïve evaluations can be
misleading

• Profile
• Social Investment Fund. Panama
• Basic Infrastructure to poor communities
• Data
• Distribution of benefits by municipalities from
  administrative data
• Baseline and results of outcome indicators from
  households surveys 1994-2001
• Technique
• Treatment and comparison group using PSM in
  double difference. The sample included 75
  municipalities.
• Potential to work with a sample of more than 250
  smaller geographic units but household survey was
  not representative at that level
• Results
• Naïve evaluation the program failed. Impact
  evaluation the program succeeded

7
Labor Training Project (positive effects)

• Profile
• Labor Training program Dominican Republic
• Data
• Simple randomization including a follow-up
  survey done at 10-14 months after graduation from
  training
• 786 treated and 563 controls
• Baseline has universe, follow up was a stratified
  random sample (size determined by standard
  formulas)
• Technique
• Estimated average Intention-to-treat on treated
  by simple diff of means, verified with weighted
  diff and regression analysis (no DD b/c faulty
  baseline)
• Results
• Employability, income and health insurance access
  increased. Program succeeded

8
Public Housing Program

• Profile
• Progressive Housing Phase I Chile
• Provision of low cost basic dwellings to poor
  families
• Data
• Household Surveys identified beneficiaries and
  applicants to the specific housing program
• Technique
• Treatment from beneficiaries and comparison from
  applicants using PSM. Single difference from a
  sample of 508 Beneficiaries and 476 applicants
• Results
• Quality of dwellings improved
• Little or not change in other welfare outcome
  indicators.
• Difference between naïve versus impact

9
Costs, benefits, and internal rate of return

• Profile
• Progressive Housing Phase I Chile
• Provision of low cost basic dwellings to poor
  families
• Data
• Household Surveys identified beneficiaries and
  applicants to the specific housing program
• Technique
• The benefits of the program are the additional
  (necessary) household income required to obtain
  equivalent dwelling
• Results
• IRR greater than 18
• Benefits Net present value per solution 1150 US

10
Rural Roads (decay of benefits over time)

• Profile
• Rural Road Peru
• Construction and upgrade of roads in rural areas
• Data
• Specific survey of beneficiaries. Baseline
  collected after program started. Follow-up survey
  3 years after program closed
• Technique
• single difference and double difference
• Results
• Positive impact on income and assets values of
  rural households.
• Decreasing impact for motorized roads not for
  non-motorized roads.

11
National Transfer Fund (dosage and
multi-treatment effects)

• Profile
• National Fund for Regional Development
• Decentralized investment to finance
  infrastructure and productive projects
• Data
• Administrative data for distribution of benefits
  by municipalities
• Baseline and results of outcome indicators from
  households surveys 1994-2001. The sample included
  343 municipalities.
• Technique
• Impact evaluation using PSM in double difference.
  The municipalities grouped by per capita
  investment using cluster analysis.
• Results
• Positive and increasing impact on poverty
  incidence (reduction) on per capita investment
• Not impact on poverty if investment is intensive
  in education
• Greater impact on welfare composite index in
  municipalities with diversified investment

12
SCIENCE AND TECHNOLOGY Research

• Profile
• Science and Technology Chile
• Financing for RD projects
• Data
• All projects that between 1988 and 2004 received
  the financial support of the program and a
  stratified sample of projects submitted to the
  program, which were not financed because they
  ranked below the threshold defined for being
  admitted to the financing.
• 2,936 different research projects (932 financed
  by the FONDECYT and 1704 not financed) 4,959
  publications recoded in the ISI SCI (1873 by
  financed researchers and 3806 by not financed
  researchers).
• Technique
• Discontinuity regression design. The selection
  process drawn by a threshold quality value that
  separates beneficiaries from non-beneficiaries
• Results
• Unsuccessful. FONDECYT has no significant
  positive impact on the scientific production of
  the financed projects.

13
Technology Development Funds

• Profile
• Public grants-credits to firms for innovation
• Data
• Administrative data on firms and firm level
  surveys (OSLO design)
• Technique
• Double difference with propensity score
  matching
• Results
• Generally positive and significant effects
  on employment, and sales, but little evidence of
  effects on patents and total productivity .

14
EXPERIENCE Findings Potable Water

• Positive effect on health outcome (treatment
  less than naïve effect)
• heterogeneity of results important. a regressive
  relationship between treatment effect and income,
  where more educated (and wealthier) households
  did better than less educated (and poorer)
  households
• Ramification for project design projects should
  include or be coordinated with, as a hypothesis
  to be tested, a health education component
  together with potable water expansion.

Impact on infant mortality
15
Balance

• Since 2004 have produced about 23 evaluations
• Cost per evaluation was about 60,000
• But
• Problem of obtaining effective counterparts (in
  Bank and country) to accompany the evaluation
  from beginning to end. Started outreach program
  to obtain formal counterparts in the country, and
  form ad hoc interested specialist for each
  thematic study.
• Mainstream impact evaluations into other
  evaluations of the Office
• Problem of communicating the findings. Started
  producing different reports for different
  audiences for the same evaluations.

16
Still far from the million words of a good picture

• Before

• After

17
Regression Approach

• Panel data
• Cross section data
• Where y is the outcome of interest, D is the
  dummy for participation in the program, V control
  variables