The WSP Global Scaling Up Handwashing Behavior Project

1
The WSP Global Scaling Up Handwashing Behavior
Project
Monitoring and Evaluation

• Six month team meeting May 29, 2007

2
ME Organization

• Impact Evaluation
• Objectives defined primarily by Gates
• IE Team is responsible for technical design, But
• coordination/integration with county programs is
  essential
• Overlaps with project monitoring are significant
• Project Monitoring
• Primarily a tool for project managers
• Managers should have ultimate authority (Is this
  true? Or should it be consultative?) over design,
  resources, strategic design and data collection
• Overlaps with IE should benefit both activities

3
Impact Evaluation

• Objective Learn what works, why and how well
• What are impacts of HW-BC interventions?
• Behavior
• Environment
• Health, social, and economic welfare
• What are the antecedents to behavioral change?
• How did the interventions affect outcomes?
• What is the cost-effectiveness of these
  interventions?
• Share lessons learned with countries, donors and
  other partners.

4
Project Monitoring/MIS

• Project Monitoring system(s) serves the project
  managers.
• It helps managers ensure that the project is
  functioning as it should.
• If done well, it expands, contracts, and
  refocuses at the discretion of project manager.

5
IE Outcomes of Interest

• Intermediate
• Sanitation coverage and open defecation
• Reduced fecal contamination of houses, hands,
  food, water (? - not yet funded country
  permission is not yet clear)
• Final
• Reduce diarrhea
• Reduce parasite loads(?)
• Reduce wasting, stunting and anemia(?)
• Improve cognitive development(?)
• Improve wages and productivity
• Social welfare (security, female schooling, etc.)

6
IE TS-SM Logical Chain of Evidence

• A compelling logical chain of evidence improves
  credibility. We hope to demonstrate
• A plausible behavioral change/ consumer demand
  model (closely linked to the Results Framework)
  through the BCF/model
• Microbiological evidence of infection
  transmission channels (not in our current RF)
• Statistical links between intermediate and final
  outcomes (sanitation ? diarrhea reductions)

7
IE Data Requirements

• Two-round household survey
• Household demographics, economics
• Household handwashing, sanitation and water
• Intervention Process indicators (knowledge, etc.)
• Measures of microbiological contamination
• Health, nutrition and cognitive status
• Employment, wages, productivity
• School enrollment attendance sense of security
• Monthly monitoring
• Diarrhea among children lt5 yrs.
• Community Sanitation
• Handwashing behavior
• Where useful, institution-based disease/absentee
  reporting

8
IE Use of Intermediate Outputs

• The Results Framework should tell us whether the
  intervention worked as expected
• Some program diagnostic information can be
  gathered, but most of these will be available
  only after the project. Usefulness to Gates is
  limited

9
IE Implementation Implications

• Baseline - the baseline survey needs to be in
  place prior to full-scale implementation. But
  can trial implementation be monitored before
  full-scale roll-out?
• Treatments controls - choosing treatment and
  control groups requires
• Buy-in/agreement from field, Govt, stakeholders,
  implementing institution(s), other development
  partners
• Detailed information on roll out, target
  population, unit of intervention (hh / community
  / village / district), number of units to be
  covered, inhabitants per unit, geographic area,
  etc.
• Country-Specific IE Concept notes - the specific
  interventions determine the IE design specific
  implementation details at country level are
  crucial to the design of the country-specific
  concept notes

10
Impact Evaluation Where are we?

• Steps already taken
• Organizational model variations on a common
  theme
• Experimental designs
• Mass-media and EE generally not randomized
• Direct Consumer Contact activities randomized
• Country PIs and supporting teams
• Peru Paul is leading, w/Sebastian Galiani, Alex
  and others
• Senegal no decision yet
• Tanzania Sebastian Martinez will lead
  supporting staff proposed Initial visit
  scheduled
• Vietnam Pascaline Dupas will lead.

11
IE Where are we? (contd)

• Steps to be taken
• Country PIs Senegals needs to be proposed.
  Perus and Vietnams needs to be contracted
  Tanzania done in-house
• Nearly all country supporting teams must be
  identified
• Survey firms
• Tanzania really only one leading candidate
  non-competitive contract may be needed
• Peru already discussing w/survey firms
• Vietnam and Senegal nothing known
• Questionnaire development
• July 9-10 meeting to prepare final x-country
  draft
• Translation/reformatting July/August
• Interviewer training August/September
• Fieldwork starting mid-late September at earliest

12
IE Implications for implementation

• Baseline - the baseline needs to be in place
  previous to implementation in the treatment
  areas. Note that the initial/trial roll-out can
  (and probably should) be implemented in non
  treatment/control areas before the baseline
  survey.
• IE Concept note - the intervention defines the IE
  design implementation details at country level
  are crucial to design country-specific concept
  notes
• Treatments controls - choosing treatment and
  control groups requires
• Buy-in/agreement from field, Govt, stakeholders,
  implementing institution(s), other development
  partners
• Detailed information on roll out, unit of
  intervention (hh / community / village /
  district), number of units to be covered,
  habitants per unit, geographic area, etc

13
Objective of Project Monitoring/MIS

• Determine whether inputs (from RF) are
  implemented as and when planned (according to
  workplan/milestones)
• Identify problems early, when they occur
• Facilitate problem diagnoses and effective
  solutions
• How? Assess, in real time, whether inputs are
  achieving intended outputs
• Do HHs know of interventions?
• Have they heard behavior-change messages?
• Do they intend to change their behavior?
• Is it reasonable to expect that they will?
• Allows for intervention improvements in real time

14
General Model of Project Monitoring
Eval. Review
Verification/QA5 - 25
15
MIS data collection tools

• Contract-based (Tanz Vietnam)
• Build routine reports into field
  activities(strike a balance between informative
  and non-disruptive reports)
• Quality Assurance (QA) required (objective
  unbiased routine reports)
• Public sector institution-based (Peru schools,
  Vietnam MOH facilities)
• Incomplete/biased reporting concerns and
  solutions
• Independent Surveys
• Most costly data collection, but needed for IE
  and some QA

16
MIS Where are we (contd)?

• Program monitoring data collection systems
• Peru School program contract-based community
  DCCs two systems needed
• Tanzania All contract-based?
• Vietnam at least two systems probably needed
  (one for contract-based work one for
  government-based work)
• Senegal?
• Do we need a strategic plan for intensive early
  KAP monitoring?
• All should establish QA Plans

17
MIS tools (cont)

• Other monitoring activities
• QR / AR
• Gates required Frequency quarterly and annually
• Based on country-specific RFs / milestones ? need
  to review / revise (if appropriate) for the 2nd
  QR
• HAMR
• Project required Frequency biweekly
• Crucial for lessons learnt
• Monitor Enabling Environment
• Monitor Implementation Effectiveness

18
MIS tools (cont)

• Other monitoring tools
• Budget Monitoring
• WSP required Frequency currently every QR and
  AR
• New TF structure under discussion only one TF
  per country. Implications ? simpler but calls
  for more rigor to not exceed (/-) 10 limit
  imposed by Gates
• Budget needs to be revised at country and global
  level before submission of 2nd QR (mid June)