Challenges and Achievements of Monitoring

1
Challenges and Achievements of Monitoring
Evaluation of OVC Programs

• Florence Nyangara, Ph.D.
• MEASURE Evaluation/Constella Futures
• Our Valuable Children Seminar Hosted by AVSI
• November 27, 2007
• Washington, DC

2
Overview

• Situation of ME/OVC?
• Overall key challenges special to the ME systems
  for OVC
• Sources of ME challenges
• Challenges specific to E
• Targeted Evaluations (USAID-MEASURE Evaluation)
• Baseline survey (AVSI)
• Achievements to-date (Global local communities)
• Way forward
• Conclusion

3
Background

• Little attention to ME for program improvements
  it is seen as synonymous with Data Reporting as
  a result -
• Most programs are number driven ( OVC or
  caregivers)
• Little focus on progress towards intended program
  objectives
• Minimal use of outcome indicators to monitor
  progress
• Minimal assessment of any intervention even
  qualitatively to find out what is working and/or
  what is not working
• Little documentation or capturing of any useful
  data not asked
• Lack of ME corporation and coordination among
  partners
• Lack consistency in definitions, approaches,
  tools, indicators for similar activities, etc.

4
Key Challenges to Better ME for OVC

• OVC programs involves diverse groups of
  stakeholders (FBO, NGO, CBO, volunteers) informed
  by different disciplines, thus
• Have uneven understanding of ME basics including
  its purpose
• Some with sufficient ME skills and capacities
  and others NOT - small NGOs
• Un-standardized interventions (multi-level and
  multi-dimensional)
• Limited resources (time, expertise, training,
  etc)
• Overwhelming responsibilities one staffs role
  - ME, finance, management, etc.
• Less need and use of data for decision-making
  (DDIU Challenge) because-
• ME data gathered doesnt result into addressing
  problems identified thus irrelevant poor
  quality information produced i.e. financial,
  inputs, outputs.
• ME is imposed by outsiders - seen as an
  obligation

5
Sources of Challenges in ME for OVC
OVC ME BUS
Multidimensional, Multilevel interventions
Several Players
6
DDIU Challenge

• Less use of available data, less demand
• No systems in place to collate and analyze data
  to gain insights about OVC programs
• The focus has been on data collection to report
  rather than information and knowledge generation
• Poor data quality not useful

7
Challenges with Evaluation

• Limited resources (time, funds, expertise, etc)
  and capacity
• Requires innovative data collection approaches
  that combines qualitative and standard methods
  lots of time
• Multi-dimensional difficult to be sure you have
  sufficient questions that capture all key
  dimensions
• Lack of validated survey tools - Numerous
  pre-testing and revisions of survey instruments
  e.g. AVSI survey
• OVC target population problematic - a moving
  target
• Relocation
• Age out (turn 18 out of program)
• Selection criteria varies i.e. different sampling
  frames
• Multiple providers/programs serving same child,
  HH, etc.
• Study Designs challenging settle for less

8
Global Achievements in ME/OVC Programs

• Global Framework to support/care for OVC
• Guide to ME of National response for OVC
  (UNICEF)
• Guide on ME at Program Level by FHI
• Guidelines on construction of national core
  indicators including for OVC programs
• Country ME for OVC framework have been developed
  with common definitions
• Increased coordination of ME across partners,
  government, and other stakeholders within
  countries through OVC/ME Steering sub-committees
• PEPFAR guidance is promoting consistency in
  structuring ME systems
• Indicators for similar activities
• Tools, methods, approaches
• Targeted Evaluation studies MEASURE Evaluation
  CRS
• Programs/communities levels are working to
  improve the content, quality, and consistency of
  ongoing assessment of children
• Child Status Index (CSI)

9
AVSI Study as an Example

• Research approach
• Sampling
• Combining qualitative quantitative
• Validating selection criteria - OVC
  characteristics?
• Longitudinal study -designed and built-in
• What works
• Stakeholders buy-in
• Motivated research team
• Enough resources

10
AVSI Study as an Example - Issues

• Not interviewing children
• Involvement of local researchers (sustainability)
• Limiting the study outcomes to education only
• Think of IGA
• Impact may not be specific to program
  interventions without a comparison group.

11
The guiding Principles to Better ME

• Sustainability recognize its value involve
  locals fully
• Focus on DDIU promote use of program data as a
  decision-making tool not just for accountability
  reporting
• Stakeholder engagement foster their
  participation
• Develop networks to develop tools and share best
  practices
• Flexibility the system should be flexible to
  allow for transparency, improvement and
  accountability.
• Reporting processes should be based on reliable
  data and credible analysis

12
Next Steps

• Cultural shift to good ME of information use
  and reporting Support
• Develop a participatory ME system for reporting
  at all levels with implementation guidelines
• Implement feedback mechanisms to maintain
  interest and ownership of ME stakeholders
• OVC/ME working groups to ensure the exchange and
  sharing of information
• Promote joint development of tools and procedures
  between different donors and simplify ME tools
• Solicit best practices from partners on OVC data
  use and share them

13
Thank YOU
14
MEASURE Evaluation is funded by the U.S. Agency
for International Development (USAID) through
Cooperative Agreement GPO-A-00-03-00003-00 and is
implemented by the Carolina Population Center at
the University of North Carolina in partnership
with Constella Futures, John Snow, Inc., ORC
Macro International, and Tulane
University. Visit us online at
http//www.cpc.unc.edu/measure.