Monitoring and Evaluation

1
Monitoring and Evaluation

• Charles Katende PhD.
• Director of Monitoring, Evaluation and Research
• JHPIEGO (An affiliate of John Hopkins University)

2
Session Objective

• To increase participants understanding of the
  concepts used in designing ME Frameworks and
  Plans
• To build participants competence in designing
  Program ME Plans

3
Expected Results

• At the end of the sessions participants will know
  about Program frameworks, ME frameworks and the
  difference between the two frameworks
• Participants will be able to identify and select
  appropriate indicators for a program.
• Participants will be able to produce a program
  monitoring and evaluation framework.

4
Introduction

• Write the health problem addressed by a major
  public health program in your country
• Write at up to three specific objectives of a
  public health program that addresses the above
  mentioned health problem.
• Write down two indicators the program mentioned
  above uses to monitor it progress or performance
  towards its objectives.

5
What is Program Monitoring, Evaluation?

• Monitoring is the routine process of data
  collection and measurement of progress toward
  program objectives.

• Evaluation is the use of social research methods
  to systematically investigate a achievement of a
  programs results

6
Key Questions

• What is the purpose of carrying out ME
• Who needs, uses ME Information
• Who carries out ME?
• How is ME carried out?
• When should ME be carried out?

7
What is the purpose.?

• Improve program implementation
• Data on program progress and implementation
• Improve program management and decision making
• Inform future programming
• Inform stakeholders
• Accountability (donors, beneficiaries)
• Advocacy

8
Who needs, uses ME Information?

• Managers
• Donors
• Governments
• Technocrats
• Donors
• Governments
• Communities
• Beneficiaries

• To Improve program implementation
• To Inform and improve future programs
• Inform stakeholders

9
Who conducts ME.?

• Program implementer
• Stakeholders
• Beneficiary
• Remember ..
• ME Technical skills

Participatory process
10
How to carry out ME? Key Features

• Program Framework Analyze and systematically lay
  out program elements
• Identify key elements to monitor and evaluate.
• Determine and describe the measures to be used
  for monitoring and evaluation
• Develop ME Framework and action plans, including
  data collection and analysis, reporting and
  dissemination of findings.

11
Program Framework

• What do you know about your program.?

12
Program Framework

• Systematic lay out of the program elements and
  path showing what the program plans to
• do ..achieve!

13
Program Framework

• Based on a theoretical, empirical model, or
  general understanding

Public health Problem
Population, system level factors that cause the
public health problem
Action/interventions that can change the factors
and ultimately alleviate/eliminate the problem
14
Results Framework
Improved Health Status
Impact
Improved (Sustained) Use of Key Health Services
and Practices/Behaviors
Strategic Objective
Increased quality of
Increased availability/ access to
Improved social / policy environment
Intermediate Result
Strategies
Strategies
Strategies
Strategies (Sub IR)
15
Example Result Framework for a Family Planning
Project
           
GOAL REDUCED FERTILITY
SO Increased FP use and improved FP/RH
practices
Increased knowledge of, improved attitudes
toward, and acceptance of key services and
behavior
Increased quality of FP counseling and services
for
Increased availability/access to FP/RH
Improved social and policy environment for FP
                                                 
                     
                                                 
                     
Strategies

•  Increase availability of educational materials
  at clinic and community level
• Community mobilization (using PRA and PDI)
  including men
• Implement mass media strategy
• Mobilize opinion leaders at national and local
  level

• Design/ implement supportive supervision System
• Train service providers (in-service and
  preservice in FP counseling and management of
  side effects
• Remodel clinic to allow for privacy
• Design and implement quality improvement program

• Strengthen logistics management
• Mobilize private sector providers
• Mobilize CHWs/CBDs
• Encourage socially marketed pills

Advocate for community based distribution of
pills Promote addition of Depo injections to EPI
outreach strategy Pilot social marketing of pills
16
Basic Logic Model
17
(No Transcript)
18
(No Transcript)
19
Case 1 To decrease maternal mortality, a 10-year
program plan to improve to train midwifes to
Delivery and ANC services at health facilities,
and to train and deploy CHWs to increase the
communitys awareness about, and use of the
improved services at the health facilities.
Case 2 To reduce high fertility, a 5-year
program plans to work with the Government to
change policies in order to allow and promote use
of modern family planning methods, train family
planning providers to provide better FP services,
and to launch public campaigns that promote
family planning methods. Case 3 To reduce HIV
infection among adolescents, a five-year program
plans to implement income generation activities
for the youth, provide and promote universal
secondary education, and build adolescent-friendly
reproductive health service delivery points.
20
Exercise

• Identify and state is the Public Health problem
  implied in the case study.
• What are population level factors will the
  program target to change in order to alleviate
  the public health problem
• Prepare a Program Framework for the scase study

21
Monitoring and Evaluations Framework
22
ME Questions

• Monitoring questions
• What is being done?
• By whom?
• Target population?
• When?
• How much?
• How often?
• Additional outputs?
• Resources used? (Staff, funds, materials, etc.)

23
ME Questions

• Evaluation Questions?
• Is the content of the intervention or the
  activity being delivered as planned?
• Does the content of the intervention or the
  activity reflect the requisite standards?
• Have the intervention achieved the expected
  results?

24
What do we need to answer these
questions?INDICATORS to take measurements.
25
Indicators Definition

• Markers that help to measure change by showing
  progress towards meeting objectives
• Observable, measurable, and agreed upon as valid
  markers of a less well-defined concept or
  objective
• Indicators differ from objectives in that they
  address specific criteria that will be used to
  judge the success of the project or program.

See comment for examples
26
Type and Level of Each Indicator

• Type
• Input/Process (Monitoring)
• Outcome / Impact (Evaluation)
• Level
• Global level
• Country level
• Program level

27
Exercise Group work

• Use your case study and identify at least two
  indicators for program monitoring and two
  indicators for program evaluation.

28
What Is a Good Indicator?

• Valid Measures the effect it is supposed to
  measure
• Reliable Gives same result if measured in the
  same way
• Precise Is operationally defined so people are
  clear about what they are measuring
• Timely Can be measured at an interval that is
  appropriate to the level of change expected
• Comparable Can be compared across different
  target groups or project approaches

29
Criteria for Indicator Selection

• Consistent with project designmeasure the
  desired result
• Usefulcontributes to project design, management,
  and evaluation
• Available
• Affordable

30
Standard Indicators

• Where possible, a project should select standard
  indicators.
• They have been tested for validity and
  reliability.
• They allow comparison between projects or sites.
• They tend to be available for SOs and some IRs.

31
How Many Indicators?

• Choose at least one or two indicators per
  intermediate result, as well as the SO for
  evaluation purposes.
• Choose one or two indicators per result for
  program monitoring.
• Choose indicators that may be able to cover
  more than one element.
• For management, think about basic activities that
  you need to monitor to judge if you are
  implementing activities as planned include
  indicators that help you make decisions.

32
Exercise Group work

• Refer to the indicators you selected..
• Were the good indicators ?
• Did you select a minimum number recommended given
  the type and size of your program?

33
ME Framework
34
Sample ME Framework
Preventing Post Partum Hemorrhage Increase
Active Management of the Third Stage of Labor
Result Indicator Definition Data source Collection Method Freq-uency Respons-ible Party
Active Management of the Third Stage of labor increased Proportion of trained clinicians performing AMTSL to standard of trained midwives performing all steps of AMTSL on all patients/ AMTSL observa-tion checklist Clinical observation Annual Zambia JHPIEGO staff
See comment for examples
35
ME Plan

• The plan is a managerial tool that specifies the
  schedule, resources, responsibilities, for your
  ME activities (data collection, data quality
  control, reporting, dissemination and use of
  data)
• Note
• The plan should specify the time points when
  evaluations will be carried out, for example
  Midterm, and End term.
• Outcome/Impact evaluation is reserved for large
  longer term programs that can make impact at
  public health status level.
• Your plan should include activities to monitor
  and evaluate the implementation, as planned, of
  the ME plan.

36
Source CDC. Global AIDS program monitoring and
evaluation (ME) field guide
37
Question

• If funding for your case study program was cut
  off and the program closed in two years. What
  changes would you make to you ME Plan?

38
Sources of Information

• http//www.cpc.unc.edu/measure
• http//www.unaids.org/DocOrder/OrderForm.aspx
• http//www.fhi.org/en/Publications/index.htm

39
THE END