Monitoring and Evaluation: Concepts and Indicators

1
Monitoring and Evaluation Concepts and Indicators

• Gugsa Yimer
• ME Officer, Juba

2
Objective

• By the end of the session, participants will be
  able to
• Explain major concepts in monitoring and
  evaluation
• Select appropriate indicators for monitoring and
  evaluation

3
Specific Objective

• By the end of the session, participants will be
  able to
• Be able to define monitoring and evaluation
• Distinguish between monitoring and evaluation
• Distinguish between the conventional and
  participatory methods of project monitoring and
  evaluation
• Distinguish between output, outcome and impact
  level indicators.
• Select appropriate indicators for measuring
  outputs and outcomes of their sector

4
Outline

• Definition/Concept
• Purposes
• Participatory ME
• Differences between ME
• Development / selection of indicator
• Definition
• Types of indicators
• Indicator selection criteria
• Steps in selecting indicators
• Monitoring and Evaluation of the Work Plan
• Sharing ME information

5
Monitoring and Evaluation

• Group work (15 minutes)
• Thoughts about ME
• What do we mean by monitoring, evaluation
• What do you think is the difference between
  monitoring and evaluation?
• Positive and negative thoughts regarding
  monitoring and evaluation?
• What are the major reasons for conducting
  Monitoring and Evaluation

6
Monitoring and Evaluation Introduction

• Definition
• Monitoring is a continuous process used to
  determine the extent to which an intervention or
  a project has been effectively implemented at
  different levels, in time and at what cost.

7
Purpose of monitoring

• to achieve the best possible performance by
  providing feedback to the beneficiaries and the
  management / donors at all levels.
• to improve work plans and to take corrective
  action in the case of shortfalls and constraints.
  
• to track progress and change that occur over time
  in inputs, process and outputs by means of
  record-keeping and regular reporting systems.
• help program managers determine which areas
  /states are in need of greater effort.
• The systematic recording of monitoring findings
  can later serve as one of the progressive
  evaluation tools of the intervention.

8
Monitoring

• Monitoring should be carried out regularly, for
  example, monthly, quarterly, etc.
• Data are usually compiled at lower level such as
  community and county levels and later forwarded
  to states and regions and finally to national
  level.
• The information has to be used at all levels of
  the system for self assessment and tracking the
  progress and decision.
• Therefore, it is critical that everyone involved
  be clear about which indicators are being
  monitored, as well as how the information will be
  collected and used.

9
EVALUATION

• Evaluation is a collection of activities designed
  to determine the value or worth of a specific
  program, intervention or project, i.e., to link a
  particular output or outcome directly to a
  particular intervention.
• It also entails detailed analysis of the
  effectiveness of the program by calculating the
  extent to which the pre-set objectives and
  targets are achieved.
• Evaluation
• to be decided during the initial planning
  process.
• requires prior planning and may also need the
  involvement of external evaluators.
• is based on the set of indicators set during the
  initial planning process.

10
Purpose of Evaluation

• The purpose evaluation is to measure the effects/
  outcomes and impacts of a program against the
  objectives it set out to accomplish, in order to
  guide decision making during program
  implementation and to improve future programs.
• It helps in determining
• Whether such projects or activities should be
  extended for more time in the same geographic
  area
• Whether the same or similar types of activities
  should be replicated elsewhere
• Whether the project requires major modifications
  in strategy and approach in order to be effective
  
• What needs to be different in terms of strategy
  and approach when replicating the project
  elsewhere .
• Participatory Versus Conventional?

11
PARTICIPATORY ME

• Participatory ME refers to the process of ME
  where all project partners beneficiaries and
  project staff are involved.
• Instead of having a team of outsiders visit the
  project to carry out the evaluation, the project
  partners themselves conduct the evaluation.
• If an outsider is involved, her or his role
  should be to facilitate the process and serve as
  a technical resource.
• In participatory evaluation, all key decisions
  regarding the evaluation are made by the project
  partners. These include
• Timing, when to carry out the evaluation
• Process, indicators and analysis
• Sharing and reporting and using the findings.
• Participatory ME is most effective when the
  project design and implementation have also been
  carried out in a participatory manner.

12
Difference between Participatory and Conventional
Monitoring and Evaluation

• There is one key difference between participatory
  and conventional monitoring and evaluation.
• In the case of participatory ME , community
  members, that is, the direct beneficiaries of the
  project, play an active role in ME.
• They maintain records at the community level,
  analyze progress, and use this information to
  make decisions about project implementation.
• The main characteristics of this process are
  inclusion, collaboration, collective action and
  mutual respect.
• In the conventional ones, monitoring and
  evaluation is done by staff members and external
  experts to address donor requirement, and
  therefore all monitoring activities are geared
  towards producing reports for the donors.

13
USING BOTH CONVENTIONAL AND PARTICIPATORY ME

• Donors who fund organizations will continue to
  need data to show how their funds have been
  invested in humanitarian and development, and how
  they have contributed to the beneficiaries.
• For the time being, organizations can expect that
  most donors will include a requirement in grants
  and contracts for the regular submission of
  program and financial reports.
• Its possible to fulfill such a requirement while
  at the same time meeting the needs of the
  beneficiaries/ communities.
• Participatory and conventional ME can be
  effectively combined what the donor requires and
  what PME offers are not mutually exclusive.
• The same information collected through a
  participatory monitoring process can often be
  presented to the donor in a slightly different
  format.

14
Difference between ME
15
Selection of Indicators

• Indicator
• A marker of performance
• A key to ME
• Indicators are milestones or markers that show
  where we are in an activity, that we are making
  progress, and that we are heading in the right
  direction.
• They also show whether we have achieved our
  objectives.
• In any project, indicators depend on the
  projects objectives. While objectives tell us
  what the project plans to achieve, indicators
  tell us how to measure to ascertain if those
  objectives are achieved or not.

16
Indicators

• Indicators can be used to monitor the following
  elements
• Changes over time (e.g. change in average
  household income over a period of time)
• Differences between population sub-groups (e.g.
  Prevalence of malnutrition between males
  females between urban rural)
• Achievement towards targets (e.g. average food
  production vis-à-vis initially planned)

17
Indicators

• Indicators can be grouped as
• Population level indicators
• Program/project level indicators
• Population level indicators
• Gathered from representative sample of the target
  population
• Not necessarily related to a specific project
  /program, Example
• Average food production per farming household
• Per capita income
• Nutritional status of children under 5
• Program/project level indicators
• Generated within the project
• E.g. number of supplies distributed
• Number of new saving and credit association
  formed in the last quarter
• No of farmers trained in agriculture extension in
  the last 1 year, etc
• Note that program level indicators are more
  useful than population level indicators in
  directly tracking input and out put of a project.

18
TYPES OF INDICATORS

• There are four types of indicators generally used
  in project monitoring and evaluation
• numerical
• scaling or ranking
• classifying and,
• descriptive.
• Numerical provides exact numbers. Numerical
  indicators are also called quantitative
  indicators. Examples of numerical indicators are
  
• number of farmers trained
• Number of animals vaccinated
• number of tools distributed by type
• number of malnourished children supported.
• The data for these indicators are counted.

19
TYPES OF INDICATORS

• Scaling or ranking provide graduated descriptions
  of assessment. For example, people can rank
  satisfaction with services on a scale of 1 to 4,
  where 4 is highly satisfied and 1 is not
  satisfied at all.
• Classifying provides answers in categories Yes
  or No Male or Female Orthodox or Catholic or
  Muslim or Protestant.
• Descriptive indicators are qualitative because
  they describe the state of something in words.
  Examples of descriptive indicators are the
  inheritance rights of women are fulfilled.

20
WHAT MAKES A GOOD INDICATOR?

• A good indicator clearly demonstrates the
  expected progress or result. It measures the
  intended change as accurately as possible. It is
  clearly defined, easily understood and easily
  measured.
• For example, in an nutrition project, the
  objective is to provide 500 malnourished children
  with nutritional supplements.
• One indicator that measures the success of this
  objective is to count the actual number of
  children provided with nutritional supplements.
• Number of malnourished children provided with
  nutritional supplements is a simple and
  straightforward indicator.
• Sometimes we have to use indirect indicators,
  which are also called proxy indicators, to
  measure change.
• E.g., in a project aiming to reduce stigma
  affecting AIDS orphans it is difficult to
  identify direct indicator, - stigma is complex
  and manifests itself in various forms.
• Use indirect indicators to measure it
• Some proxy indicators for community-level stigma
  reduction
• number of AIDS orphans being hosted in extended
  family households
• Number people who are willing to kiss HIV
  positive people for greetings

21
Rules of thumb in selecting indicators

• Review objectives carefully. Try to understand
  exactly what they are saying.
• Avoid formulating objectives in a broad manner
  such objectives are not clear and make it
  difficult to identify indicators for monitoring
  and evaluation purposes.
• For example, Agricultural growth through
  diversification is a broad objective.
• Use specifics of the projects intentions in the
  objective such as Provide agricultural inputs to
  ____ HHs in village XYZ about to increase
  productivity.
• That can lead to specific indicators such as
  Number of HHs who received agricultural inputs
  Percent of HHs who received agricultural input
  reporting an increase in production. Also define
  project beneficiaries for example, is it the
  entire village, or selected families or
  individuals?
• Be clear about what type of change is implied.
  What does the project expect to change?
  Knowledge, attitudes, practice, laws or policies.
• And at what level? Individual, household, group,
  community.

22
Rules of thumb in selecting indicators

• It is important to define the indicators clearly
  at the very beginning.
• While numerical indicators are easy to adapt,
  count and report, they are not necessarily the
  most useful and meaningful indicators.
• Indicators that will be monitored over a period
  of time need to remain relatively stable over
  time.
• Make sure that the indicators you choose are
  practical for data which can be collected on a
  regular basis.
• The data should be collected at a reasonable cost
  and in reasonable time.

23
Selection/Construction of Indicators

• People who design projects will develop
  monitoring plans and indicators.
• In PME, the community and the implementing
  organization select the indicators together, with
  input from the donor, and conduct monitoring.
• Selecting indicators and setting targets
• is usually done during the process of program
  planning,
• in a participatory way with communities and key
  stakeholders.
• should also include information from similar
  types of interventions, so that the targets set
  are realistic from the perspective of the target
  population, resource allocation, and intervention
  type.
• In addition to the community and the implementing
  organizations, donors also select indicators,
  based on what they need to know about the
  projects outcome.

24

• Monitoring
  Evaluation
• Process
  Effectiveness

INPUT
OUTPUT
ALL
MOST
OUTCOME
IMPACT
SOME
No. of projects
FEW
Long-term
Short/medium term
Level of evaluation efforts
25

• Monitoring
  Evaluation
• Process Effectiveness

INPUT
OUTPUT
ALL
MOST
OUTCOME
IMPACT
SOME
FEW
No. of projects
Knowledge change Increase in production as a
Result of agri. Input supply Behavioral change
Increase in per capita income Child
Mortality Maternal mortality
Resource Staff Funds Facilities Supplies
No of farmers trained Trained staff Agri. Inputs
provided
Level of evaluation efforts
26
Example of indicators by input output framework
27
Steps in selecting indicators

• Selecting appropriate and useful indicators is a
  fairly straightforward process, but requires
  careful thought, collaboration, and consensus
  building.
• Although these steps are presented as discrete
  steps, in practice some of these can be
  effectively undertaken simultaneously.

28
Step 1 Clarify the results statements

• Good indicators start with good results
  statements or objective that people can
  understand and agree on. For each result
  statement
• Carefully consider the result desired/ objective
  
• Avoid overly broad result statements/ objectives
  
• Be clear about what type of change is implied
• Identify more precisely the specific target for
  change
• Study the activities and strategies directed at
  achieving the change.

29
Step 2 Develop a list of possible indicators

• There are usually many possible indicators for
  any desired outcome/objective, but some are more
  appropriate and useful than others. Start by
  creating an initial list of possible indicators,
  tapping the following sources
• Internal brainstorming by the project team and
  other stakeholders
• Consultations with experts in the substantive
  program area and
• Experience of other operating units with similar
  indicators.
• The key to creating a useful initial list of
  indicators is to be inclusive.

30
Step 3 Assess each possible indicator

• When assessing and comparing possible indicators,
  the following seven criteria may be helpful
• Direct Indicator should measure as closely as
  possible the result/objective it is intended to
  measure. If using a direct measure is not
  possible, proxy indicators might be appropriate.
• Objective An objective indicator has no
  ambiguity about what is intended to measure. That
  is, there is general agreement over
  interpretation of the results.
• Adequate An indicator and its companion
  indicators should adequately measure the
  objective / result in question. The number of
  indicators needed to measure any given result/
  objective, depends on the complexity of the
  result being measured, the level of resources
  available for monitoring performance, and the
  amount of information needed to make reasonably
  confident decisions.

31
Step 3 Assess each possible indicator

• Quantitative (where possible) Quantitative
  indicators are not necessarily more objective,
  however, their numerical precision lends them to
  more agreement on interpretation of results data,
  and are thus usually preferable.
• Disaggregated when data are disaggregated by
  gender, age, location, or some other dimensions,
  they help track whether or not specific groups
  participate in or benefit from activities
  intended to include them.
• Practical An indicator is practical if data can
  be obtained in a timely way and at a reasonable
  cost. Managers require data that can be collected
  frequently enough to inform them of progress and
  influence decisions.
• Reliable Data obtained should be of sufficiently
  reliable quality for confident decision-making.

32
Step 4 Select the best indicators

• The last step is to narrow the list to the final
  indicators that will be used in the monitoring
  system. The indicators selected should be the
  optimum set that meets the information needs of
  management, at a reasonable cost.
• Remember
• Limit the number of indicators
• Identify the type of data collection needed for
  each indicator
• Priorities by importance and ease of collection
• Group selected indicators by source of data
• Decide on what the project is able to do given
  the resource available

33
Monitoring and Evaluation of the Work Plan

• Sector plans
• Sector Objectives
• SMART
• Link between Project sheets and sector plans
• Indicators
• Direct, objective, adequate, practical. ?
• Beneficiaries clearly stated

34
Monitoring and Evaluation of the Work Plan

• Project Sheets
• Project Objectives
• SMART
• Link between Project sheets and sector plans
• Targets/Indicators
• Direct, objective, adequate, practical?
• Link between Project sheet and sector plan
  indicators
• Beneficiaries clearly stated

35
Sharing monitoring information

• Once the monitoring report is ready, it should be
  shared with all project staff so that progress
  can be reviewed with concrete evidence.
• Most projects hold monthly review meetings, and
  are conducted in two steps.
• The first review meeting is held with project
  staff.
• The monitoring information is discussed at this
  meeting and decisions at a project level are
  taken.
• The second meeting is held with representatives
  from the communities, so that they also get a
  chance to review progress and present their
  views.
• This provides an opportunity to take joint
  decisions for community-level activity.
• Report will be also shared with higher level
  managers

36

• Format for comparing actual performance with
  standard/planned outputs - Example

37
Group work

• 3 groups (30 Minutes)
• Based on the indicator selection criteria,
  examine the indicators of Food Security and
  Livelihood Sector of South Sudan (Work Plan 2006)
• Select key indicators from the list (You can
  modify indicators listed)
• Present to the plenary
• Thank You