Evaluation and Monitoring Methodologies

1
Evaluation and Monitoring Methodologies

• Strengthening the Legislature Challenges and
  Techniques
• K. Scott Hubli, NDI

2
Overview

• General Comments on Monitoring andEvaluation
• Special Considerations in Monitoring and
  Evaluating Legislative Strengthening Programs
• Practical Tips and Considerations

3
General Comments onMonitoring and Evaluation

• Evaluation (and Baseline Assessments)
• --Use to develop program design use for major
  course corrections
• --More costly and less frequent than monitoring
  (every two to three years)
• --Typically done at the beginning and the end of
  a program, but often also done after a major
  change in the political landscape (e.g., regime
  change, ethnic conflict settlement, etc.)
• --Used for accountability to partners, donors,
  stakeholders, not for ongoing project management

4
General Comments onMonitoring and Evaluation

• Performance Monitoring
• --Ongoing monitoring used to manage performance
  of implementation
• --Track changes (but less analysis)
• -- Informed by baseline assessment and, if well
  designed, it can reduce future evaluation costs
• -- May indicate a need for a evaluation or
  updated baseline
• -- Focus on low cost, regular data collection
  (workshop evaluations, information available from
  parliament, regular focus groups, etc.)

5
General Comments onMonitoring and Evaluation

• Always distinguish among
• --Inputs (e.g., consultants, computers, etc.)
• --Outputs (e.g., 40 people trained in a workshop
  onoversight techniques)
• --Outcomes (e.g., increased knowledge of
  oversight investigation techniques)
• --Objectives (e.g., increased oversight hearings)
• --Goals (e.g., increased government
  accountability)

6
How are legislative strengthening programs
different from other programs with respect to
monitoring and evaluation?
7
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Legislatures are highly complex institutions
  --They involve multiple actors seeking to
  achieve multiple goals simultaneously
• --Where possible, disaggregate data (by gender,
  party, region, etc.)
• --Identify clear goals and targeted groups
  watch for unintended consequences

8
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Long-term goals, short-term programs
• -- Resist the tendency to monitor outputs rather
  than progress in achieving desired outcomes,
  objectives and goals
• -- Find ways to measure small changes in large
  goals or outcomes that can be affected with
  the project time frame

9
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Programs focus on process, not outputs
• --Example number of laws passed--Emphasize
  qualitative over quantitative information
• --Use detailed process descriptions in
  establishing baselines
• --Use monitoring and evaluation to help
  strengthen this process and to teach
  results-based management, where possible
  (Monitoring and evaluation should be managed as
  joint exercises with development partners.)

10
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Monitoring and evaluation is often highly
  political
• --Involving partners can sometimes further
  politicize evaluation and monitoring use
  caution and judgment
• --Can be hard to get necessary information
• --Politics may cause people to be less than
  fully honest
• --Results can be used as a political weapon

11
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Legislatures have natural cycles
• --Elections, post-election learning curves,
  legislative floor periods, recesses, budget
  processes, etc.
• --Example constituency relations
• --Expect uneven development in performance
  monitoring, but try to attribute fluctuations in
  data
• --Time evaluations carefully look for
  normal periods

12
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Many intervening variables
• --Economic conditions, geopolitical
  developments, ethnic conflict, death of a key
  politician, etc.
• --No substitute for nuanced political analysis
• --Measure outcomes, objectives, goals not
  just outcomes this can help identify these
  intervening variables

13
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Perceptions matter
• --Importance of qualitative over quantitative
  indicators
• --Use of focus groups, opinion polls, etc.
• --Even anecdotal evidence is useful if it
  captures a political mood or issue

14
Special Considerations in Monitoring and
Evaluating Legislative Strengthening Programs

• Difficulty of comparative benchmarking
• --Only one national legislature
  cross-countrycomparisons are of limited utility
• --Comparisons across time more important use
  ofthorough baselines
• --Implications on setting goals and targets use
  ofreasonable/consensus expectations

15
What are some practical strategies for dealing
with these unique aspects of monitoring and
evaluating legislative strengtheningprograms?
16
Practical Tips Considerations

• General Issues
• --Be pragmatic in designing an evaluation or
  monitoring plan tie evaluation and monitoring to
  the purpose or objectives. Avoid evaluation for
  evaluations sake. Consider
• --resource availability for evaluation
• --novelty of the program
• --confidence in program design or
  implementation
• --needs of funder
• --Budget sufficient resources (Costs for
  legislative strengthening evaluation may exceed
  those for other program types soft assistance,
  new field, etc.)

17
Practical Tips Considerations

• Issues in Doing a Baseline
• --Limit scope to allow for detailed coverage of
  program areas
• --Protect against biases of person(s) doing the
  baseline by --Using teams
• --Using clear, detailed terms of
  reference
• --Incorporating documentary evidence
• --Seeking consistency in future assessments

18
Practical Tips Considerations

• Issues in Doing a Baseline (cont.)
• --Pick timing carefully describe any special
  circumstances--Prepare carefully for baseline
  assessment team
• --Cover the range of stakeholders
• --Get out of the capital
• --Consider focus groups or creative methods for
  documenting perceptions and processes (e.g., a
  sample of 10 legislators to track periodically
  every 3 years)--Pay attention to protocol build
  good will.

19
Practical Tips Considerations

• Using outside evaluators
• --Outside evaluators can not only provide
  objectivity but also insulation from the
  political consequences of an evaluation
• --Combine multiple backgrounds (academic or
  legislative strengthening specialists and MPs or
  staff from similar systems) --Recognize
  value of time in the trenches
• --Designate a lead person with responsibility
  for producing the document
• --Get a sufficient time commitment

20
Practical Tips Considerations

• Issues in Performance Monitoring
• --Draw on baseline and prior evaluations
• --Design performance monitoring plan up
  front adjust it as project evolves
• --Imposes discipline keeps program on
  track --Provides clarity of expectations to
  partners --Keep it current, modify as needed
• --Make these changes explicit

21
Practical Tips Considerations

• Issues in Performance Monitoring (cont.)
• --Tie to likely performance issues
• --Draw on low-cost existing information
  sources may be more quantitative, with less
  analysis --May focus on outcome level, rather
  than objective or goal level
• --Consider quarterly or semi-annual monitoring
• --Expect, but explain, fluctuations
• --When you cant explain repeated fluctuations,
  consider updating a baseline to try to
  identify issues --Often done, in part, by those
  implementing program

22
Final Thoughts

• Be creative legislative strengthening is anart,
  not a science
• Be willing to accept criticism fight structural
  bias for spinning results
• Share lessons learned, both internally and
  externally