Prsentation PowerPoint

1
BALANCING ADVICE AND INDEPENDENCE IN
EVALUATION Departmental Evaluation Services June
16, 2005
2
INDEPENDENCE

• THREATS, SUPPORTS, SOLUTIONS

3
WHY IS INDEPENDENCE IMPORTANT?

• Sustains neutrality and objectivity
• Influences credibility
• Supports evaluation as research, e.g. methods
  chosen to meet research needs, not to produce the
  right answers
• Increases impact

4
WHAT WORKS AGAINST INDEPENDENCE?

• Evaluators are departmental employees ultimate
  accountability is to a DM who has to respond to a
  Minister, to Cabinet, to committees
• Nice people dont like to dump on their
  colleagues Sorry, your program is no longer
  relevant.
• Easy to get succumb to the lure of the program
  enthusiast/apologist especially when emotions
  kick in
• Easy to succumb to the threats of the program
  bully evaluators have career aspirations too
• Evaluators understand the political context that
  knowledge could keep them from even attempting to
  look at an issue

5
WHAT SUPPORTS INDEPENDENCE?

• Evaluators have strongly held values dont want
  to compromise on independence, neutrality and
  objectivity
• Managers and colleagues expect it they want
  evaluators to give them an outsiders perspective
• Under normal conditions, few threats to
  independence no pressure until the crisis hits

6
WHAT CAN HEADS OF EVALUATION DO TO SHORE UP
INDEPENDENCE?

• Look to
• Leadership
• Reporting relationships
• Study quality
• Relationship building
• Staff education
• Consultation

7
LEADERSHIP

• Heads of evaluation need to set
• Explicit values
• Clear direction
• Expectations for research quality
• Limits of tolerance for dissenting views
• The right tone

8
REPORTING RELATIONSHIPS

• Closer to DM usually better
• Make-up and engagement of decision-making
  committee critical
• Personality and management style of superiors
  important

9
STUDY QUALITY

• Best defence is a strong offence prove to me
  that this is not a legitimate finding, a
  substantiated conclusion, a reasonable and
  realistic recommendation
• Evaluations stand or fall by the quality of the
  work, recognizing that there will always be grey
  areas give and take is inevitable
• A good negotiation strategy is key in the first
  round of discussions, stretch conclusions and
  recommendations to the ultimate, be confident
  start from a position of strength

10
RELATIONSHIP BUILDING

• Need to build strong relationships with
  program/policy managers long before the
  evaluation starts
• Good service, e.g. helping on RMAFs, PAA, DPR,
  strategic planning
• Collegiality
• Openness
• Sharing of expertise
• Respect

11
STAFF EDUCATION

• Evaluation staff need to be educated on the rules
  of engagement for involving stakeholders
  internal or external in the evaluation process
• Roles and responsibilities
• Decisions points, authorities, accountabilities
• Compromise versus capitulation

12
CONSULTATION

• Consult, brainstorm, discuss, argue with, provoke
  stakeholders, but
• Do it at the right time
• Do it at the right level
• Do it for the right purpose
• Then, draw a line and stand firm

13
ADVISORY COMMITTEES

• TOOLS FOR CONSULTATION, ENGAGEMENT NEGOTIATION

14
COMPOSITION

• Anyone who needs to have a say about findings,
  conclusions and/or recommendations before the
  report hits the website
• Could be just departmental managers, including
  senior managers up to ADM, or even DM
• Could also include external stakeholders such as
  program clients, funding recipients, regulatees,
  representatives of interest groups, other levels
  of government, general public

15
PURPOSE

• To get buy in to process
• To anticipate reaction to evaluation results
• To increase likelihood of acceptance of results
  and implementation of recommendations
• To capture special knowledge or perspectives,
  including the political perspective
• To deal with cultural sensitivities
• To breach social/economic barriers

16
IMPACT ON INDEPENDENCE

• Positive
• Advisory committees have TORs, so participants
  should know the limits of their influence, in
  particular their decision-making powers
• Stakeholders come into the process at the time
  when evaluators need their input have less
  opportunity to interfere when it isnt
  appropriate
• Stakeholder input and influence is dealt with in
  an open forum at intervals throughout the
  evaluation process, not behind closed doors as
  the report is being finalized

17
IMPACT ON INDEPENDENCE contd

• Negative
• Evaluators overpowered by stakeholders
  leadership becomes problematical
• Danger of evaluators getting too close to
  stakeholders
• Advisory committees think they are steering
  committees
• Too formal, not flexible give and take becomes
  difficult if stakeholders take hard positions for
  political or other reasons
• Committee members stall an evaluation they dont
  agree with as a deliberate strategy to keep the
  evaluation hung up until information is out of
  date or events have overtaken it