Valuing the evaluation function: problems and perspectives

1
Valuing the evaluation function problems and
perspectives

• Danielle Labbé
• Centre of Excellence for Evaluation
• June 1, 2006

2
BackgroundProblemsPerspectives
3
For many, evaluation is similar to internal
audit. However, they are two different things.

• The purpose of internal audit is to examine how
  we do things it identifies strengths and
  weaknesses in the management control framework.
• Evaluation indicates whether we are doing the
  right things and if we are proceeding in a
  cost-effective manner.
• Evaluation pertains to the effectiveness and
  value of programs, whereas audit covers financial
  attest, compliance, insurance and risk.

Audit addresses the manner in which programs are
delivered, while evaluation indicates whether
programs are obtaining the desired value.
4
The federal government has a solid infrastructure
for evaluation and for internal and external
audit.

• All of the main departments and agencies
• have internal audit and evaluation sections
• publish the findings of their audits and
  evaluations.
• The Treasury Board Secretariat
• established the Centre of Excellence for
  Evaluation in 2001 in order to
• serve as a focal point for leadership of the
  evaluation function in the federal Public Service
• launch initiatives to meet the challenges shared
  by the community
• support capacity building, practice improvement,
  and strengthening of the evaluation community.
• The Office of the Auditor General
• conducts performance audits of departments and
  agencies.

5
In 2001, the TB introduced a four-year investment
strategy that strengthened the evaluation
infrastructure.
SOURCES 2002-2003 CEE evaluation 2005-2006
annual departmental survey
6
TBSs investment has also led to an increase in
evaluation products
Evaluation products -- 2001 to 2005
but this trend did not continue when the
investment ended in 2004-2005, thus raising
concerns.
SOURCES 2002-2003 CEE evaluation 2004-2005
and 2005-2006 annual departmental surveys
7
If enacted, the Federal Accountability Act will
entail even more careful review of program
performance

• Auditor General to investigate how monies awarded
  under funding agreements with the federal
  government are used by third parties.
• Parliamentary Budget Office to provide analysis
  to Parliament regarding the nations finances and
  trends in the national economy.
• Departments to evaluate, at least once every five
  years, the relevance and effectiveness of their
  grants and contributions programs.

thus taxing evaluation capacity.
8
BackgroundProblemsPerspectives
9
There are some real challenges for the evaluation
function.

• Problems with quality, timely delivery and
  strategic focus complicate the use of evaluation
  to support decision-making
• often focused on small programs and not strategic
• can take too long to complete and produce results
  that are difficult to understand
• can be biased if funded by program managers
• Government-wide capacity issues have made
  evaluation difficult to deliver
• lack of properly trained evaluators
• skills of those leading the evaluation function
  not consistent
• definition of evaluation product has not changed
  in 20 years
• The new Internal Audit Policy forces evaluation
  to rethink its relationship with audit.

10
Although they often explore the question of
resource optimization, convincing evaluations of
a programs value are rare.

• Under the current policy, evaluations must take
  into account
• Relevance
• Success
• Cost-effectiveness
• Most evaluations focus on program improvement
  (72)
• Few evaluations attempt to determine whether the
  program is cost-effective (26)

Evaluation issues considered (115 evaluation
reports)
SOURCE EKOS study on quality of federal
evaluation, 2004
11
Evaluations are not conducted at a high enough
level to properly support decision-making with
regard policies and expenditures.
Size of programs evaluated (Millions ) (N129,
2004-05)

• Half of all evaluations cover programs under 4
  million.
• 12 cover programs of less than 500 000.
• Deputy ministers want to know about departmental
  policy and program performance at the highest
  level.
• Since 1996 the OAG has been recommending
  evaluation of larger programs of interest to
  parliamentarians.

SOURCES Annual CEE survey of heads of
evaluation Breen Report, 2004-2005 Report
of the Auditor General of Canada May Chapter
3 1996 and Report of the Auditor General of
Canada December 2000 Chapter 20 Appendix D

12
Evaluations are difficult to understand and take
too long to perform.

• Quality increased by 18 in the three years
  following the CEEs creation.
• However, 23 of evaluations are still
  inadequate.
• Deputy ministers feel that evaluation reports
  tend to be too long, theoretical and difficult to
  understand.
• Evaluations are not conducted in time to respond
  to decision-making needs.

Number of months evaluation takes (Does not
include time required for publication approval)
(N128, 2004-05)
SOURCES EKOS study on the quality of federal
evaluation, 2004-2005 Breen Report, 2004-2005
Annual CEE survey of heads of evaluation,
2004-2005
13
The scope of evaluations is limited, thus
highlighting the need to adopt new, more
appropriate approaches to evaluating the value of
programs.
Percentage of departmental program spending in a
year

• Only 23 (9) of departments indicated they
  intended to evaluate all programs over a
  five-year period.
• These departments are responsible for most
  transfer payment programs.
• This poor coverage is also related to capacity
  problems.

SOURCES CEE, The Health of the Evaluation
Function report, 2004-2005 and 2005-06 Annual
CEE survey of heads of evaluation, 2004-2005
14
Deputy ministers and deputy heads, TBS and heads
of evaluation are concerned about capacity.

• Deputy ministers indicate that the contribution
  of evaluation is diminished by
• a lack of resources and competent evaluators
• current focus on internal audit
• weak linkages with policy and expenditure
  management.
• For heads of evaluation, access to competent
  evaluators is a major obstacle.
• TBSs need for evaluations on the effectiveness
  of horizontal initiatives and programs is
  increasing, but the evaluation function has
  difficulty delivering the goods.
• Heads of evaluation see TBSs lack of
  coordination of horizontal evaluations as a major
  obstacle.

MAF findings, 2005-2006 (36 departments and
large agencies)
Significant
Acceptable
Attention needed
Room for improvement
SOURCES Breen Report, 2005 Report on CEE
consultations, 2004 2005-2006, TBS Internal
Audit and Evaluation, independent CEE study, 2005
15
Most small agencies do not have sufficient
capacity.

• Although small agencies account for only 3
  billion in program spending and 3 000 FTEs, they
  can present major risks.
• For example, the following agencies have little
  or no evaluation capacity
• Office of the Commissioner of Official Languages
• Canadian Nuclear Safety Commission
• Financial Consumer Agency of Canada
• National Energy Board

Small agencies

• 17 (5) completed evaluations in 2003-2004
• 20 completed evaluations in 2004-2005

SOURCE Business case analysis of small
agencies, CEE, 2004-2005
16
DEPUTY MINISTERS doubt the reliability and
neutrality of evaluation given the use of
consultants and funding through programs.
Major sources of evaluation funding
Internal vs. contract resources
Internal resources
Evaluation unit
Shared costs
Funded through programs
SOURCES Breen Report, 2004-2005 CEE annual
report on The Health of the Evaluation Function ,
2004-2005
17
The function also needs to be repositioned within
the government structure.

• Deputy heads are concerned about inadequate
  linkages between evaluation and policy and
  expenditure management.
• The governance requirements of the new audit
  policy raise questions with respect to the
  position and the governance mechanisms most
  suitable for evaluation.
• Evaluations position must enhance access to the
  deputy minister and ADM community to enable it to
  play its strategic role in developing and
  evaluating a departments results structure.
• There is a lack of connection between evaluation
  and similar types of performance-related
  activities, including performance evaluation,
  reports to Parliament, MRSS and PAA, and
  strategic planning.

18
The introduction of the new Internal Audit Policy
The majority of departments have combined audit
and evaluation committees
Governance issues

• Until now, audit and evaluation were governed by
  the same deputy-led committee.
• Under the Internal Audit Policy, audit committees
  and audit heads did not need to be involved in
  decision making.
• Departments are seeking clarification as to the
  role and governance of evaluation.

Evaluation committees
Combined audit and evaluation committees
encourages departments to contemplate the best
way of using and governing evaluation.
19
Deputy heads are concerned about the integration
of evaluation within their organizations .
The majority of audit and evaluation units are
co-located
Governance issues
Evaluation

• There is a variety of reporting relationships
  across departments.
• The majority of departments have combined audit
  and evaluation, along with combined audit and
  evaluation committees.
• Overall, the ADMs commitment to managing and
  advocating on behalf of the function is low.

Audit and evaluation
Evaluation units in the departments report to
various officials
and the lack of connection with decision making
on policies and expenditures.
SOURCE CEE annual survey of heads of
evaluation, Breen Report, 2004-2005
20
Predominant Current Approach

• 77 of departments and large agencies use the
  current model OAG questioned whether
  co-location is consistent with ensuring each
  functions distinctiveness.
• The audit policy requires that this function be
  more independent, whereas evaluation is a
  management function that directs policies and
  spending.
• Combined audit and evaluation function does not
  reflect new Internal Audit Policy
• no Chief Internal Audit Executive
• no external audit committee.
• Lack of alignment of evaluation with performance
  measurement and strategic planning.
• Lack of influence of evaluation on expenditure
  management and policy decisions.

21
BackgroundProblemsPerspectives
22
Canadian evaluators want changes, but they will
be facing some substantial implementation
challenges.

• Demographics High turnover rates will need to
  be addressed, with a government-wide recruitment
  target
• Workload Will be difficult to manage without a
  realistic plan
• Skills Will take time to rebuild training
  programs and partner with universities
• Jurisdictional Continually reinforcing the
  distinction between internal audit and evaluation
• Relevance Good quality, timely products need to
  be used to inform allocation and reallocation
  decisions

23
Step 1 is to renew our policy so that evaluation
is ...

• better recognized as a core function of public
  management
• an integral part of the decision-making process
• related to budgeting and expenditure management
• a neutral and independent function in line with
  the governments management priorities
• the source of a relevant and quality product.

24
Step 2 is to zone in on implementation.

• Make sure the function is properly resourced
• Build capacity by re-establishing training and
  recruitment programs and partnering with
  universities on certification.
• Guarantee objectivity it is up to evaluators to
  fund evaluations
• Redefine the product  to promote relevance and
  better coverage
• Strategic policy evaluation
• Impact assessment
• Evaluation of resource optimization
• Evaluation of implementation.
• Treasury Board Secretariat to assess quality and
  set out clear competencies for heads of
  evaluation and evaluators.
• Present a government-wide evaluation plan.

25
Strengthening the scope depends on the
restructuring of evaluation products to reduce
usage times and enhance their relevance.

• Prospective in nature, review of high-level
  programs in PAA, focusing on relevance, lessons
  learned, options and potential consequences.
  Rapid evaluation that combines policy analysis
  and evaluation methods at a higher level to
  identify potential long-term consequences and
  options.
• In-depth review of program effectiveness and
  impacts (both intended and unintended) and
  alternatives.
• Review of program relevance and performance
  (cost-effectiveness and overall effectiveness).
  Tool used would lead to program improvement plan
  or more thorough evaluation.
• Review of program operation. Focuses on
  implementation process and on management issues
  rather than results. Issues include
  accountability, governance and delivery
  mechanisms.
• Supports the delivery of client-centred programs
  through the Common Measurement Tool and related
  tools and approaches for monitoring client
  satisfaction.

Strategic policy evaluation Impact
assessment Evaluation of resource
optimization Evaluation of implementation
Evaluation of client services
26
ANNEX Reference documents

• Government of Canada
• http//canada.gc.ca/main_e.html
• Treasury Board of Canada Secretariat
• http//www.tbs-sct.gc.ca/index_e.asp
• The Health of the Evaluation Function in the
  Government of Canada 2004-2005
• http//www.tbs-sct.gc.ca/eval/dev/health-santE9
  /hefgc-sfegc_e.asp
• Report on CEE consultations, Peter Hadwen, 2005
• http//www.tbs-sct.gc.ca/eval/pubs/repcon-rapcon
  /repcon-rapcon01_e.asp
• Decision-Making in Government the Role of
  Program Evaluation, Peter Aucoin, 2005
• http//www.tbs-sct.gc.ca/eval/tools_outils/Aucoi
  n/Aucoin_e.asp
• Professional Ethics and Standards for the
  Evaluation Community in the Government of Canada,
  Ken Kernaghan, 2006
• http//www.tbs-sct.gc.ca/eval/dev/career/ethics-
  E9thiques/ethics-E9thiques_e.asp
• Review of the Quality of Evaluations Across
  Departments and Agencies, 2004