The Evaluation Function and the New Policy on Evaluation

1
The Evaluation Function and the New Policy on
Evaluation

• Centre of Excellence for Evaluation
• Results-Based Management Division
• Expenditure Management Sector
• Treasury Board Secretariat

2
Key Topics of Discussion

• Renewal of the Policy on Evaluation
• Competencies for evaluators
• Evaluation and renewal of the Expenditure
  Management System

3
Renewal of the Policy on Evaluation
4
Drivers for the Renewal of the Policy on
Evaluation

• Canadians want government to ensure that they are
  getting value for money. They want high quality
  services and high value programs, as well as
  achievement of results at a reasonable cost
• Government has committed to using results and
  value-for-money to inform priority setting and
  decision-making through a renewed Expenditure
  Management System (EMS)
• Strategic Reviews are to ensure that overall
  government spending supports programs that are
  relevant, effective and efficient
• Decision-making must rely on objective
  evidence-based information
• Evaluation will be critical to deliver on this
  commitment
• The Federal Accountability Act requires all
  Transfer Payment Programs to be evaluated
  (relevance and effectiveness) once every five
  years
• Even before these developments, a TBS diagnostic
  indicated that current evaluation policy was not
  fully meeting government information needs to
  support expenditure management
• The AG has consistently noted the need for the
  evaluation of ongoing programs, not just transfer
  payments

5
Challenges to Address

• Quality, timeliness and strategic focus issues
  hinder systematic use of evaluation to support
  decision-making
• often small programs - not strategic
• about 10 of spending each year - too low
• take too long and difficult to understand
• can be seen as not neutral when directed/funded
  by program
• Capacity issues have made it difficult to deliver
• Need more trained evaluators
• Need consistent competencies for evaluators
• Definition of the evaluation product same as 20
  years ago - need to evolve product as
  Results-Based Management culture is growing

6
Objectives of the New Policy

• Improve the information base for expenditure
  management - increased focus on value for money
  (relevance and program performance)
• Comprehensive coverage of programs through a
  regular and systematic cycle
• Improved credibility through agreed upon
  standards, flexible tools for evaluation and
  neutrality of the evaluation function
• Improved quality by having the right capacities,
  people and systems
• Continuous learning and improvement - stronger
  Treasury Board Secretariat capacity to lead the
  function, monitor capacity and use evaluation
  information

7
2009 Policy on Evaluation 3 Policy Instruments
Three Instruments Policy on Evaluation Direct
ive on the Evaluation Function Standard on
Evaluation for the Government of Canada
Policy Objective To create a comprehensive and
reliable base of evaluation evidence that is used
to support policy and program improvement,
expenditure management, Cabinet decision making,
and public reporting
8
Key Features of the New Policy
Results Sought
Policy Provisions

• Expand and re-focus evaluation coverage
• Increased coverage of direct program spending
• Focus on value-for-money
• Flexibility in evaluation approach and
  calibration of effort for individual evaluations
  based on risks and quality of performance
  measurement
• Strengthen Governance
• Clarify Deputy Head, Head of Evaluation
  responsibilities
• Clarify role of Departmental Evaluation Committee
• Clear Evaluation Standard
• Clear role in Performance Measurement
• Competency requirements

• Comprehensive and timely program performance
  information for Deputy Heads and Ministers
• Provide evidence to support to Expenditure
  Management
• Effective allocation of evaluation resources

• Primary clients of evaluations clear
• Neutral and effective evaluation function that
  consistently contributes to decision making

• More timely and cost-effective evaluations
• Consistent, known standards that apply across
  government

• Support to performance measurement to improve
  evaluations

• Competent evaluators

• Improved stewardship of the GoC evaluation
  function

9
Evaluation Core Issues (As per the Directive)

• Issue 1 Continued Need for Program (Assessment
  of the extent to which the program continues to
  address a demonstrable need and is responsive to
  the needs of Canadians)
• Issue 2 Alignment with Government Priorities
  (Assessment of the linkages between program
  objectives and (i) federal government priorities
  and (ii) departmental strategic outcomes)
• Issue 3 Consistency with Federal Roles and
  Responsibilities (Assessment of the role and
  responsibilities for the federal government in
  delivering the program)
• Issue 4 Achievement of Expected Outcomes
  (Assessment of progress toward expected outcomes
  (including immediate, intermediate and ultimate
  outcomes) with reference to performance targets
  and program reach Assessment of program design,
  including the linkage and contribution of outputs
  to outcomes)
• Issue 5 Demonstration of Efficiency and Economy
  (Assessment of resource utilization in relation
  to the production of outputs and progress toward
  expected outcomes)

10
Deputy Heads Responsibilities Some Key
Deliverables

• Policy Provisions
• Establishing a robust, neutral evaluation
  function designating Head of Evaluation (HoE)
• Ensures that HoE has direct and unencumbered
  access to DH
• Establishing a Departmental Evaluation Committee
  (DEC)
• Advisory to DH, in relation to departmental
  evaluation plan, resourcing and final evaluation
  reports
• Ensure follow-up to action plans
• Review resources allocated to performance
  measurement activities as they relate to
  evaluation
• Reviews performance of evaluation function
• Approves for annual submission to TBS a rolling
  5-year departmental evaluation plan, covering all
  direct program spending and administrative
  component of statutory programs
• Ensure that ongoing performance measurement is
  implemented throughout the department to
  effectively support evaluation of programs
• Approves evaluation reports, management responses
  and action plans in a timely manner

• Sections
• Sections 6.1, 6.1.1
• Section 6.1.3 and Annex B
• Section 6.1.7
• Section 6.1.11

11
Deputy Heads Responsibities (contd)

• DH is responsible for monitoring compliance with
  policy in their departments to ensure its
  effective implementation, incl.
• Neutral assessment of their departmental
  evaluation function is conducted at a minimum of
  once every five years (section 7.1)
• Addressing issues arising regarding compliance
  (section 7.2)
• Secretary of Treasury Board (TB) will monitor
  compliance through ongoing monitoring of
  evaluations and DEP, including evaluation
  coverage and quality of evaluations (section
  7.3), incl. Through
• MAF annual assessments
• Annual report to TB on the health of the
  evaluation function (section 6.3.1)
• In case of non-compliance, measures can include
• Measures allowed by the FAA that the TB would
  determine as appropriate to the circumstances
• Secretary of TB to request that the DH take
  corrective action and report back on results
  achieved

12
Head of Evaluations Responsibities

• Sections
• Section 6.1.1
• (phase-in until March 31, 2013)
• Section 6.1.2.a
• Section 6.1.2.e
• Section 6.1.2.c
• Section 6.1.2.d
• Section 6.1.3.a, c
• Section 6.1.4
• a)

• Directive Provisions
• Meets competency requirements, as specified by
  TBS
• Ensures that persons involved in conducting
  evaluation has (or collectively have) appropriate
  training, competencies and experience
• Manages evaluation units to ensure that
  evaluations are conducted in a neutral and
  cost-effective manner
• Submit evaluation reports directly to DH and the
  DEC
• Consult with Head of Comm. on work that could be
  POR
• Support the DEC
• Develop a rolling 5-year departmental evaluation
  plan, covering all DPS adm. Comp. of major
  statutory programs
• Risk-based approach for evaluation approach and
  level of effort, based on program risk and
  performance measurement
• Review and provide advice on
• performance measurement strategies to ensure that
  they effectively support an evaluation
• Accountability performance provisions in
  Cabinet documents
• MRRS PMF

13
Program Managers responsibities

• Directive Provisions
• Develop and implement ongoing performance
  measurement strategies for their programs
• Ensure that credible and reliable performance
  data are being collected to effectively support
  evaluation
• Develop and implement management response and
  action plan in a timely and effective manner
• Consult with HoE on the performance measurement
  strategies for all new and ongoing DPS

• Sections
• Section 6.2.1
• Section 6.2.1
• Section 6.2.2
• Section 6.2.3

14
Standard on Evaluation

• The Standard on Evaluation for the Government of
  Canada sets minimum standards for quality,
  neutrality, and utility of individual evaluations
  by communicating expectations to heads of
  evaluation and to evaluators regarding
• Protocols, governance and stakeholder involvement
  in individual evaluations
• Planning and designing individual evaluations
• Cost-effective and risk-based evaluation
• Standard on measurement and analysis
• Managing evaluation projects
• Ethically-based
• Team competencies
• Project management
• Information management, privacy and
  confidentiality
• Report-writing and use of evaluation reports
• Monitoring and Reporting

15
Small Departments Agencies policy requirements

• The policy instruments include a deferral of the
  application of the most demanding policy
  requirements to small organizations
• Deferred requirements include
• coverage requirement to evaluate all direct
  program spending over and above those programs
  subject to the legislative requirement of section
  42.1 of the Financial Administration Act
• DM responsibility, as deemed appropriate to the
  needs of the department or agency
• All ongoing programs of grants and contributions
  are to be evaluated very five years, as per
  section 42.1 of FAA
• the competency requirements for heads of
  evaluation
• the requirement for establishing a departmental
  evaluation committee
• the requirement for developing an annual
  departmental evaluation plan

16
Evaluation Coverage RequirementDirect Program
Spending - Detailed Definition

• Every 5-year, ensure evaluation of
• All direct program spending, which includes
• internal services, as a program cost
• Program of grants contribution as per section
  42.1 of FAA
• Administrative aspect of major statutory programs
• Programs set to terminate automatically, if
  requested by the Secretary of TB
• Specific evaluations, if requested by the
  Secretary in consultation with the affected DH
• ( internal services are not to be evaluated as
  a unit of analysis, unless DH decides to. They
  should be dealt as a component of a program,
  especially if significant to the delivery of the
  program.  The rationale is that it allows the
  performance of the program to be gauged against
  the full cost of delivering the program)
• Definition of Direct Program Spending
• Operating and capital budgets
• Grants and contribution expenditures
• Expenditures associated with programs that are
  deemed quasi-statutory
• Minor statutory program expenditures
• Vote-netted revenues and statutory revenues,
  which are expenditures funded through revenues
  generated by fees charged to clients inside and
  outside the federal government.
• Administrative costs associated with managing
  major statutory programs including major
  transfers to individuals and other levels of
  government, but exclude the payments themselves.

17
Supporting Policy Implementation

• Work continues to update guidance documents to
  support evaluation policy renewal and other new
  policies (e.g. MRRS Policy, Policy on Transfer
  Payment)
• Major pieces of work will include
• Guide to Evaluation Function
• Guide for preparation of departmental evaluation
  plans
• Performance and evaluation strategy guidance
  (replaces RMAF guide)
• Sharing of good practices/approaches to
  evaluation
• Working Group - Grant Contribution programs
• Working Group - Low-risk programs
• Working Group Policy programs
• Working Group Horizontal Initiatives
• Competency requirements for heads of
  evaluation/evaluators
• Training/Professional development for evaluators
  (CSPS, University Consortium)
• Recruitment initiatives

Tools
Capacity
18
Competencies for Evaluators
19
Why Competencies for Evaluators?

• The credibility of the evaluation function is
  directly linked to the competence of heads of
  evaluation and evaluators
• From diagnostic studies conducted in 2005, we
  know that deputies identified the lack of
  qualified evaluators as a contributor to the lack
  of quality and use of evaluation
• Evaluation in the context of the Canadian Federal
  government requires a consistently competent
  evaluation workforce in order to support
  credibility, quality, and ultimately use of
  evaluations
• Directive on the Evaluation Function includes
  requirement for heads of evaluation to meet
  competency requirements set by TBS
• Competency profiles can provide the foundation to
  guide other HR initiatives or tools (e.g. aid in
  identifying learning and development needs,
  inform recruiting and staffing)

20
Developing Competencies for Evaluators

• A draft set of competencies (March 2008) was
  developed and needs further refinement and
  broader, more thorough consultation
• Competencies profiles developed for junior,
  intermediate, senior evaluators as well as head
  of evaluation
• Current draft includes following five categories
  of competencies (with sub-categories)
• Intellectual (Cognitive Capacity, Creativity)
• Leadership (Organizational Awareness and
  Strategic Thinking, Action Management)
• Management (People and Financial Management)
• Relationship (Interpersonal, Communication,
  Teamwork / Partnering)
• Personal (Stamina and Stress Resistance
  Standards, Ethics and Value Professional
  Credibility Behavioural Flexibility)
• Consultant will be helping with this work over
  the next two months

21
Evaluation and Renewal of the Expenditure
Management System
22
Drivers of Expenditure Management Renewal

• Canada is renewing its Expenditure Management
  System to ensure that management excellence and
  fiscal credibility are integral to the system
• In June 2007, Expenditure Management System
  Renewal changed the way the government operates,
  so as to ensure
• Aggregate fiscal discipline (i.e. control of
  overall growth in spending)
• Effective allocation of all government resources
  to areas of highest relevance, performance and
  priority
• Efficient and effective program implementation

23
Key Pillars of the Expenditure Management System

• The renewed Expenditure Management System is
  supported by three key pillars
• All spending must be managed to results/outcomes
  that are transparent to Canadians
• Have clear measures of success
• Be formally assessed and evaluated systematically
  and regularly
• Demonstrate value for money
• For new spending proposals Up-Front Discipline
  to manage overall spending growth. New proposals
  must
• Include clear measures of success
• Provide solid information about how the proposal
  fits with existing spending
• Provide reallocation options for funding
• For existing spending Expenditure Reviews assess
  all existing programs and spending to ensure
• Alignment with overall priorities and results
• Relevance, effectiveness, efficiency and economy
• Decision-making relies on objective
  evidence-based information

24
The New Expenditure Management System What We
Had, Whats New and Improved
Systematic Review and Approval of Programs,
Funding and Results
Integrated Data Systems
Detailed Program and Performance Information for
Decision Making
MRRS Increasing Evaluation Coverage
Program Implementation, Evaluations and
Reporting
Gov. Expenditure Plan
Budget
Alignment
25
(No Transcript)