Title: Planning For Evaluations Module 2
1Planning For Evaluations Module 2
- Essential Skills Series
- An Introduction to Evaluation Concepts and
Practice - Canadian Evaluation Society
Date_________________ Location______________
2Workshop Agenda
Registration 830 - 900 a.m. Evaluation
Frameworks 900 - 1000 a.m. Logic Models
000 - 1015 a.m. Break
1015
- 1030 a.m. Logic Models (continued)
1030 - 1200 p.m.
Small group exercises Lunch 1200 - 100
p.m. Performance Measurement Plan 100 - 215
p.m. Small group exercises Break 215 - 230
p.m. Performance Measurement Plan (continued)
230 - 300 p.m. Evaluation Plan and Reporting
Strategy 300 - 400 p.m. Planning and
Managing Frameworks 400 - 420 p.m. Group
discussion Workshop Evaluation 420 - 430
p.m.
3Workshop Objectives
- Understanding of evaluation frameworks
- Understanding of logic models
- Awareness of performance measurement plans
- A basic understanding of evaluation planning and
reporting - Awareness of principles for managing frameworks
4Section 1.
Evaluation Frameworks
5 Planning For Evaluations
- Why Plan?
- Increase understanding of evaluation process and
rationale - Enhance relevance of evaluation for
decision-making - Build support for ongoing performance measurement
and evaluation - Help ensure data are available for eventual
evaluation - Make the evaluation process manageable
- Facilitate the coordination of efforts
- When Does Planning Take Place?
- Ideally, when the program or policy is being
designed (evaluation framework) - Just before the evaluation is conducted
6Evaluation and the Management Life-cycle
Design an intervention
Initial Situation
Reconsider, redesign, expand, reduce or end
Identify a need
Program/Policy start-up
FINAL OUTCOMES
IMMEDIATE OUTCOMES
INTERMEDIATE OUTCOMES
Summative/Outcome/ Impact Evaluation
Formative/Mid-term/Process Evaluation
Needs Assessment
Baseline
Evaluation Framework
Evaluability Assessment/ Planning
Ongoing Performance/Outcome Monitoring/Measurement
Source Adapted from Birch-Jones, J.,
Integrating PM and Evaluation Bridging the
Chasm, CES-NCR, 2002.
7Preliminary Considerations for Evaluation Plan
Development
- Why the evaluation is being conducted
- Who the client is for the evaluation
- The decisions the evaluation is intended to
support - The evaluation questions that will provide the
evidence to help with the decisions - The outline of linkages between inputs,
activities, outputs, and outcomes for the
proposed policy, program, initiative, or function
(e.g., logic model, cause-and effect, and
implementation theories) - Performance indicators and availability of
performance information - The methods that would be appropriate for the
evaluation - Time lines
- Resource implications
- Source Health Canada (July 2008) Evaluation
Project Workplan Assessment Guide
8Why is the Evaluation Being Conducted? And Who is
the Client?
- What decisions are to be taken?
- Accountability?
- Learning?
- Who is the client / are the clients?
- Internal vs. External
- Staff and operational management vs. senior
management - Line vs. policy
- Others
9Whats in a Framework?
- Key Elements
- Profile/background/context
- Results logic
- Evaluation Issues/Questions
- Performance Measurement Evaluation Strategy
- Framework is a plan for an evaluation
- Blueprint for results-based management
- Provides information on what a program, policy or
initiative is expected to achieve and how this
will be demonstrated
10 Evaluation Frameworks
- One key to a good framework is understanding the
need for the evaluation - The need for an evaluation may come from
- statutory/funding requirements
- program sunset/renewal/redesign
- senior management concern
- combination
- Rationale for evaluation will influence scope and
focus - knowing managements rationale and level of
support for the evaluation is key strategic
information
11Evaluation Frameworks
- Another key to a good framework is understanding
the program, policy, or initiative itself - How do you find out about a program?
- review documents (reports, other studies, files,
legislation) - conduct a review of literature
- browse the website (if applicable)
- examine program data (files and databases)
- conduct interviews (managers, staff, clients)
- observe program/staff in action
- What do you find out?
- how the program works
- whats important to the program
- whats working well and whats not
- potential evaluation issues
- key information upon which to base the profile
12Evaluation Frameworks Profiles
- Whats in a Profile?
- The profile section
- describes the policy, program or initiative
- provides context and rationale
- provides a clear picture of what the policy or
program or initiative is intended to achieve and
how it intends to do so - Concise description of
- origin and rationale (demonstrated need for the
program, policy or initiative) - resources allocated and how these will be used
- key stakeholders including delivery partners and
primary intended beneficiaries (clients or target
population) - objectives of the program, policy or initiative
and how these link to the organizations
strategic outcomes
Source Treasury Board Secretariat (2005).
Preparing and Using Results-based Management and
Accountability Frameworks. http//www.tbs-sct.gc.c
a/eval/pubs/RMAF-CGRR/guide/guide_e.pdf. Accessed
March 17, 2009.
13Evaluation Frameworks
- Tips for Developing a Good Profile
- Keep in mind that you are trying to describe the
program, not evaluate it (yet) - Use the key source documents available
- Avoid cheerleading in the profile use neutral
wording - Have program staff write the profile evaluator
provides advice/suggestions - Be prepared to clarify objectives/goals that are
not clear - Revisit the profile after developing the logic
model to ensure consistency between stated and
actual outcomes (rhetoric versus reality)
14Section 2.
Logic Models
15Logic Models
- A logic model identifies the linkages between the
activities of a policy, program or initiative and
the achievement of its outcomes - Serves as a road map, showing the chain of
results connecting activities to the final
outcomes and what progress looks like along the
way - Logic models test whether a policy, program or
initiative makes sense from a logical
perspective - Provides a fundamental backdrop on which the
performance measurement and evaluation plans are
based
Source Treasury Board Secretariat (2005).
Preparing and Using Results-based Management and
Accountability Frameworks. http//www.tbs-sct.gc.c
a/eval/pubs/RMAF-CGRR/guide/guide_e.pdf. Accessed
March 17, 2009.
16Logic Models Key Elements
- Inputs The financial and non-financial
resources used to produce outputs and accomplish
outcomes. - Activities An operation or work process
internal to an organisation, intended to produce
specific outputs (e.g. products or services).
Activities are the primary link in the chain
through which outcomes are achieved. - Outputs Direct products or services stemming
from the activities of a policy, program, or
initiative, and delivered to a target group or
population. Usually things you can count. - Outcomes An external consequence attributed to
an organisation, policy, program or initiative
that is considered significant in relation to its
commitments. Outcomes may be described as
immediate, intermediate or final (end), direct or
indirect, intended or unintended. A good outcome
statement represents the type of change wanted,
includes reference to the target population or
intended beneficiary and does not include
reference to the how.
Source TBS Results-based Management Lexicon
http//www.tbs-sct.gc.ca/rma/lex-lex_e.asp
Accessed Dec 15, 2008
17Logic Models Flow Chart Example
Inputs/
18Logic Models Flow Chart Example (2)
Source Adapted from An Evaluation Framework
for Community Health Programs, The Center for
the Advancement of Community Based Public Health,
2000.
19Logic Models Results Chain
Area Of Control Internal to the Organization
Area of Influence External to the Organization
Immediate Outcomes (Direct)
Intermediate Outcomes (Indirect)
Ultimate Outcome
Inputs (Resources)
Activities
Outputs
External Factors
Efficiency
Effectiveness
Source Treasury Board Secretariat,
Results-based Management and Accountability
Framework Guidance, 2001 www.tbs-sct.gc.ca/eval/pu
bs/RMAF-CGRR/rmafcgrr_e.asp
20A Logic Model With Assumptions and Factors
Source http//www.uwex.edu/ces/pdande/evaluation/
evallogicmodel.html Retrieved November 14, 2008
from the University of Wisconsin-Extension
website.
21Program Model Components (Recap)
OUTCOMES Short-Term ? Long-Term
OUTPUTS
INPUTS
ACTIVITIES
Resources placed into the program
Core programtasks
Products services of the program
Impact or effectiveness of the program
Examples Money Staff Operational
expenses Capital assets
Examples Change in Knowledge/awareness
Skills Attitude/opinions ?
Behaviour ? Condition Status
Morbidity, mortality
Examples Teaching Presentations
Counselling Mentoring Treatment
Examples materials distributed
classes taught sessions conducted hours
of service delivered -------------------------
--------- of participants (reach)
Source Adapted from Measuring Program Outcomes
A Practical Approach. United Way of America, 1996.
22Benefits of the Logic Model
- Helps with understanding overall structure,
function of program as well as rationale behind
activities - Helps to ensure that program activities and
intended results correspond - Helps identify key questions for the evaluation
- Helps communicate the elements of the program to
policy makers, staff, external funding agencies,
media, and colleagues - Helps to reveal where steps in the program break
down
Source Adapted from K. Farell et al.,
Evaluation made very easy, accessible, and
logical, 2002.
23Limitations of the Logic Model
- Initially time consuming (days/weeks/months)
- Requires patience
- Does not always capture all aspects of the
program (e.g., program costs may not be included
in the model) - May not represent all external influences and
factors
Source Adapted from K. Farell et al.,
Evaluation made very easy, accessible, and
logical, 2002.
24 Characteristics Of Good Logic Models
- Activities and outputs are distinct from
outcomes the general rule is that if you control
it, its an activity/output, if you can only
influence it, then its an outcome - Outputs demonstrate that you are busy
- Outcomes have a distinct who and what, what
change in whom? - Outcomes demonstrate that you are making a
difference - Outcome statements are simply worded, contain
only one outcome, and no hows - The linkages between outputs and outcomes are
clear - Outcomes are not tied to particular timeframes
- No more than 2 or 3 final outcomes
- Linked to strategic outcomes and program goals
but reflect reality
Sources US GPRA Guidance 1999, Mayne (various),
Montague Focusing on Inputs, Outputs and
Outcomes, Canadian Journal of Program Evaluation,
2000
25Outcome Examples
Type of Change Illustration
Change in circumstances Children safely reunited with their families of origin from foster care
Change in status Unemployed to employed
Change in behaviour Truants will regularly attend school
Change in functioning Increased self-care getting to work on time
Change in attitude Greater self-respect
Change in knowledge Understand the needs and capabilities of children at different ages
Change in skills Increased reading level able to parent appropriately
Maintenance Continue to live safely at home (e.g., the elderly)
Prevention Teenagers will not use drugs
Source Patton, M.Q., Utilization-Focused
Evaluation The New Century Text, 1997.
26Thinking About What to Include Where
- Some program components are difficult to classify
as activities, outputs, or outcomes. These
examples provide general guidelines (although
exceptions may be appropriate) - recruiting and training staff and volunteers,
purchasing or upgrading equipment, and various
support and maintenance activities are usually
foundational (and not on a logic model) - number of participants served is sometimes an
output, but if participation is discretionary,
could be considered an early outcome - participant satisfaction is usually an early
outcome
27Where to Start?
- There is no single way to create a logic model.
Where you start often depends on the
developmental stage of the program - Strive for simplicity and dont be
over-inclusive. Dont include all the
implementation details. Try to fit the whole
logic model on one page. - Discuss the logic model with staff involved at
all levels in the program (or involve them in a
logic model development workshop) - Post-it notes are a great tool for logic model
development
Source Adapted from Porteous, N.L. et al.,
Program Evaluation Tool Kit A Blueprint for
Public Health Management, 1997.
28Small Group Exercise 1
- Educational grants program (grants provided to
students for post-secondary education, targeting
low-income population) - What would be some outputs?
- What would be some immediate outcomes?
- What would be some intermediate outcomes?
- What would be some final outcomes?
29Small Group Exercise 2 Building an Evaluation
Framework
- Clarifying the Programs Logic and Theory of
Change - The problem gambling needs assessment found that
gambling has become a very important pastime for
young adults aged 18-24 years. Analysis of
Rockwoods databases showed a steady increase in
the number of young adults with gambling
problems. Focus groups painted a bleak picture
although casinos, lotteries and mass media are
aggressively marketing gambling to young adults,
the message has not gotten out to young adults
about the dangers of problem gambling and the
importance of seeking prompt treatment. In fact,
a recent study has found twice as many young
adults aged 18 to 24 developed severe or moderate
gambling problems as the overall population.
Other findings, such as the fourfold rise between
2001 and 2005 of young adults now playing online
poker, indicate this trend is likely to continue
or increase. - As one response, the Rockwood Program Planning
Team decided to offer a Gambling Prevention
Program for young adults. The prevention program
intends to make young adults aware of dangerous
gambling behaviour (such as skipping class or
work to gamble) in themselves or their friends,
help them to see gambling as a form of occasional
entertainment rather than a glamorous and
lucrative lifestyle, minimize serious financial
problems, and encourage young adults with
gambling problems to get help when necessary. - Program activities include staging gambling
awareness events in schools featuring
celebrities, placing edgy public service
announcements in the mass media, displaying
posters in malls and on mass transit that
discourage gambling, and having booths about
problem gambling at health fairs and staffing
them with trained peers who will provide advice
in a non-threatening and non-judgmental way.
30Small Group Exercise 2
- Using case study material
- Select a logic model template
- Identify some key activities and outputs,
immediate outcomes, intermediate outcomes and
final outcomes - Present them in a logic model format
- Spend a few minutes discussing what you found
challenging when developing your logic model
31Small Group Exercise 2 Worksheet
32SECTION 3.
Performance Measurement Plan
33 Performance Measurement Plan
- A performance measurement plan identifies
- What needs to be measured on an ongoing basis
- How
- How often, and
- What is the cost of the performance measurement
plan (where possible) - Use performance monitoring to produce yearly
performance reports of program results for
funders, boards, senior managers, staff, and
public - Performance monitoring systems are only tools.
When properly developed and used, they can reveal
problems, point to solutions and are a check on
the effectiveness of solutions once implemented - Performance measurement plans help organizations
to avoid the tendency to concentrate on the
things that are easiest to measure, such as The
number of people trained (service process
measure) rather than The number of people with
jobs (outcome measure)
34 Performance Measurement Plan (2)
- Performance indicators show whether an output was
produced or a specific outcome was achieved
(success) - Describes specific, observable, measurable
characteristics or changes that represent
achievement of an output or outcome - Can be quantitative or qualitative (mix of both
is good) - Indicators include
- number/percent of ...
- incidence of ...
- proportion of
Source Hatry, Harry P. (1999). Performance
Measurement Getting Results. The Urban Institute
Press, Washington, D.C.
35Examples Of Indicators
Smoking Cessation Program
- Indicators
- Number and percent of participants who report
that they have quit smoking by the end of the
class - Number and percent of participants who have not
relapsed 6 months after program completion
- Outcome
- Participants stop smoking
Information and Referral Service
- Outcome
- Callers access services to which they are
referred or about which they are given information
- Indicators
- Number and percentage of community agencies that
report an increase in new participants who came
to their agency as a result of a call to the
information and referral line
Source Outcome Measurement Showing Results in
the Nonprofit Sector, United Way of America, 1999.
36Examples Of Indicators (2)
37Examples Of Indicators (3)
- Number and percent of newborns weighing at least
5.5 pounds and scoring 7 or above on APGAR scale - Number and percent of teen mothers who graduate
from high school over a 4-year period - Percent of participants who recall content of
brochure, posters, or presentations - Number of action plans developed by participants
- Number and percent of participants who have not
relapsed six months after program completion - Number and percent of participants who
demonstrate increase in ability to read, write,
and speak English by end of program - Number and percent of youth who return home
- Number and percent of teen mothers using a
recommended form of birth control - Number and percent of legislators who voted in
support of your position before and after an
advocacy campaign
Source Outcome Measurement Showing Results in
the Nonprofit Sector, United Way of America, 1999.
38 Selecting Indicators
- Things to think about
- Useful to decision makers
- Decisions to be based on results information
- Understandable to everyone
- Clearly defined
- Measurable
- Use proxy indicators when necessary
- Obtain baseline data and set targets (where
possible) - Involve stakeholders
- Use quantitative and qualitative indicators as
appropriate - Try to limit the number of indicators
39Criteria For Good Performance Indicators
- Relevance
- Importance
- Understandability
- Program influence or control over the outcome
- Feasibility
- Cost of collecting the indicator data
- Uniqueness
- Manipulability
- Comprehensiveness
Source Hatry, Harry P. (1999). Performance
Measurement Getting Results. The Urban Institute
Press, Washington, D.C.
40Measurement Through Monitoring
- A monitoring system is typically key for
performance measurement - Key monitoring system steps
- Step 1 Obtain Top-level Support
- Step 2 Develop Logic Model
- Step 3 Select Measures
- Step 4 Decide When to Measure
- Step 5 Select a Data Collection Method
- Step 6 Implement the Plan
- Step 7 Analyze Data
- Step 8 Use Data
41Strengths and Weaknesses ofMonitoring Systems
- STRENGTHS
- Provides information about process and key
results - Easy to understand and use
- Improves accountability for effective programs
- Improves program management
- Provides early warning about potential problems
with achievement of objectives - Facilitates evaluation
- WEAKNESSES
- Depends on setting realistic and appropriate
measures - Requires good grasp of client outcomes
- Can be expensive and time-consuming to collect
data and to maintain the system - Does not establish causal relationship between
services and outcomes (Describes rather than
explains) - Difficult to compare programs in different
settings, since indicators provide little
contextual information
42Small Group Exercise 3
Gambling Education Program
For each performance indicator listed have your
group identify whether it is a PROCESS or OUTCOME
indicator.
Type of Indicator Process Outcome
1. Thirty students attend the gambling
education workshops. _____ _____ 2. Six group
workshops are conducted. _____ _____ 3.
Students awareness of problem gambling
increases. _____ _____ 4. Students behave
in a way that reduces the risk of developing
gambling problems. _____ _____ 5. Students
participate in role plays and group
discussions. _____ _____
43 Small Group Exercise 4
- Thinking about your logic model components for an
educational grants for students program - For the immediate, intermediate and final
outcomes developed in the previous exercise,
develop at least 1 indicator for each - Discuss how this information could be used
- Spend a few minutes discussing what you found
challenging about identifying indicators
44 Small Group Exercise 4 Worksheet
Educational Grants Program
45Small Group Exercise 5
- Using the case study material and your logic
model - For the immediate and intermediate outcomes
developed in the previous exercise, develop at
least one indicator for each outcome - Discuss how this information could be used
- Spend a few minutes discussing what you found
challenging about identifying indicators
46 Small Group Exercise 5 Worksheet
Case
47Performance Measurement Plan
Expected Outputs and Outcomes (1) Expected Outputs and Outcomes (1) Indicator (2) Baseline Measure (3) Data Source (4) Collection Method (5) Responsibility for Collection (6) Timing / Frequency of Measurement (7) Timing / Frequency of Measurement (7) Timing / Frequency of Measurement (7)
Expected Outputs and Outcomes (1) Expected Outputs and Outcomes (1) Indicator (2) Baseline Measure (3) Data Source (4) Collection Method (5) Responsibility for Collection (6) Ongoing Measure-ment Formative Evaluation Summative Evaluation
Outputs Output 1
Output 2
Output 3
Output x
Outcomes Immediate Outcome 1
Immediate Outcome 2
Immediate Outcome x
Intermediate Outcome 1
Intermediate Outcome x
Final Outcome 1
Final Outcome2
Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs Estimated Performance Measurement(PM) Costs
Source Health Canada (2006) Results-based
Management and Accountability Framework
Assessment Guide
48Section 4.
Evaluation Plan
49 Evaluation Plan
- The evaluation plan sets out a strategy for a
periodic, in-depth look at how well a policy,
program or initiative is doing - It identifies
- the key evaluation issues/questions that should
be addressed - the key pieces of information that need to be
collected to answer the evaluation questions - the data collection plan
- potential costs for an evaluation
- suggested timing for evaluation
- reporting requirements
- Builds on a performance measurement plan
- adds context and depth to information being
collected through ongoing performance
measurement.
50 Evaluation Plan Issues and Questions
- Evaluation questions usually relate to 3 kinds of
issues - Relevance
- Does the policy, program or initiative continue
to be consistent with organizational priorities,
and does it realistically address an actual need? - Success (Effectiveness)
- Is the policy, program or initiative effective in
meeting its intended outcomes, within budget and
without unwanted negative outcomes? Is the
policy, program or initiative making progress
toward the achievement of the final outcomes? - Cost-effectiveness and alternatives
- Are the most appropriate and efficient means
being used to achieve outcomes, relative to
alternative design and delivery approaches? - Evaluation also addresses design and delivery
issues
51 Evaluation Plan Issues and Questions (contd)
- Some issues and questions may be more relevant
during an early evaluation (i.e., formative
evaluation), some later (i.e., summative
evaluation), and others may be relevant to both - The key to a good evaluation plan lies in linking
the evaluation issues and questions back to the
program rationale (the need) and logic model
(what success will look like) - Mid-term evaluation usually looks at
implementation and delivery issues (for a new
program), as well as early outcomes, and the
adequacy of the performance measurement plan - Data from a performance measurement plan can
contribute to questions about design and delivery
as well as early outcomes - Summative evaluation usually looks at performance
information on immediate and intermediate
outcomes, as well as their contribution to final
outcomes - Summative evaluations may also look at relevance
and cost-effectiveness and alternatives, as well
as any other important issues that are identified
52 Evaluation Plan Issues and Questions (contd)
- Evaluation issues are often identified by program
staff and stakeholders - Interviews with a small number of program staff,
senior management, partner organizations, and
clients is an excellent way to identify potential
evaluation issues - At this point, will have a long list of
potential evaluation issues that will need to
be prioritized and any nice to know/less
important issues eliminated
53 Evaluation Plan Data Requirements
- Based on the evaluation issues/questions, the
evaluation data requirements can be identified - these are the key pieces of information that need
to be collected to answer the issue/question - if derived from the logic model, might already
have a measurement plan (e.g., for issues related
to success) - for new issues, will need to identify indicators
and a data collection strategy - Once developed, can select the best set of
indicators, based on consideration of - reliability, validity and credibility of the
indicator - cost-effectiveness (in terms of cost to collect
and to process) - whether it is directly linked to the evaluation
question - Data collection strategy identifies
- indicator
- data source
- method of data collection
- frequency of data collection
- potential costs for the evaluation
- Similar to performance measurement plan but more
comprehensive
54Evaluation Information Summary Table (Example)
Evaluation Issues and Questions (1) Indicator (2) Baseline Measure (3) Data Source (4) Collection Method (5) Collection Method (5) Responsibility for Collection (6) Responsibility for Collection (6) Timing / Frequency of Measurement (7) Timing / Frequency of Measurement (7) Timing / Frequency of Measurement (7)
Evaluation Issues and Questions (1) Indicator (2) Baseline Measure (3) Data Source (4) Collection Method (5) Collection Method (5) Responsibility for Collection (6) Responsibility for Collection (6) Ongoing Performance Measurement Formative Evaluation Summative Evaluation
Relevance
Question 1
Question 2
Implementation
Question 1
Question... x
Effectiveness
Question 1
Question 2
Question ...x
Cost Effectiveness
Question 1
Question 2
Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues Expenditure Review Committee Questions / Issues
Question 1
Question... x
Estimated Costs
Source Health Canada (2008) Evaluation Project
Workplan Assessment Guide
55Section 5.
Reporting Strategy
56 Reporting Strategy
- Identifies what plans are in place to
systematically report on the results of ongoing
performance measurement and evaluation - Includes information on what, where and when, for
example - Baseline measurement - (typically no report)
- Ongoing performance measurement - Annual Report
- Formative evaluation study - Mid-term Report
- Summative evaluation study - Final Evaluation
Report - Think about who you are reporting to
- Senior decision-makers, central agencies, funders
- Program managers
- Stakeholders
- Clients/communities
- May also be useful to incorporate other reporting
requirements which may draw on the same evidence
(e.g., federal government Departmental
Performance Reports and Reports on Plans and
Priorities)
57Small Group Exercise 6
- Considering your case, logic model and indicators
as well as the typical issues shown both in
this session and in module 1 (slides 10, 36 and
37) - What would be some issues, indicators, sources
and methods? (Address other elements of the
Evaluation Information Summary Table if you wish.)
58Section 6.
Planning and Managing Frameworks
59 Managing Frameworks Lessons Learned
- Allow for some months of elapsed time between
start and finish of an evaluation framework and
plan - Managers might not see the difference between
performance measurement and evaluation - Past experience will influence their perception
of what it is/entails - results reporting, balanced score card, deputy
dashboard, etc. - traditional frameworks and evaluations
- May need to adjust perceptions to new way of
thinking and approach to performance measurement
and evaluation - Need to clarify terminology (early and often)
60 Managing Frameworks Lessons Learned (2)
- Need to clearly define roles and responsibilities
early on - Decide on management structure for the framework
and plan - Working Group/Steering or Advisory Committee
- internal/external
- who is lead?
- who reviews what?
- who signs off/approves?
- The simpler the co-ordinating mechanism, the
better
61 Managing Frameworks Lessons Learned (3)
- Logic model is the key to bridging the chasm
between activities and reporting on outcomes - Facilitated inclusive approach to logic model
development is key - Takes longer but critical to understanding and
buy-in - Requires a unique approach - developmental
facilitation - Can involve staff, Board, partners, funders,
clients, users, regulatees - OK to let them know that we are all learning this
together - Involve them early on in the process
- Partners and co-deliverers need to understand and
agree on performance logic and measures before
they can provide results information - The framework doesnt have to be perfect will
evolve as program capacity grows - Need to review and adjust performance measurement
system over time - Implementation of a framework is key otherwise,
it is back to outputs, opinions and anecdotes for
the evaluation - Development of a framework is only the first step
in a lengthy process
62 Managing Frameworks Lessons Learned Group
Discussions
- Do you have any lessons learned to share around
managing evaluation frameworks?