Measuring and Reporting Soft Outcomes

1
Measuring and Reporting Soft Outcomes

• Centre of Excellence for Evaluation
• November 2003

2
Measuring Soft Outcomes

• Does the TBS Evaluation Policy require they be
  measured?
• Is there a difference between traditional
  evaluations (eg regulatory programs, economic
  development, research, judicial, etc) and those
  that measure soft-outcomes?
• Are managers accountable for them?
• Is there an easier route to take?

3
What is an outreach intervention?

• Social engineering programs (eg health, crime,
  and the environment)
• Public outreach, products or services that
  involve one time or short duration contact with
  the learner
• Reaches beyond a select group of leaners and
  brings information to a much broader audience
• Seeks behavioural change by raising level of
  awareness, understanding, and change in attitudes
• Promotes the adoption of certain practices
  through
• a) indirect activities (eg advertising --
  television, radio, print campaigns)
• b) direct methods (eg workshops, training)
• c) incentive structures (promotions, discounts,
  etc..).

4
Soft Outcomes and Results Based Management --
Issues

• Departments are required to report on program
  results how can this be done in a soft outcome
  program model?
• Challenges
• Timing Interventions often of a short-term
  nature, seeking a long-term impact
• Size Level of impacts often beyond what the
  program can logically be considered to affect
• Attribution Circumscribing the program to
  isolate its impacts from externalities can
  be complex

5
Reporting on Soft Outcomes

• Proper focus of accountability (level of
  influence)
• Know at what level of the casual chain your
  accountability ends (setting the preconditions or
  affecting the change)
• Need to balance the demand to focus on results
  with logic of program intervention
• Can the intervention reasonably be held to
  account for relatively small contribution to
  overall the social issue being addressed?

6
Measuring Soft-Outcomes
Causal chain
Related Questions

• Target Group Defined
• Program intervention (eg transfer of knowledge)
• Awareness
• Understanding
• Attitudinal Change
• Behavioural change
• End result

• Is the right group being targeted?
• Is the intervention the most appropriate given
  the nature of the social problem being addressed?
• Does the intervention reach the target group(s)?
  (eg is the intervention actually being
  delivered?)
• What is the level of awareness being produced?
• Is this awareness accompanied by increased
  understanding?
• Does this knowledge result in changed attitudes?
  
• Does the attitude translate into changed
  behaviour?

7
Example Identifying the Level of Accountability

• Environment Canada

Our Vision Environment Canada's vision is to see
a Canada where people make responsible decisions
about the environment, and where the environment
is thereby sustained for the benefit of present
and future generations Our Mission Environment
Canada's mission is to make sustainable
development a reality in Canada by helping
Canadians live and prosper in an environment that
needs to be respected, protected and conserved.
8
Example Identifying the Level of Accountability
Environment Canadas Weather and Environmental
Predictions   Objective To help Canadians adapt
to their environment in ways which safeguard
their health and safety, optimize economic
activity and enhance environmental
quality.   Some activities provide info on
states of physical environment issue weather
warnings advice on adaptation to changing
weather and climate   Intended results changes
in public behaviour in response to warnings,
advice, etc.
9
Example Identifying the Level of Accountability
Business Line Health Promotion and
Protection Objective To provide an integrated
approach to the management of risks and benefits
to health by promoting healthy environments and
behaviours, protecting Canadians against risk
factors over which they have little control, and
by providing information and tools so that they
can make informed decisions about their health.
10
Defining Outcomes

• Clear statements of what the intervention should
  result in (eg what the learner should be able to
  do and understand as a consequence of their
  learning)
• Know at what level of the casual chain your
  accountability ends (setting the preconditions or
  affecting the change)

11
Results Chain
AREA OF INFLUENCE EXTERNAL TO THE ORGANIZATION
AREA OF CONTROL INTERNAL TO THE ORGANIZATION
Immediate Outcomes (Direct)
Intermediate Outcomes (Indirect)
Final Outcome
Inputs (Resources)
Activities
Outputs
EXTERNAL FACTORS
Level of Influence of External Factors
EFFICIENCY
EFFECTIVENESS
12
Example of a Logic Model Diabetes Prevention
Television Advertising Campaign
Economic Factors
Cultural Ethnic Factors
External Factors Influencing Outcome
Location
Immediate Outcomes (Direct)
Intermediate Outcomes (Indirect)
Final Outcome
Inputs (Resources)
Activities
Outputs

• Fewer complications for people with diabetes
• Lower incidences of diabetes

• Design and test advertisements
• Produce ads
• Purchase commercial time

• Advertisements are run on television

• People gain awareness and knowledge on diabetes,
  risk factors and prevention
• Attitudes affected

• Healthier eating habits
• Healthier lifestyle

13
Methods Principles

• Multiple lines of evidence
• Key lines of inquiry for each target group,
  stakeholders, and step in causal chain
• Ability to circumscribe causal chain (control for
  external variables)
• Recognize limits of resources and time
• Consistency of data while seeking
  improvements/innovations in data collection

14
Methods

• Anecdotal information (views of key stakeholders,
  program deliverers, etc)
• Survey data
• Client satisfaction data
• Pre and post intervention data of participants
  (eg class testing)
• Centrally collected statistics (eg Stats Cdn)
  (eg change in end result sought after)
• Statistical analysis (eg Correlation Analysis to
  circumscribe casual change)
• Indirect Indicators (eg increase or reduction in
  complaints)

15
Measuring Soft-Outcomes
Causal chain
Related Questions
Measures

• Target Group Defined
• Program intervention (eg transfer of knowledge)
• Awareness
• Understanding
• Attitudinal Change
• Behavioural change
• End result

• Is the right group being targeted?
• Is the intervention the most appropriate given
  the nature of the social problem being addressed?
• Does the intervention reach the target group(s)?
  (eg is the intervention actually being
  delivered?)
• What is the level of awareness being produced?
• Is this awareness accompanied by increased
  understanding?
• Does this knowledge result in changed attitudes?
  
• Does the attitude translate into changed
  behaviour?

• Types of audiences being reached?
• Percent of target audience being reached?
• Percent of audience that is aware of issue?
• Percent of audience that knowledgeable about the
  issue or demonstrated capabilities, self-assessed
  learning, level of recall)
• Percent of audience that feels/believes in the
  suggested change?
• Percentage of households that affect the change
  or self-assessed practice and behaviour
• Actual number of incidences (change over time)?

16
Possible Suggestions for Reporting on Soft
Outcomes

• Identify what you know and what you need to know
• Implement a performance measurement strategy that
  uses short term monitoring of program activities
  and outputs to inform whether program is
  impacting on final outcome
• Monitor long term impact on a periodic basis
• Use methods than enable you to circumscribe your
  program

17
Thinking Strategically about Soft Outcomes

• Work to link program activities and outputs to
  strategic outcomes being pursued by department
• For example
• EC business line controlling and preventing
  pollution in order to secure for Canadians a
  Clean Environment
• How do program activities contribute to this?

18
Tools to Report on Soft Outcomes

• DPRs what progress is being made on achieving
  short term outcomes? How is the program
  contributing to departmental strategic outcomes?
• RMAFs developing a logic model that shows
  linkages between activities and all stages of
  outcomes plans to attribute program impact on
  ultimate objectives
• Annual Program Reports
• Periodic Evaluations

19
(No Transcript)