Evaluating success of research to action

1
Evaluating success of research to action

• How to evaluate whether research was translated
  into action in the community, practice and policy
  environment.
• How to measure
• impact,
• direct and indirect outcomes,
• success or failure,
• short term or long term benefits, etc..
• How can we learn from our and others mistakes,
  communicating successes and failures.

Jerry Spiegel MA MSc PhD Director,
Global Health
Director, Liu Institute for Global Issues,
Centre for International Health University of
British Columbia jerry.spiegel_at_ubc.ca
2
Learning objectives

• Be familiar with program evaluation approaches
  techniques
• Be familiar with resources that can be used
• Consider particular viewpoints for KT evaluation
• Develop a network and approach for mutual
  learning in an emerging community of practice

3
Benefit of a team approachTransdisciplinary
Intersectoral
4
KT Cycle Framework
Researchers Knowledge Users
5
Knowledge Translation
INDIVIDUAL
SYSTEM
6
Knowledge Translation
INDIVIDUAL
Knowledge
Practice
Policy
SYSTEM
7
Research Into Action

• A framework for research / action interaction

Action Into Research
Knowledge t
Reality t
RESEARCH Review (incl. relevance) Research
question Hypothesis Methodology Results /
Analysis Discussion - implications

• Stakeholders
• Individuals / communities
• Practitioners
• Policy

ACTION
Reality t1
Knowledget1
8
University Community Expectations of Academic
Researchers
9
Program Evaluation

• Overview
• Steps in evaluation
• Evaluability assessment
• Developing a logic model
• Specifying indicators outcomes

10
What Is Program Evaluation?

• Evaluation is a process of determining the
  merit, worth and value of things, and evaluations
  are the products of that process. Scriven (1981)
• Comparison of an object of interest against a
  standard of acceptability. Green (1974)

11
Why evaluate?

• Determine achievement of objectives
• Improve program implementation
• Answer to stakeholders
• Increase support for a program
• Contribute to scientific base of knowledge
• Inform policy decisions

Why evaluate KT?

• Integration of KT in all research
• Evaluation of specific KT techniques

12
Types of Evaluation Activities

• Needs assessment - conditions a program is
  intended to address
• Assessment of program theory - conceptualization
  and design
• Assessment of program process - operations,
  implementation, delivery
• Impact assessment - outcomes, impact
• Efficiency assessment - cost, cost effectiveness

13
Social programs are inherently inhospitable
environments for research purposes.

• Weiss, 1972

14
Evaluation A rational enterprise in a
political context

• Policies/programs are creatures of political
  decisions
• Evaluation reports are part of the political
  arena
• Evaluation itself represents a political stance

15
Steps For Conducting an Evaluation

• Step 1 Engage stakeholders
• Step 2 Describe the program
• Step 3 Focus the evaluation design
• Step 4 Gather credible evidence
• Step 5 Justify conclusions
• Step 6 Ensure use and share lessons learned
• CDC (US) Evaluation Working Group
• http//www.cdc.gov/eval

16
What is evaluability assessment?

• Process for clarifying program design
• Determines if a program can be evaluated
• Useful for planning an evaluation
• Two products
• 1. Program description
• 2. Program logic model

17
Step 1 Program Description

• To evaluate must have a clear description of
  program
• Review available documents
• Interview staff, managers
• Concise description with focus on key elements

18
Elements of Program Description

• Need or problem being addressed
• Purpose and rationale for program
• Programs organizational structure
• Programs goals and objectives
• Major activities and components
• Target population

19
Step 2 Develop a Logic Model

• Logic model is a diagram that shows how program
  works
• Typically diagramed on a flow chart with program
  components, activities, goals, outputs, and
  outcomes
• Describes cause and effect relationships

20
Step 2 Logic Model cont.,

• Can help identify problems in design of programs
• Useful framework to examine outcomes
• Identifies program components that need to be
  tracked
• Prepares evaluator for designing the evaluation

21
Advantages of Logic Models

• Clarifies program assumptions
• Clarifies goals and objectives
• Relates activities to goals/objectives
• Facilitates agreement about indicators of success
• Determines if program has potential to achieve
  outcomes
• Prepares staff/evaluator for evaluation

22
Disadvantages of Logic Models

• Can be tedious
• Cannot replace good program planning
• Sometimes difficult to get consensus
• Programs often have little documentation about
  program
• Requires skill in specifying indicators and
  outcomes

23
How to Construct a Logic Model

• Talk to key stakeholders
• Clarify program rationale
• Describe program components and activities
• Define goals of components
• Determine program outcomes
• Specify short term and long term outcomes
• Verify logic model

24
Basic Elements of a Logic Model

• Components groups of closely related program
  activities
• Activities identifiable program tasks
• Goals General statements about desired program
  direction
• Outputs products of program activities

25
Basic Elements of a Logic Model cont.

• Outcomes the results program hopes to achieve
• Outcome indicators each outcome should have
  clearly defined markers of success (specific,
  measurable, relevant)

26
What is an indicator?

• Indicators are the specific items of information
  that describe observable, measurable
  characteristics or changes in corresponding
  outcomes
• It is specific, observable, and linked to desired
  outcome
• Examples
• 60 of program participants report they know
  where to go for vision care
• 25 of program participants report they used
  vision care in the past year

27
Example Outcome with Single Indicator
Improved Academic Performance
Outcome
Improved Grades
Indicator
Official StatisticsImprovement in participant
GPA overtime
Potential Method for Data Collection
Is this a valid and reliable indicator? Not
everything that can be COUNTED necessarily COUNTS
(i.e. is important)!
28
For DiscussionGeneral Evaluation Logic Model
UBC Undergraduate Program
Inputs
Process
Outputs

• Formal Curriculum
• I. Orientation
• II. Human biology
• Foundations medicine
• DPAS
• Clinical skills
• Family practice
• III.Rural underserved community practice
• IV.Clinical Clerkship
• V. Advanced clinical electives/ selectives
• VI. Learning skills for practice

• Curriculum
• Faculty
• Students
• Facilities
• Funding
• Other?

• Completion of UGP
• Competencies
• Qualifying
• exam
• Degree
• Other?

• Shortage of MDs
• Lack of access to care
• Health disparities
• Expansion UBC UGP

Context

• Informal Curriculum

• Hidden Curriculum

Process Implementation Evaluation
ImmediateOutputs
29
For DiscussionGeneral Evaluation Logic Model
UBC Undergraduate Program
Products
Working in academic/ research settings

• Completion of residency
• placement
• Generalists
• Specialists
• Peer review
• Certification
• Licensing
• Research training
• Other?

Academic Physicians

• Working in clinical practice
• Setting
• BC/Canada
• Rural/urban
• Aboriginal
• Other?

Reduced Health Disparities in BC
Inputs
Process
Immediate Outcomes
Context
Access to clinical care
Public Health Infrastructure
Enhanced Community Capacity
Intermediate Outcomes
Long-term Outcomes
Goals
2 February 2004
30
Model to evaluate the effectiveness of the Cayo
Hueso intervention
Inputs
Outputs
Results
Impacts
Needs reduced
MATERIALS - ministries - residents
Averted costs
HOUSES REPAIRED STREET REPAIRS LIGHTS CULTURAL
ACTIVITES WATER SUPPLY CONNECTIONS SOLID WASTE
COLLECTED
Improved housing conditions
LABOUR - ministries - residents
Improved Health
Improved cultural life in community
ADMINISTRATIVE TIME, TRAVEL etc
Improved safety / feeling of security
Improved satisfaction and quality of life
Community Support
Improved living conditions in community
At individual and community level
Community Mobilization
Community integration maintained and enhanced
Social cohesion enhanced
31
The Results Chain
32

• A result can be
• An improvement (e.g. in the health conditions)
• An increase (e.g. of the revenues of a given
  community)
• A strengthening (e.g. of the capacities of local
  partners)
• A reduction (e.g. in the infant mortality rate)
• A transformation in the attitudes, the practices
  of behaviour of a given group (e.g. adoption of
  contraceptive methods by the women of a region
  and acceptance of birth-control methods by the
  men)

A KT result can be
33

• Impacts
• Impacts are long-term socio-economic results that
  usually happen beyond the life of the project and
  are attributable to the achievements of outcomes.
• Impacts involve changes in living conditions of
  populations in developing countries.
• Impacts are the changes necessary to achieve the
  project goal.

34

• Outcomes
• Outcomes are specific measurable institutional
  and community level results that will be produced
  by the end-date of the project.
• Outcomes, which are the logical results of
  project outputs, include new programs or
  processes that will be sustained after the
  project completion.
• Outcomes are the changes necessary to achieve the
  project purpose.

35
Review
36
Review (continued)
37

• Outputs
• Outputs are the measurable results necessary to
  produce the outcomes.
• Outputs are short-term consequences of completed
  activities.

38
Role Play

• Study Advisory Committee
• Member of Government Ministry
• Member of local community NGO
• Practitioner
• Researcher (PI)

DISCUSS / DEBATE THE SELECTION OF AT LEAST 3
INDICATORS OF SUCCESSFUL KT FOR YOUR PROJECT
39
Group Development Sequence

• Forming
• Testing and dependence
• Orientation to the task Ground Rules
• Storming
• Intragroup Conflict
• Lack of unity
• Norming
• Development of group cohesion
• Open exchange
• Performing
• Functional role relatedness
• Emergence of solutions