Evaluation

1
Evaluation Principles and methods

• Matt Barnard
• Head of Evaluation
• NSPCC

2
Purpose of evaluation

• Definition
• Examine how a policy or intervention was
• designed and carried out and with what
  results. (Magenta Book)
• Asks objective questions
• What were the impacts?
• How was it delivered?
• What were barriers and facilitators?
• Did it deliver value for money?
• Aims to provide
• Scientific basis for policy making

3
Evaluation Design Process
Evaluation Design
Evaluation implementation
Evaluation findings
Intervention Design
Logic model
Intervention implementation planning
Intervention implementation
4
Logic models

• Characteristics
• Mechanisms not processes
• Key steps not every step
• Explanatory not descriptive
• Reflects theoretical assumptions
• Benefits
• Sense check
• Identifies realistic outcomes
• Facilitates evaluation design

5
Strength of design
Strength of design matrix
Weak design (Poor/ no counterfactual) Strong design (Realistic counterfactual)
Low power (Small numbers/ effect size) Unlikely to detect difference/ Low confidence in attribution Unlikely to detect difference/ High confidence in attribution
High power (Large numbers/ effect size) Likely to detect difference/ Low confidence in attribution Likely to detect difference/ High confidence in attribution
6
Strength Evaluation design

• Randomized controlled trial
• Quasi-experimental design
• Before and after measures

7
Types of design

• RCT
• Individual randomization
• Cluster randomization/ roll out
• BAU/waiting list/alternative services
• Quasi-experimental designs
• Matched area/ groups
• Matched individual
• Interrupted time series
• Regression discontinuity

8
Factors influencing methodology

• Intervention stage of development
• Early exploration
• Defined and established but not proven
• Transferability
• Potential Costs and benefits
• Resources
• Timescales

9
Key principles

• Clarity about key question
• Avoid default questions
• Methods matched to question
• Ensure methods match desired questions
• Claims match evidence
• Avoid over-claiming
• Have a coherent story to tell