Prime Ministers Strategy Unit

1
Prime Ministers Strategy Unit
Jerry Lee Lecture 2004Is Evidence-Based
Government Possible?
Philip Davies PhD Government Chief Social
Researchers Office Prime Ministers Strategy
Unit Cabinet Office London SW1A 2WH England
February 19 2004
2
Prime Ministers Strategy Unit
Outline

• What is Evidence-Based Government?
• Factors Other Than Evidence
• Different types of research evidence
• Mechanisms for getting evidence into practice
• Some Examples of EBG/EBP in the UK

February 19 2004
3
Prime Ministers Strategy Unit
There is nothing a government hates more than
to be well-informed it makes the process of
arriving at decisions much more complicated and
difficult John Maynard Keynes
February 19 2004
4
Prime Ministers Strategy Unit

• Government policy must be
• Evidence-based
• Properly evaluated
• Based on best practice

February 19 2004
5
Prime Ministers Strategy Unit

• Policy making must be soundly based on evidence
  of what works
• Improve departments capacity to make best use of
  evidence
• Improve the accessibility of the evidence
  available to policy makers

February 19 2004
6
Prime Ministers Strategy Unit
Better Policy Making
UK policy making is now more informed by
evidence
Cabinet Office

• Reviewing Existing Policies
• Commissioning New Research
• Piloting New Initiatives
• Evaluating New Policy

November 2001
February 19 2004
7
Prime Ministers Strategy Unit
What is Evidence-Based Government?
Evidence-based government helps people make
well-informed decisions about policies,
programmes and projects by putting the best
available evidence from research at the heart of
policy development and implementation

(Davies, P.T., 1999a)
February 19 2004
8
Prime Ministers Strategy Unit
Yeah, but.what about peoples

• Experience?
• Judgement?
• Skills?
• Expertise?
• Tacit Knowledge?

February 19 2004
9
Prime Ministers Strategy Unit
What is Evidence-Based Government?
Evidence-based government means integrating
experience, expertise and judgement with the best
available external evidence from systematic
research.
(Davies, P.T.,
1999a)
February 19 2004
10
Prime Ministers Strategy Unit
Factors Influencing Policy Making
Experience Expertise
Pragmatics Contingencies
Judgement
Evidence
Lobbyists Pressure Groups
Resources
Habits Tradition
Values
February 19 2004
11
Prime Ministers Strategy Unit
Evidence-Based Government
Opinion-Based Government
Evidence-Based Government
Increasing Pressure (Time)
Adapted from Muir Gray 1997
February 19 2004
12
Prime Ministers Strategy Unit
Problems with Social Scientific Knowledge

• Not all research is of sufficient quality

• Unclear objectives
• Poor research design
• Methodological weaknesses
• Inadequate data reporting
• Selective use of data
• Unsupported conclusions

• Uncertainty of scientific knowledge
• Different status of different fields of knowledge

February 19 2004
13
Prime Ministers Strategy Unit
Evidence-Based Government
Opinion-Based Government
Evidence-Based Government
Increasing Pressure (Time)
Adapted from Muir Gray 1997
February 19 2004
14
Prime Ministers Strategy Unit
Different Types of Evidence for Government

• Systematic Reviews
• Rapid/Interim Evidence Assessments
• Single Studies
• Pilots and Case Studies
• Experts Evidence (GOBSATT)
• Internet Evidence

February 19 2004
15
Types of Research Evidence
Experimental Quasi-Experimental Counterfactual
Experimental Quasi-Experimental
Qualitative Theories of Change
Social Ethics Public Consultation
Surveys Admin Data Comparative Qualitative
Implementation Evidence
Ethical Evidence
Descriptive Analytical Evidence
Cost-Benefit Cost-Effectiveness Cost-Utility Econo
metrics
Impact Evidence
Surveys Qualitative
Economic and Econometric Evidence
Attitudinal Evidence
Statistical Modelling
Linear and Logistic Regression
February 19 2004
16
Prime Ministers Strategy Unit
Challenge for Intelligent Providers

• Multi-Method Capability
• Training and Professional Development
• A Challenge Function

February 19 2004
17
Prime Ministers Strategy Unit
A Challenge Function
Now when I hear NASA telling me things like
Gotta be true! or We know this to be true,
all my alarm bells go offWithout hurting
anybodys feelings, or squashing peoples egos,
were having to say, Were sorry, but were not
accepting that answer. Hal
Gehman,Chief Investigator, Columbia Disaster
Investigation
February 19 2004
18
Prime Ministers Strategy Unit
Challenge for Intelligent Providers

• Multi-Method Capability
• Training and Professional Development
• A Challenge Function

• Improving Accessibility of Research Evidence

February 19 2004
19
Prime Ministers Strategy Unit
Mechanisms for Evidence into Practice

• Integrating Research into Professional
  Competence
• Ownership of the Evidence
• Appropriate Buy-In
• Shared Notions of Evidence
• Incentives to Use Evidence
• Availability of Sound Evidence
• Strategic and Integrated Research and Policy

February 19 2004
20
Prime Ministers Strategy Unit
Examples of Evidence-Based Government in UK

• Sure Start Programme
• Educational Maintenance Allowance
• Connexions Programme
• New Deals Programmes- Welfare-to-Work
• New Deal for Communities
• Neighbourhood Renewal Programme
• Home Buying and Selling Policy

February 19 2004
21
Prime Ministers Strategy Unit
Examples of Evidence-Based Government in UK

• Strategic Audit
• Benchmarking
• Social change and social processes
• Scoping and better understanding of problems
• Social attitudes
• Public understanding of policy and government

February 19 2004
22
Prime Ministers Strategy Unit
ERA Demonstration Project

• What is the most effective and efficient way of
• Retaining low paid people in work?
• Advancing low paid people in the labour market?

February 19 2004
23
Prime Ministers Strategy Unit
ERA Demonstration Project

• Review of Existing Evidence
• Integrated Evaluation with Policy Development
  and Implementation
• Programme Theory Evaluation
• Qualitative Interviews (clients and employers)
• Experimental Method (R.C.T.)
• Administrative Data and Surveys
• Local Context Analysis (Qualitative and
  Quantitative)
• Economic Appraisal (CBA)

February 19 2004
24
Prime Ministers Strategy Unit
Take Home Message

• Evidence-Based Government is possible
• In UK, EBG/EBP is well established
• But in a broad sense of evidence-based
• And not universal
• Driving force the question(s) being asked
• Effectiveness of outcomes/impact and
• Effectiveness of implementation and delivery
• There are influences other than evidence

February 19 2004
25
Prime Ministers Strategy Unit
Take Home Message for the Campbell Collaboration

• The availability of such evidence is in short
  supply
• There is much to be done
• The delivery of useable products in overdue
• Time to get on with it!

February 19 2004
26
http//www.policyhub.gov.uk
Prime Ministers Strategy Unit
Contact
phil.davies_at_cabinet-office.x.gsi.gov.uk Government
Chief Social Researchers Office Prime
Ministers Strategy Unit Cabinet Office Admiralty
Arch The Mall London SW1A 2WH England Tel 44
(0)20 7276 1862 Fax [phone]

February 19 2004