NPO conference Amsterdam 19 20 June

1
NPO conferenceAmsterdam 19 20 June

• The Importance of Evidence-Based Policy Making
• Professor Denise Lievesley
• UN African Centre for Statistics
• Addis Ababa, Ethiopia
• and President, International Statistical Institute

2
Content of presentation

• What is evidence-based policy?
• Need for an evidence base at all stages in the
  policy cycle
• Why is evidence-based policy important?
• Judging what works
• Evidence but one input into policy process
• What is evidence?
• Evidence initiatives
• Challenges for statisticians
• Examples from the developing world

3
What is evidence-based policy ?

• Helping people to make well-informed decisions
  about policies, programmes and projects, by
  putting the best available evidence from research
  at the heart of policy development and
  implementation
• Enlightening through making explicit what is
  known through scientific evidence and importantly
  what is not known
• Better statistics, better decisions, better
  outcomes Vision of the African Centre for
  Statistics

4
In contrast to opinion based policy

• which relies heavily on
• either the selective use of information or
• on the untested views of individuals or groups
  often inspired by ideological standpoints,
  prejudices or speculative conjecture.

5
What counts is what works (Labour Gov
manifesto)

• Too narrow
• Need clarity as to the problem
• who is affected
• what causes it
• what might be done about it
• at what cost
• by whom
• with what effects?

6

• A modern perception of evidence-based policy
  making is sometimes characterised as a process
  whereby the evidence is assembled almost
  independent of the policy options and then the
  preferred policy choice emerges. Holt DT

7
Need an evidence base at all stages in the policy
cycle

• in shaping agendas,
• in defining issues,
• in identifying options,
• in making choices of action,
• in delivering them and
• in monitoring their impact and outcomes.

8
Context in the UK

• Modernising Government white paper commits to
  policy making that is
• strategic,
• outcome-focussed,
• joined up (works across organisational
  boundaries),
• inclusive (is fair and takes account of the
  interests of all gender sensitive, promotes
  equity),
• flexible,
• innovative (means taking risks and so risks
  must be identified, monitored and managed)
• robust (works and continues to work).
• Tackles causes not symptoms.
• Is acceptable to the public.

9
Why is evidence-based policy important?

• Sherlock Holmes it is a capital mistake to
  theorise before you have all the evidence. It
  biases the judgement
• Policy makers may be well-intentioned but
  misguided

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Results of a meta-analysis

• Collation of the results of many studies
  contradict this advice
• Extract from publicity prepared for the UK
  Reduce the Risk Campaign (early 1990s)
• The risk of cot death is reduced if babies are
  not put on the tummy to sleep. Place your baby on
  the back to sleep. .Healthy babies placed on
  their backs are not more likely to choke.

14
(No Transcript)
15
Iain Chalmers

• No doubt like millions of his other readers, I
  passed on and acted on this apparently rational
  and authoritative advice.
• We now know that the advice promulgated so
  successfully in Spock's book led to thousands, if
  not tens of thousands, of avoidable cot deaths.
• (Letter to BMJ)

16
Judging what works

• Outcomes often defined too narrowly eg
  performance management whereby targets are often
  set top down and may have unintended
  consequences.
• Hitting the target but missing the point
• The good, the bad and the ugly report of the
  Royal Statistical Society
• Difficulty is that sometimes different things
  work in different circumstances and this is
  interpreted as nothing works.
• (Ref Robert Martinsons 1974 review of
  American offender rehabilitation programmes- He
  concluded that there was no single approach which
  worked consistently different things worked in
  different circumstances but this was misread as
  nothing works.)

17
Concern about an over-concentration on quantity
not quality

• Who decides what works?
• Examples
• Focus on growth rather than equity in Africa
• Focus on waiting times rather than health quality
  in the NHS
• Ref targets are crowding out compassion in
  the NHS The Independent Sunday June 1 2008
  Shadow health minister quoted as saying This is
  not a criticism of the NHS staff but of the ethos
  that has been created in the health service.
  They are working in systems designed around
  throughput and not quality

18

• The gross national product does not allow for
  the health of our children, the quality of their
  education, or the joy of their play it does not
  include the beauty of our poetry or the strength
  of our marriages, the intelligence of our public
  debate or the integrity of our public officials.
  It measures neither our wit nor our courage,
  neither our wisdom nor our learning, neither our
  compassion nor our devotion to our country, it
  measures everything in short except that which
  makes life worth while.
• Robert Kennedy

19
The policy making processPolicy making is the
process by which governments translate their
political vision into programmes and actions to
deliver desired changes in the real world.

• Evidence but one input into policy process

Self interest
Values/ beliefs
Campaign promises
Ideology
Judgement
Tradition
Lobbies
Expert views
Acceptability to public
Experience
Resource constraints
20

• There is nothing a government hates more
  than to be well-informed for it makes the
  process of arriving at decisions much more
  complicated and difficult.
• John Maynard Keynes

21
What is evidence?

• Expert knowledge published research existing
  statistics stakeholder consultations previous
  policy evaluations the Internet outcomes from
  consultations costings of policy output from
  economic and statistical modelling. UK Cabinet
  Office 1999
• Scientific, rigorous, critically appraised, well
  documented
• Fit for purpose
• Beware policy-based evidence?
• Unbiased (note problems of publication bias)
• But data are never value free

22
All scientific evidence is imperfect.

• The absence of excellent evidence does not
  make evidence-based decision making impossible
  what is required is the best evidence available
  not the best evidence possible
• Muir Gray 1997
• Evidence rarely provides neat and tidy
  prescriptions to decision makers as to what they
  should do. Often it generates more questions to
  be resolved
• Petrosino et al 2001

23
Evidence should be based on experiments

• Adrian Smith Mad cows and ecstasy choice or
  chance in evidence-based society
• Experiments used too little in social and
  economic research
• Rich data from different sources
• There is a need to shift goalposts as to the
  definition of robustness so that it becomes more
  inclusive of quantifiable quantitative research.
  Only this can ensure that social issues do not
  remain confined to anecdotal boxes, but provide
  information of equal comparability in poverty
  assessments
• Claus Moser

24
Challenges for statisticians

• increase understanding of the policy process,
  where and how evidence can feed into it (eg
  SPATS)
• improve interpretation and communication of data
  particularly in relation to uncertainty
• speak the language of policy makers
• forge strong relationships with policy analysts,
  increase number of policy analysts
• combine humility and confidence
• improve training for statisticians
• (International Research Forum on statistical
  reasoning, thinking and literacy
  http//www.srtl.stat.auckland.ac.nz)

25
Particular challenges with respect to timing

• collection of quality data takes time
• demand for quick fixes means that policy makers
  often do not have time for in-depth research
• Demands by Ministers are short term, quick fix
  solutions rather than major carefully considered
  strategic initiatives (Bullock et al)
• yet to be effective the research strategy needs
  to look beyond the timescales of one government
• many issues are chronic beware unwarranted
  impatience (Boruch)

26
Evidence compilations example initiatives

• Data archives
• Cochrane collaboration
• Campbell collaboration
• National Library of Health
• African Programme on Rethinking Development
  Economics
• Communities of practice

27
Joint Marrakech Memorandum

• Affirm a commitment to fostering a global
  partnership on managing for development results.
  Awareness is growing that getting better
  development results requires management systems
  and capacities that put results at the center of
  planning, implementation and evaluation. We
  need to align cooperation programs with desired
  country results, define the expected contribution
  of our support to country outcomes and rely on
  and strengthen countries monitoring and
  evaluation systems to track progress and assess
  outcomes better distil the lessons of
  countries experiences and disseminate knowledge
  about what gets results in different country
  contexts.

28
Evidence base urgently needed in developing
countries

• Problems are severe and urgent
• Need ownership by countries
• and to empower them
• Develop policies which are relevant to their
  needs
• and ensure effective implementation
• Need to counteract corruption
• Paucity of data is a major obstacle
• Lack of recognition /acceptance of much data

29
C.Scott for Paris 21Use of good statistics
having a positive effect on policy

• Uganda
• Poor public service delivery caused by
  governments failure to ensure that budgeted
  funds reached front line agencies
• Brazil and Mexico
• Tackled child poverty and education by a
  programme to give child benefits to mothers
  according to the attendance of their children in
  school (experimental research)

30
C.Scott for Paris 21Absence of data or failure
to use available information has negative effect
on policy making

• Malawi
• Data from Save the Children on child malnutrition
  disregarded by government because of conflict
  with crop data
• Botswana
• Deficiencies in data on HIV/AIDS (from sentinel
  surveillance systems) legitimised the rejection
  of the message on the scale of the problem

31
Current challenge in policy development in poor
countries

• Development literature of the 1990s dominated by
  the view that growth is central to any strategy
  aimed at poverty reduction.
• Since modified not growth per se but structure
  of growth that matters.
• Further recognised that income inequality matters
  when making progress on poverty reduction.
• Example Nigeria
• Aigbokhan asks why the rate of poverty is so high
  in Nigeria despite the strong growth performance
  and concludes it is because of a lack of explicit
  concern with inequality in public policy. He
  analyses data on Nigeria and on other countries
  for comparative purposes to argue that if growth
  occurs in the sectors that require skills that
  the poor of Nigeria do not possess although the
  economy grows it has little impact on poverty
  reduction.

32
conclusions

• Overview of evidence-based policy
• Question for us as to whether this is rhetoric or
  reality ?
• Policy people still often pay lip service to
  evidence and evaluation Bullock et al
• How can we work together to ensure that policies
  are as well as being well-intentioned are
  well-informed and transparent?
• Our aims to develop a strategic vision, an
  honest assessment of progress and an
  institutional flexibility to adapt to new
  information.

33
(No Transcript)