Mark Petticrew

1
Systematic reviews for public health new
challenges and new directions

• Mark Petticrew
• MRC Social and Public Health Sciences Unit
• Glasgow

2

• MRC SPHSU Evaluation programme
• Evaluation of the impacts of social interventions
  such as policies (impacts on public health and
  health inequalities)
• (i) Primary research on social housing renewal,
  urban regeneration, transport policies,
  employment... (ii) systematic reviews

3

• Glasgow (Mark Petticrew, Sally Macintyre, Matt
  Egan, Val Hamilton, Sian Thomas)
• Liverpool (Margaret Whitehead, Clare Bambra, Beth
  Milton)
• Lancaster (Hilary Graham, Liz McDermott, Pam
  Attree)

4
Systematic reviews on the health effects of

• Housing improvement and urban regeneration
• Transport policies (e.g., new road building
  modal shifts physical activity promotion)
• Employment (inc. privatisation, workplace
  re-organisation, welfare-to-work industrial
  subsidies shiftwork)
• Tobacco control interventions and inequalities
• (MRC SPHSU /CRD / University of Liverpool,
  ongoing)
• Reviews of effects of interventions

See www.msoc-mrc.gla.ac.uk/Evidence/Evidence.htm
l
5
Wanless and systematic reviews

• The Wanless, and Wellcome reports pointed to the
  need for systematic reviews to identify gaps in
  the public health evidence base, and to direct
  the development of future interventions
• Also reports from HDA and others
• What challenges will this pose?

6
(i) Public health evidence What do policymakers
think of it so far?
Cameron House policymakers workshop (2002)

• Policy-free evidence - research that does not
  answer clear, or policy relevant questions
• Researcher naïveté of the policy environment
  (which militates against knowledge transfer
  between science and policy )
• Too much evidence from far down the causal chain,
  (often concerned with health behaviours and
  clinical issues, not with broader social
  determinants of health)

J Epidemiol Community Health 2004 58(10)811-6
7
Generalisability and transferability of evidence

• Researchers are overly concerned with critically
  appraising internal validity (bias)
• Not concerned enough with assessing whether
  research evidence is generalisable

8
Other problems with current public health evidence

• Problems with high concept notions of evidence
  preferred by academics
• In policy circles a mixed economy of evidence
  prevails, with different types of scientific and
  non-scientific evidence used

9
So, when looking forward

• We need to be aware of
• 1) criticisms of existing evidence (including
  systematic reviews) as well as
• 2) the gaps

10
Problems with public health systematic reviews
(1)

• We want to know from systematic reviews about the
  impacts of interventions - What Works
• But What Works is only part of the answer -
  evidence on effectiveness alone is often not
  enough
• We also need to know about implementation,
  sustainability, and how something works -
  usually overlooked in systematic reviews

11
Problems with systematic reviews(ii)

• Sometimes guilty of naïve inductionism If we
  stick all these studies together, the end result
  is bound to to be meaningful, and useful for
  predicting the future
• Spuriously restrictive inclusion criteria (We
  found no trials, therefore there is no evidence)
• As a result reviews may be subject to a
  conservative bias biased towards reviewing
  easily-evaluated, individual-level interventions

12
The Utilitarian bias of systematic reviews
Systematic reviews, like the primary research on
which they are based, are concerned with the
greatest number - not with the distribution of
impacts or impacts on inequalities in
society (Jackson, Waters et al. 2004)
Jeremy Bentham
13
Current public health evidence lacks an equity
dimension

• Systematic reviews are needed to assess
  differential effectiveness of interventions -
  whether the effects vary by social class, gender,
  education, ethnicity...
• Most primary studies consider socioeconomic
  variables in the study design, but generally did
  not stratify in the analysis (confounders)
• We (CRD/MRC SPHSU) are now re-reviewing all
  reviews (19) and primary studies (about 60) of
  population-level tobacco control interventions
  (bans, restrictions on youth access, price
  increases etc)
• In general we need to re-focus existing
  systematic reviews (and primary research) around
  equity issues

(Tobacco Control 200413(2)129-31)
14
New challenges and opportunities for new research

• New equity-focussed systematic reviews
• Cochrane Equity Field / Campbell Equity Methods
  Group (Tugwell et al.) to promote and support
• GRADE working group has acknowledged the
  importance of equity factors by proposing that
  equity considerations be reflected in separate
  recommendations

15
Sifting the evidence and evidential nihilism

• Evidence on interventions to improve public
  health is in short supply, and often hard to find
  - particularly evidence on the social
  determinants of health (housing, transport,
  education, employment)
• Evidence on the means of tackling inequalities
  through the social determinants of health is even
  more difficult to locate
• Its easy to set inclusion criteria, sift through
  the evidence and conclude that there isnt enough
  of it, and what there is, isnt very good

16
Alvie Two elderly women are at a Catskill
mountain resort, and one of 'em says, Boy, the
food at this place is really terrible. The
other one says, Yeah, I know and such small
portions.
from Annie Hall (1977)
17
Public health evidence and the Alvie Singer
Phenomenon

• The same criticisms seem to apply to public
  health evidence. Theres not a lot of it, and
  what there is is not high quality
• Its challenging to make the best use of the
  (sometimes poor, often sparse, and certainly
  heterogeneous) evidence that we do have and it
  can be difficult to use this to directly inform
  decision-making
• But its the best weve got we are searching for
  Best available evidence rather than best
  evidence

18
To conclude Grasping Nettles and Picking
Low-hanging Fruit

• A lot of the low-hanging fruit is already being
  picked straightforward systematic reviews of
  easily-defined interventions
• But can we produce new frameworks within which
  different types of research evidence can be
  collected and integrated to inform decision
  making?
• RCTs non-RCTs, observational epidemiological
  studies qualitative data
• Thats still the real challenge

19

• End!

20
Systematic review of UK urban regeneration
programmes

• Area-based regeneration and neighbourhood
  renewal to strengthen disadvantaged communities
  has been recommended by the UK Treasury, and the
  Department of Health as a key strategy to tackle
  health inequalities
• Systematic review of major UK Urban Regeneration
  programmes
• Approximately 11bn spent from 1980-2002. Mostly
  evaluated using case study methods, very often
  presented without supporting data
• Most that can be said is that the impacts on
  health are small and may be positive (negative
  impacts largely unknown)
• (Thomson et al. 2006 Journal of Epidemiology and
  Community Health)

21
(No Transcript)