Systematic Reviews: When are they good enough

1
Systematic ReviewsWhen are they good enough?

• Hierarchy of evidence
• Brief introduction to systematic reviews
• Requirements for masters dissertations
• Masters students the LREC challenge
• Optimum vs good enough

2
Hierarchy of Evidence

• Systematic reviews and meta-analyses
• Randomised controlled trials with definitive
  results (confidence intervals that do not overlap
  the threshold clinically significant effect)
• Randomised controlled trials with non-definitive
  results (a point estimate that suggests a
  clinically significant effect but with confidence
  intervals overlapping the threshold for this
  effect)
• Cohort studies
• Case-control studies
• Cross sectional surveys
• Case reports.
• Trisha Greenhalgh. How to read a paper getting
  your bearings BMJ 1997315243-246

3
Not a literature review

• Very focused question
• Accesses ALL the literature
• Involving searching
• Electronic databases
• Grey literature
• References of references
• Hand searching journals
• Contacting authors

4

• Systematic Review
• review of a clearly formulated question that
  uses systematic and explicit methods to identify,
  select and critically appraise relevant research
  and to collect and analyse data from the studies
  that are included in the review. Statistical
  methods (meta-analysis) may or may not be used to
  analyse and summarise the results of the included
  studies
• Clark M and Oxman AD 2003. Cochrane Reviewers
  Handbook 4.2.0 Oxford The Cochrane Library

5
Uses of systematic reviews

• Help decision makers cope with the volume of
  studies by summarising them
• Provide new' knowledge which may not be apparent
  from individual studies where the effects under
  investigation are small

6
Essential Steps to a Systematic Review

• Establish researchable question population,
  intervention, outcome
• Identify and consider all available evidence
• Use an explicit process to identify, and combine
  evidence if appropriate
• Justify all included and excluded studies
• Interpret the evidence and its applicability to
  the population of interest
• Assess the strength of the evidence gathered

7
Interpretation of results

• How appropriate is the question
• How good is the quality of included trials
• How significant are the observed effects
• How consistent are the effects across trials
• Is there a clear dose-response relationship
• Is there indirect evidence that supports the
  inference
• Have other plausible competing explanations of
  the observed effects been ruled out

8
Essential skills for a Systematic Review

• The ability (and facilities) to carry out a
  thorough literature search
• Awareness of sources of bias
• The ability to critically appraise the quality of
  included trials and to check for consistency in
  the effects across trials (i.e., homogeneity)
• Understanding how to measures the significance of
  observed effects (e.g. NNTs)

9
Stages and Phases of a Review CRD Report
Number 4

• Stage 1 planning the review
• Phase 0 identification of the need for a review
• Phase 1 preparation of a proposal for a review
• Phase 2 development of a review protocol
• Stage 2 conducting a review
• Phase 3 identification of research
• Phase 4 selection of studies
• Phase 5 study quality assessment
• Phase 6 data extraction and monitoring progress
• Phase 7 data synthesis
• Stage 3 reporting and dissemination
• Phase 8 the report and recommendations
• Phase 9 getting evidence into practice

10
Challenges and Complexities

• Clearly, must be rigorous, competent,
  confidence-inspiring
• Lengthy time-scale
• Considerable cost around 50 000 (often much
  more)
• Disseminated through Cochrane library and high
  impact journals
• Researchers qualifications (eg PGDip SR)
• Often by highly regarded research groups (eg
  Moore and McQuay Pain research group _at_
  Churchill Hospital, Oxford)

11
Masters Dissertations

• Integrates research knowledge and philosophies
  underpinning the Masters course
• Student is required to demonstrate a high level
  of understanding of the philosophy and principles
  of research
• Show competence in the design and execution of
  such a piece of work

12
Systematic Reviews and Dissertations Choosing
to do them

• Increasingly students wish to do SRs
• Avoid logistical challenges raised by research
  governance
• Lack of access to subjects (particularly those
  not in practice and/or overseas students)
• Seen as easy option - initially

13
Potential problems

• Who supervises?
• Time scale
• Short cuts eg only English papers, no
  contacting authors, restricted searching for grey
  literature
• Examples in practice practitioners see SRs as
  quick and easy solution

14
Questions we need to answer

• What makes a SR good enough? Is this even a
  possible concept? Does it have to be excellent?
• Should Masters students be allowed to undertake
  them?
• Where is that point where they become
  acceptable/good enough and unacceptable/not good
  enough? How do we identify the boundary? Are
  there specific features that we need to identify?
• How proficient/familiar should students be in
  statistical methods?
• Many others?