Critical Appraisal of Systematic Reviews

1
Critical Appraisal of Systematic Reviews

• Douglas Newberry

2
Systematic Reviews or How to make a Monkey
out of EBM without hardly trying!
3
Systematic ReviewsObjectives

• Appraise a systematic review for validity
• Discuss Meta Analysis / use Odds Ratios
• Obtain Number Needed to Treat (NNT) from Odds
  Ratios
• Consider clinical implications of a Systematic
  Review including when to bin it instead!

4
We can see further than our forbearers because we
stand on the shoulders of Giantsand have better
spectacles

• these ideas are cribbed unashamedly from friends,
  books previous courses

5
Systematic ReviewsWhat are your Objectives

• What do you want to cover?
• Please interject with helpful questions!

6
Did I really want a systematic review? (but
please do not pretend)

• admit your ignorance expert review or consensus
  guidelines gt broad introduction, cover many areas
  (class C evidence).
• if the question is important gt formulate it!
• Systematic review gt narrow but rigorous focus.

7
Systematic Reviews Where do I start

• Start with your 4 (or 3) part clinical question!
• Is a systematic review a sensible approach?
• Does THIS systematic review address MY question?
• Is it a systematic review at all?

8
Is it a systematic review? does it

• define a four part (answerable) clinical
  question?
• combine Randomized Controlled Trials (RCTs)?
• describe PRE-DEFINED search methods?
• PRE-DEFINED inclusion criteria?
• PRE-DEFINED methodological exclusion criteria?

9
Sceptical View? Take it with a grain of salt

• transparent declaration of funding of work?
• Drug Company sponsorship of Reviews vs.
  Methodological qualitygtCochrane review!
• who employs the authors?
• open discussion of existing controversy
  commercial gain?
• Dont waste salt on your food, keep it for your
  reading!

10
Meta analysis combine what with what?

• Low Molecular Weight Heparin (LMWH) in hip
  surgery begin before or after the operation?
• meta analysis of placebo controlled RCTs of
  heparin in hip surgery gtgt
• pre-op post-op LMWH vs. placebo
• post-op LMWH Vs placebo
• pre-operative gtgt less intra-op bleeding??

11
Can we believe it ?

• bias free search inclusion criteria?
• appraisal of methodology of primary studies?
• consistent results from all primary studies?
• if not, are the differences sensibly explained?
• are the conclusions supported by the data?

12
If we believe it does it apply to our patient?

• Is our patient (or population) so different from
  those in the primary studies that the results may
  not apply?
• consider differences in
• time many things change.
• culture both treatments and values of outcomes
  can be different
• stage of illness or prevalence can effect
  results.

13
We believe it ! butgtgt does it matter?

• Is the benefit worthwhile to our patient?
• Ask the patient about cultural values.
• Think about Relative Risk Reduction vs. Absolute
  Risk to our patient.
• Potential benefit is the Absolute risk avoided in
  our patient Absolute Risk Reduction (ARR)!

14
Absolute Riskgt The risk our patient is facing!

• How likely is our patient to die (or have the
  outcome of interest) without intervention?
  Control Event Rate (CER)
• consider this individual patients risk factors
  to estimate Patient Expected Event Rate PEER.
• Absolute Risk usually increases with age.
• Improvement measured as Absolute Risk Reduction
  (ARR)

15
Relative Risk Reduction

• Usually reported in studies.
• Ratio of the improvement of outcome over outcome
  without intervention (Rx)
• Control Event Rate (CER) Experimental Event
  Rate (EER) / CER
• i.e. CER-EER/CER
• often independent of prevalence!
• often similar at different ages!

16
Our patient wants an absolute Risk Reduction
(ARR)

• is a 40 reduction in Cardiac Risk worth taking
  pills daily for 10 years?? gtvote!
• if I have a 30 risk of MI or death 30 out of
  100 people like me will suffer MI or death in
  next 10 years gt 40 RRR gtgt only 18 out of 100
  will have MI or death. ARR 12 out of 100! gtgtI
  like that!
• BUT if I have a 1 risk in 10 years, 40 less is
  a 0.6 risk gtgt hardly different!

17
Number Needed to Treat (NNT) (very trendy but
tricky)

• only defined for specific prevalence-Patients
  Expected Event RatePEER!
• only defined for a specific intervention!
• only defined for a specific outcome!
• eg. Pravastatin 40 mg nocte x10 years, in a 65
  year old male, ex-smoker with high BP and
  Diabetes, to reduce MI or Death.
• NNT is the inverse of Absolute Risk Reduction
  i.e. NNT 1/ARR

18
Number Needed to Treat (NNT) for previous example

• 12 fewer MI or death in 10 years per 100 persons
  treated ARR12/100
• NNT 1/(12/100)100/12 8.3
• But the same Relative Risk Reduction (RRR) of 40
  with a low prevalence
• 0.4 fewer MI/death per 100 treated, ARR0.4/100.
• NNT 1/(0.4/100) 100/0.4 250!

19
Why Odds Ratios? gt compare results of different
studies.

• consider 2x2 table
• RRR is (a-b/a) but you can only go in rows
  within same study!
• Odds ratio is (a/c)/(b/d) ad / bc the
  individual ratios are in columns, and therefore
  are independent of the prevalence which is
  different in different studies.
• must use odds ratios to combine RCTs

20
Odds Ratio (OR) to NNT is the improvement worth
the trouble?

• 1gtORgt0, lower the OR better the treatment (Rx)
  gtgt lower NNT.
• for any OR, NNT is lowest when PEER0.5
• estimate the PEER (patients risk)
• apply the OR to get patient's NNT.

21
Convert PEER OR to NNT
22
Formula used in the table
23
Table induced nausea!

• lower OR gtgt lower NNT
• Patient needs to be at risk (non-trivial PEER) in
  order for risk reduction to be worth the effort.
• for any OR, NNT lowest when PEER0.5
• more effective treatment gt lower NNT
• BUT are your patients values satisfied by the
  intervention and its sequelae?

24
Subgroup analysis Sceptical unless

• the subgroups make biological and clinical sense?
• the differences are both clinically
  statistically significant?
• was a-priori hypothesis (before this data)?
• other evidence supports these sub-groups?
• few (OK) or many (nix) sub-group analyses?

25
Any Questions?
26
Summary 1 Set your goals.

• define your 4 (or 3) part question.
• do you want a true systematic review?
• does this narrow review address my question?
• PRE-DEFINED search, inclusion, exclusion!

27
Summary 2 Be Sceptical!

• look for bias, conflict of interest.
• critical appraisal of primary studies?
• consistent results? if not, why not?
• does our patient fit the groups studied?
• does it matter to our patient?

28
Summary 3 Risks that matter.

• Absolute risk gt estimate the Patient Expected
  Event Rate (PEER)
• obtain Relative Risk Reduction (RRR) or Odds
  Ratio (OR) from a Meta-analysis
• plug into a table to estimate Number Needed to
  Treat (NNT)

29
Summary 4 Sceptical common sense!

• beware of post-hoc sub-group analysis, especially
  if multiple.
• step back and consider if the systematic review
  really related to our patients situation (PEER),
  culture and expectations?
• do not loose sight of common sense!

30
Coffee Now!

• Small Groups Afterwards