Pippa Orr

1
Critical Appraisal Skillsquantitative reviews

• Pippa Orr
• Knowledge Support Librarian

With acknowledgements to CASP for their slides
2
Critical Appraisal Skills Programme (CASP)
Critical appraisal is the process of weighing
up evidence to see how useful it is in decision
making
http//www.phru.nhs.uk/casp/critical_appraisal_too
ls.htm
3
Effectiveness of Health Care

• doing the right thing
• to the right patient
• in the right way
• at the right time
• at the right cost
• in the right place

4
Kinds of evidence

• Descriptive
• cross-sectional, longitudinal
• Analytic
• case-control study
• cohort study
• Experimental
• randomized controlled trial

5
Hierarchy of evidence
6
Why does good evidence from research fail to get
into practice??
- 75 cannot understand the statistics - 70
cannot critically appraise a research paper
Using Research for Practice A UK Experience of
the barriers scale Dunn V, Crichton C, Williams
K, Roe B, Seers K
7
Critical appraisal helps the reader of research
...

• Decide how trustworthy a piece of
  research is (validity)
• Determine what it is telling us (results)
• Weigh up how useful the research
• will be (relevance)

8
Primary Research Evidence
Randomised Controlled Trials (RCT)

• Robust randomisation procedures
• to ensure that the variables are equal in both
  groups
• to remove all bias
• to ensure that the results are generalisable

9
Randomised controlled trial
new treatment
group 1
Outcome
population
Outcome
group 2
control treatment
10
Blinding
Blinding participants dont know what
intervention they are getting Double blinding
those giving the intervention dont know what the
participant is receiving
11
Loss to follow-up
It is important to ensure that all those that
are randomised into the trial are followed up to
the trials conclusion
12
Intention to treat analysis
Analysing people, at the end of the trial, in the
groups to which they were randomised, even if
they did not receive the intended intervention.
13
Types of review
Reviews
Systematic reviews
Meta-analysis
14
Publication bias

• Papers with "interesting" results are (or may be)
  more likely to be
• submitted for publication
• accepted for publication
• published in a major journal and in English
  Language
• quoted by authors
• quoted in newspapers

15
Odds Ratio, Relative Risk Measures of risk The
likelihood of something happening V The
likelihood of something not happening
16
Odds ratio (OR)

• The odds of an event happening in the
  experimental group expressed as a proportion of
  the odds of an event happening in the control
  group
• The closer the OR is to 1, the smaller the
  difference in effect, i.e. no effect OR 1

17
Confidence intervals/ limits

• Presents the range of likely effects
• The 95 confidence interval, for example,
  includes 95 of results from studies of the same
  size and design in the same population
• This is close, but not identical, to saying that
  the true size of effect (never exactly known) has
  95 chance of falling within the confidence
  interval
• The narrower/ shorter the confidence interval,
  the more precise/ confident we can be about the
  estimate

18
Forest plots

• Common approach to presenting the results of a
  meta-analysis
• Also known as a blobbogram or odds ratio
  diagram
• Graphical representation of individual trial
  results included in a review, together with the
  combined meta-analysis result

19
line of no effect
confidence interval
meta-analysis result
20
p-value

• The probability (ranging from 0 to 1) that the
  results observed in a study (or results more
  extreme) could have occurred by chance if in
  reality the null hypothesis was true, ie if you
  did nothing.
• If this probability is less than 1/20 (which is
  when the p value is less than 0.05), then the
  result is conventionally regarded as being
  statistically significant.

21
The p-value in a nutshell
Could the result have occurred by chance?
The result is likely to be due to chance
The result is unlikely to be due to chance
0
1
p lt 0.05 a statistically significant result
p gt 0.05 not a statistically significant result
p 0.05 or 1 in 20 result fairly unlikely to
be due to chance
p 0.5 or 1 in 2 result quite likely to be
due to chance
1 20
1 2
22
Number needed to treat
Is the number of people you would need to treat
with a specific intervention to see one
additional occurrence of a specific beneficial
outcome.
23
Critical appraisalquestions to apply to reviews

• is it trustworthy? validity
• what does it say? results
• will it help? relevance