Critical Appraisal

1
Critical Appraisal
2
Critical Appraisal

• Definition assessment of methodological
  quality
• If you are deciding whether a paper is worth
  reading do so on the design of the methods

3
Types of Study

• Primary these report research first hand.
• Experimental i.e. humans, animals artificial and
  controlled surroundings.
• Clinical trials intervention offered.
• Survey something is measured in a group.

4
What type of study?

• Secondary summarise and draw conclusions from
  primary studies.
• Overview
• Non systematic (summary)
• Systematic (rigorous and pre-defined methodology)
• Meta-analyses (integration of numerical data from
  more than one study)
• Guidelines (leads to advice on behaviour)
• Decision analyses (to help make choices for
  doctor or patient)
• Economic analyses (i.e. is this a good use of
  resources?)

5
Small Groups

• 15 minutes
• Appoint feedback person
• List the different types of study you have heard
  of
• Describe them give an example

6
Specific Types of Study

• Randomised Controlled Trial (RCT)
• Population is randomly allocated to two groups
• One group is given a specific treatment or
  intervention
• On average the groups are identical because they
  are randomised and therefore any difference in
  the measured outcome is due to the intervention
• Specified follow up period and specified outcomes
• e.g. drug better than placebo surgical procedure
  compared with sham

7
Randomised Controlled Trial (RCT)

• Advantages
• Allows rigorous evaluation of a single variable
  in a previously defined population e.g. a new
  drug.
• Prospective i.e. collect the information after
  you decide to do the study.
• Tries to disprove the null hypothesis
• Tries to eradicate bias because the two groups
  are identical.
• Allows for meta-analysis later.

8
Randomised Controlled Trial (RCT)

• Disadvantages
• Expensive and time consuming which can lead to
  problems including
• Too few subjects
• Too short a time
• Who controls the study?
• End point not clinical
• Possibility of hidden bias
• Imperfect randomisation
• Failure to randomise all eligible patients who
  is included/excluded.
• Assessors not blinded.

9
Definitions

• Single blind subjects dont know which
  treatment they are receiving.
• Double blind neither subjects nor investigators
  know who is receiving treatment.
• Cross over each subject received both the
  intervention and controlled treatment (randomly)
  often with wash out.
• Patients act as own control.
• Placebo controlled controls received inactive
  or sham treatment

10
Cohort study

• Two (or more) groups of people are selected on a
  basis of a difference in exposure to a particular
  agent i.e. vaccine, environmental toxin,
  medicine.
• Group followed up (usually for years) to see how
  many in each group develop a particular
  disease/outcome.
• e.g. Peto 40,000 UK doctors.
• e.g. COCP causes breast cancer?

11
Case Control Study

• Patients with a particular disease are identified
  and matched with controls.
• Data is collected retrospectively either from
  medical records or from memory, looking for a
  causal agent.
• Looks for associations but not necessarily the
  same as cause.
• e.g. SIDS and sleeping position.
• Does whooping cough vaccine cause brain damage?
• Do overhead cables cause leukaemia?

12
Cross Sectional Survey

• A representative sample of subjects or patients
  are studied (interviewed, questionaired,
  examined) to answer a specific clinical question
  at a specific time.
• e.g. normal height of three year olds
• what do most GPs think about the use of Viagra?

13
Case Reports

• Medical history of a single patient in a story
  form.
• Lots of information given which may not be seen
  in a trial or a survey.
• Often written and published fast compared to
  studies
• e.g. Thalidomide

14
(No Transcript)
15
Hierarchy of Evidence

• (Systematic Review and Meta-analysis)
• Randomised Controlled Trial
• Cohort Studies
• Case Control Studies
• Cross Sectional Surveys
• Case Reports

16
Assessing Methodological Quality

• Questions to Ask
• general framework
• specifics dependant on type of paper
• Logical Progression
• Introduction - Title
• - Abstract
• - Introduction
• Methods
• Results (Statistics!)
• Discussion

17
Seven essential questions

• Introduction
• 1. Why was the study done?
• Is the study original or does it add to the
  literature in any way? e.g. bigger, better,
  larger, more rigorous
• Is it interesting?
• Is there a clear research question?

18

• Is there a clear research question?
• i.e. what is the key research question/ what
  hypotheses are the author testing?
• Hypothesis is usually presented in the negative
  the
• null hypothesis
• Studies try to disprove this lack of difference
  or null hypothesis.

19
Seven essential questions

• Methods
• 2. Who is it about?
• How recruited?
• Who included?
• Who excluded?
• Studied in real life circumstances?
• Applicable?

20
Seven essential questions

• 3. What kind of study was done?
• Was it well designed?
• i.e. does the study make sense?
• What specific intervention or manoeuvre was being
  considered and what was it being compared to?
• Is what happened what the author said happened?
• What outcome was measured and how?
• i.e. length of life, quality of life, reduction
  in pain
• need to be objective.

21
Was design appropriate?

• In general
• Therapy i.e. effect of intervention RCT
• Diagnosis ? test valid (can we trust it) or
  reliable (? same result if repeated) cross
  sectional survey with both gold standard and new
  test
• Screening large population, pre-symptomatic
  cross sectional survey
• Prognosis i.e. what happens to someone if a
  disease is picked up at an early stage
  longitude cohort study
• Causation e.g. ? possible harmful agent leads
  to cause cohort or case control study
• - ? case report.

22
Seven essential questions

• 4. Was systematic bias avoided?
• i.e. was it adequately controlled for?
• Bias anything that erroneously influences the
  conclusions about groups and distorts comparisons
  
• e.g. RCT method of randomisation, assessment ?
  truly blind.
• Cohorts population differences
• Case control true diagnosis, recall (and
  influences)

23
Seven essential questions

• 5. Was it large enough and long enough to make
  results credible?
• Size is important!

24
Seven essential questions

• Results
• 6. What was found?
• Should be logical simple complex

25
Seven essential questions

• Discussion
• 7. What are the implications?
• For
• - you
• - practice
• - patients
• - further work
• and do you agree?

26
Four possible outcomes from any study

• Difference is clinically and statistically
  significant i.e. important and real.
• Of clinical significance but not statistically
  so. ?sample size too small.
• Statistically significance but not clinically
  i.e. not clinically meaningful.
• Neither clinically nor statistically significant.

27
Recommended Reading

• Ian Crombie The Pocket Guide to Critical
  Appraisal
• Trish Greenhalgh How to read a paper the basis
  of evidence based medicine

28
???????????(statistics)
29

• I am NOT a statistician
• I am not a number
• I am a free man

30
Need to know-

• Need to be able to understand what some of the
  concepts are .
• Other people (including authors) dont understand
  statistics and may use this to mislead reader.

31
General questions which you need to ask(not
related to knowing how to do statistics)

• What is the size of the sample?
• What is the duration of follow-up?
• Is the follow-up complete?
• What sort of data has been collected?
• Have appropriate tests been used?
• If statistical tests are obscure why? Are they
  referenced?
• Have data been analysed according the original
  study protocol? (beware of retrospective
  sub-group analysis).
• Have assumptions been made regarding association
  and cause.

32
The Specifics
33
Size Of The Sample (Power)

• Trials should be big enough to have a high chance
  of detecting as statistically significant, a
  worthwhile effect if it exists and therefore be
  reasonably sure that no benefit exists if it is
  not found in the trial.

34
Possible to calculate the sample size (power)

• What difference would be clinically significant?
• Look up statistical tables to find the number
  needed to have a moderate, high, or very high
  chance of detecting a true difference.
• Usually 80 to 90

35

• Numerical data is analysed differently dependent
  on whether it is parametric or
  non-parametric.
• Parametric data sampled from a particular form
  of distribution e.g. normal distribution.
• Non-parametric does not assume the data sampled
  from a particular form of distribution.
• Parametric tests are more powerful and
  preferable.

36

• Normal distribution particular shape of curve
•  
• Skewed Distribution
•  
• It is possible mathematically to transfer a
  skewed to a normal distribution.

37

• Mean average
• Mode most frequent
• Median mid-point
• Standard deviation way of describing spread
  around the mean
• In a normal distribution,
• 95 of values lie within /- 2SD
• 66 of values lie within /- 1SD

38

• Significance test when comparing two
  populations e.g. intervention and
  non-intervention, you start from the assumption
  that there will be no difference null
  hypothesis.
• Experiment/trial being done to disprove this.
• The type of study, and the data used will
  determine which test is used to obtain a number
  as a way of measuring this.
• The letter P significance value of the test
  used to do this (tests vary).
• The value of P probability that a particular
  outcome would have arisen by chance.

39

• Standard practice (arbitrary)
• P of less then 1 in 20 or lt 0.05 is said to be
  statistically significant.
• P of less than 1/100 or P lt 0.01 is statistically
  very significant.
• This leads to rejection of null hypothesis i.e.
  reject there is no difference.

40

• If P is lt 0.05
• This suggests there is a 95 chance that the null
  hypothesis can be rejected i.e. there is a
  difference between the two groups.
• Difference between statistical significance and
  clinical significance.

41

• Type 1 error
• If the test suggests a difference but there is
  not really a difference.
• Dependent on significance level.
• Type 2 error
• If tests suggests no difference but a difference
  does exist.
• Related to size of populations.

42

• Confidence Intervals (CI)
• This allows for an estimation of whether the
  strength of evidence is strong or weak.
• A range of values within which it can be stated,
  with a certain degree of confidence (usually 95)
  that the population statistic (answer) lies.
  Upper and lower levels are given.
• 95 confidence intervals imply that there is a
  95 chance that the real answer lies between
  the two limits given.
• The narrower this range the better.
• If 0 is included the test is not significant i.e.
  P gt 0.05.

43
Risk reduction

• Absolute risk reduction
• The absolute difference in event rates
• X Y
• Relative risk reduction
• The proportional reduction in rates between
  experiment and control
• (X Y)/X x 100
• Number needed to treat
• The number of patients who need to be treated to
  achieve one additional favourable outcome
• 1/(X Y)

44
Screening Tests
Validation study comparing the gold standard
with a new screening test.
45

• Sensitivity
• True positive rate a / (ac)
• How good is this test at picking up people who
  have this condition?
• Detects a high proportion of true cases.
• Specificity
• True negative rate d / (bd)
• How good is this test at correctly excluding
  people without the condition?
• A specific test has few false positives.

46

• Positive predictive value
• If a person tests positive, what is the
  probability he/she has the condition?
• a / (ab) i.e. the proportion of test positives
  who are truly positive.
• Negative predictive value
• If a person tests negative what is the
  probability that he/she does not have the
  condition?
• d / (cd) i.e. the proportion of test negatives
  who are truly negative.
• Accuracy
• What proportion of all tests have given the
  correct results
• i.e. true positive and true negatives as a
  proportion of all results (ad) /
  (abcd)