Sources of Bias in Randomised Controlled Trials

1
Sources of Bias in Randomised Controlled Trials

• David Torgerson
• Director, York Trials Unit
• djt6_at_york.ac.uk
• www.rcts.org

2
Selection Bias - A reminder

• Selection bias is one of the main threats to the
  internal validity of an experiment.
• Selection bias occurs when participants are
  SELECTED for an intervention on the basis of a
  variable that is associated with outcome.
• Randomisation or other similar methods abolishes
  selection bias.

3
After Randomisation

• Once we have randomised participants we eliminate
  selection bias but the validity of the experiment
  can be threatened by other forms of bias, which
  we must guard against.

4
Forms of Bias

• Subversion Bias
• Technical Bias
• Attrition Bias
• Consent Bias
• Ascertainment Bias
• Dilution Bias
• Recruitment Bias

5
Bias (cont)

• Resentful demoralisation
• Delay Bias
• Chance Bias
• Hawthorne effect
• Analytical Bias.

6
Subversion Bias

• Subversion Bias occurs when a researcher or
  clinician manipulates participant recruitment
  such that groups formed at baseline are NOT
  equivalent.
• Anecdotal, or qualitative evidence (I.e gossip),
  suggest that this is a widespread phenomenon.
• Statistically this has been demonstrated as
  having occurred widely.

7
Subversion - qualitative evidence

• Schulz has described, anecdotally, a number of
  incidents of researchers subverting allocation by
  looking at sealed envelopes through x-ray lights.
• Researchers have confessed to breaking open
  filing cabinets to obtain the randomisation code.

Schulz JAMA 19952741456.
8
Quantitative Evidence

• Trials with adequate concealed allocation show
  different effect sizes, which would not happen if
  allocation wasnt being subverted.
• Trials using simple randomisation are too
  equivalent for it to have occurred by chance.

9
Poor concealment

• Schulz et al. Examined 250 RCTs and classified
  them into having adequate concealment (where
  subversion was difficult), unclear, or inadequate
  where subversion was able to take place.
• They found that badly concealed allocation led to
  increased effect sizes showing CHEATING by
  researchers.

10
Comparison of concealment
Schulz et al. JAMA 1995273408.
11
Case Study

• Subversion is rarely reported for individual
  studies.
• One study where it has been reported was for a
  large, multicentred surgical trial.
• Participants were being randomised to 5 centres
  using sealed envelopes.

12
Case-study (cont)

• After several hundred participants had been
  allocated the study statistician noticed that
  there was an imbalance in age.
• This age imbalance was occurring in 3 out of the
  5 centres.
• Independently 3 clinical researchers were
  subverting the allocation

13
Mean ages of groups
14
Example of Subversion
15
Concealment

• Both the Schulz and Kjaergard considered sealed
  opaque envelopes to be adequate measures of
  concealment.
• Envelopes can be subverted by being opened in
  advance.

16
More Evidence

• Hewitt and colleagues examined the association
  between p values and adequate concealment in 4
  major medical journals.
• Inadequate concealment largely used opaque
  envelopes.
• The average p value for inadequately concealed
  trials was 0.022 compared with 0.052 for adequate
  trials (test for difference p 0.045).

Hewitt et al. BMJ2005 March 10th.
17
More Examples

• Berger has collected 30 case examples of
  potential subversion of the allocation process in
  clinical trials.
• Because allocation subversion is scientific
  misconduct it is likely that there are many
  other, undetected, cases.

Berger. Selection Bias and Covariate Imbalances
in Randomized Clinical Trials 2005 Wiley,
Chicester.
18
Recent Blocked Trial

• This was a block randomised study (four patients
  to each block) with separate randomisation at
  each of the three centres. Blocks of four cards
  were produced, each containing two cards marked
  with "nurse" and two marked with "house officer."
  Each card was placed into an opaque envelope and
  the envelope sealed. The block was shuffled and,
  after shuffling, was placed in a box.

Kinley et al., BMJ 2002 3251323.
19
What is wrong here?
Kinley et al., BMJ 3251323.
20
Problem?

• If block randomisation of 4 were used then each
  centre should not be different by more than 2
  patients in terms of group sizes.
• Two centres had a numerical disparity of 11.
  Either blocks of 4 were not used or the sequence
  was not followed.

21
Restricted allocation and subversion

• The drawback with any form of allocation
  restriction is that it allows some prediction.
• Simple randomisation has no memory of the
  previous allocation. In contrast, blocked
  allocation allows the probability of an
  allocation to be linked to the previous
  allocation.
• Merely guessing that the next allocation will be
  the opposite of the previous one will result in a
  prediction more accurate than by chance.
• This can, in theory, allow subversion.

22
Possible subversion

• In a RCT of rehabilitation for the treatment of
  hip fracture gross baseline imbalances were
  detected favouring the control group.
• Secure telephone allocation had been used. But
  blocked allocation, size 6, had been used.
• Exploratory analysis of imbalances suggested
  partially successful prediction of block
  allocation.

Turner J. 2002, Unpublished PhD Thesis,
University of York.
23
Wither restricted allocation?

• Simple randomisation followed by analysis of
  covariance (ANCOVA) is as efficient as restricted
  randomisation and ANCOVA for sample sizes 50.
• Restricted allocation increases risk of
  prediction and predictability.
• For large trials simple allocation followed by
  ANCOVA reduces risk of prediction.

Rosenberger WF, Lachin JM. Randomisation in
clinical trials Theory and practice. Wiley
Interscience, 2002, John Wiley and Sons, New York.
24
Subversion - more evidence

• In a survey of 25 researchers 4 admitted to
  keeping a log of previous allocations to try
  and predict future allocations.

Brown et al. Stats in Medicine, 2005,243715.
25
Testing for subversion

• Comparison of baseline characteristics may help
  if subversion is suspected. Although this will
  only identify gross subversion.
• If blocked allocation is used a statistical test
  Bergner-Exner test, may help identify
  subversion.

26
Concealment Recommendations

• Allocation sequence must be independently
  generated and kept secret from the people who are
  enrolling participants.
• A secure method of giving allocation to the
  recruiters must be developed, opaque envelopes
  are inadequate.

27
Subversion - summary

• Appears to be widespread.
• Secure allocation usually prevents this form of
  bias.
• Need not be too expensive.
• Essential to prevent cheating.

28
Secure allocation

• Can be achieved using telephone allocation from a
  dedicated unit.
• Can be achieved using independent person to
  undertake allocation.

29
Technical Bias

• This occurs when the allocation system breaks
  down often due a computer fault.
• A great example is the COMET I trial (COMET II
  was done because COMET 1 suffered bias).

30
COMET 1

• A trial of two types of epidural anaesthetics for
  women in labour.
• The trial was using MIMINISATION via a computer
  programme.
• The groups were minimised on age of mother and
  her ethnicity.
• Programme had a fault.

COMET Lancet 200135819.
31
COMET 1 Technical Bias
32
COMET II

• This new study had to be undertaken and another
  1000 women recruited and randomised.
• LESSON Always check the balance of your groups
  as you go along if computer allocation is being
  used.

33
Attrition Bias

• Usually most trials lose participants after
  randomisation. This can cause bias, particularly
  if attrition differs between groups.
• If a treatment has side-effects this may make
  drop outs higher among the less well
  participants, which can make a treatment appear
  to be effective when it is not.

34
Attrition Bias

• We can avoid some of the problems with attrition
  bias by using Intention to Treat Analysis, where
  we keep as many of the patients in the study as
  possible even if they are no long on treatment.

35
Selection bias after randomisation

• Selection bias is avoided if ALL participants who
  are randomised are completely followed up.
• Often there is some attrition after
  randomisation some refuse to continue to take
  part.
• Or some may refuse the intervention but can still
  be tracked IMPORTANT to distinguish between
  these.

36
What is wrong here?
37
Ascertainment Bias

• This occurs when the person reporting the outcome
  can be biased.
• A particular problem when outcomes are not
  objective and there is uncertainty as to
  whether an event has occurred.
• Example, of homeopathy study of histamine, showed
  an effect when researchers were not blind to the
  allocation but no effect when they were.
• Multiple sclerosis treatment appeared to be
  effective when clinicians unblinded but
  ineffective when blinded.

38
Resentful Demoralisation

• This can occur when participants are randomised
  to treatment they do not want.
• This may lead to them reporting outcomes badly in
  revenge.
• This can lead to bias.

39
Resentful Demoralisation

• One solution is to use a patient preference
  design where only participants who are
  indifferent to the treatment they receive are
  allocated.
• This should remove its effects.

40
Hawthorne Effect

• This is an effect that occurs by being part of
  the study rather than the treatment.
  Interventions that require more TLC than controls
  could show an effect due to the TLC than the drug
  or surgical procedure.
• Placebos largely eliminate this or TLC should be
  given to controls as well.

41
Analytical Bias

• Once a trial has been completed and data gathered
  in it is still possible to arrive at the wrong
  conclusions by analysing the data incorrectly.
• Most IMPORTANT is ITT.
• Also inappropriate sub-group analyses is a common
  practice.

42
Intention To Treat

• Main analysis of data must be by groups as
  randomised. Per protocol or active treatment
  analysis can lead to a biased result.
• Those patients not taking the full treatment are
  usually quite different to those that are and
  restricting the analysis can lead to bias.

43
Sub-Group Analyses

• Once the main analysis has been completed it is
  tempting to look to see if the effect differs by
  group.
• Is treatment more or less effective in women?
• Is it better or worse among older people?
• Is treatment better among people at greater risk?

44
Sub-Groups

• All of these are legitimate questions. The
  problem is the more subgroups one looks at the
  greater is the chance of finding a spurious
  effect.
• Sample size estimations and statistical tests are
  based on 1 comparison only.

45
Sub-Group and example.

• In a large RCT of asprin for myocardial
  infarction a sub-group analysis showed that
  people with the star signs Gemini and Libra
  aspirin was INEFFECTIVE.
• This is complete NONSENSE!
• This shows dangers of subgroup analyses.

Lancet 1988ii349-60.
46
Sub groups

• To avoid spurious findings these should be
  pre-specified and based on a reasonable
  hypothesis.
• Pre-specification is important avoid data
  dredging as if you torture the data enough it
  will confess.

47
Summary

• Despite the RCT being the BEST research method
  unless expertly used it can lead to biased
  results.
• Care must be taken to avoid as many biases as
  possible.