Avoiding bias in RCTs

1
Avoiding bias in RCTs

• David Torgerson
• Director, York Trials Unit
• djt6_at_york.ac.uk
• www.rcts.org

2
A reminder

• Randomised trials are needed for 4 reasons
• Avoiding selection bias
• Controlling for temporal changes
• Controlling for regression to the mean
• Basis for statistical inference.

3
Background

• Randomisation, when undertaken properly, prevents
  selection bias.
• Selection bias occurs when participants are
  allocated in such a way that allocation
  correlates with outcome.
• Selection bias is one of the main threats to
  validity that randomisation seeks to avoid.
• However, forms of selection and other sources of
  bias can still undermine the validity of a RCT.

4
Non-random methodsAlternation

• Alternation is where trial participants are
  alternated between treatments.
• EXCELLENT at forming similar groups if
  alternation is strictly adhered to.
• Problems because allocation can be predicted and
  lead to people withholding certain participants
  leading to selection bias.

5
Non-Random MethodsQuasi-Alternation

• Dreadful method of forming groups.
• This is where participants are allocated to
  groups by month of birth or first letter of
  surname or some other approach.
• Can lead to bias in own right as well as
  potentially being subverted.

6
Example Quasi alternation

• Before mailing, recipients were randomized by
  rearranging them in alphabetical order according
  to the first name of each person. The first 250
  received one scratch ticket for a lottery
  conducted by the Norwegian Society for the Blind,
  the second 250 received two such scratch tickets,
  and the third 250 were promised two scratch
  tickets if they replied within one week.(Finsen
  and Storeheier, Biomed Central 2006) 

7
Randomisation

• Some text books typically suggest the use of
  random number tables or coin tossing to allocate
  participants.

8
What is wrong with that?

• Coin tossing there is no audit trail. We have
  to take your word for what you did.
• Random number tables are open and again we have
  to take the researchers word that they did what
  the said.
• A separate computer allocation system from a
  third party is best.

9
Simple or restricted?

• Simple allocation is best when the number of
  units to be allocated gt50. Because it is always
  unpredictable it is difficult to subvert or
  sabotage.
• Restricted randomisation has statistical
  advantages for lt50 (e.g., cluster trial of 20
  schools/classes would benefit from using a
  restricted technique such as blocking or
  minimisation).

10
Will people subvert the allocation?

• Schulz 1 has described, anecdotally, a number
  of incidents of researchers subverting allocation
  by looking at sealed envelopes through x-ray
  lights.
• Researchers have confessed to breaking open
  filing cabinets to obtain the randomisation code.
• In a survey 2 of 25 researchers 4 admitted to
  keeping a log of previous allocations to try
  and predict future allocations.

1 Schulz JAMA 19952741456. 2 Brown et al.
Stats in Medicine, 2005,243715.
11
Case Study

• Subversion is rarely reported for individual
  studies.
• One study where it has been reported was for a
  large, multicentred surgical trial.
• Participants were being randomised to 5 centres
  using opaque, sequentially numbered, sealed
  envelopes.

12
Case-study (cont)

• After several hundred participants had been
  allocated the study statistician noticed that
  there was an imbalance in age.
• This age imbalance was occurring in 3 out of the
  5 centres.
• Independently 3 clinical researchers were
  subverting the allocation.

13
Mean ages of groups
Kennedy Grant. 1997Controlled Clin Trials
18,3S,77-78S
14
Recent Blocked Trial

• This was a block randomised study (four patients
  to each block) with separate randomisation at
  each of the three centres. Blocks of four cards
  were produced, each containing two cards marked
  with "nurse" and two marked with "house officer."
  Each card was placed into an opaque envelope and
  the envelope sealed. The block was shuffled and,
  after shuffling, was placed in a box.

Kinley et al., BMJ 2002 3251323.
15
What is wrong here?
Kinley et al., BMJ 3251323.
16
Problem?

• If block randomisation of 4 were used then each
  centre should not be different by more than 2
  patients in terms of group sizes.
• Two centres had a numerical disparity of 11.
  Either blocks of 4 were not used or the sequence
  was not followed.

17
Randomisation summary

• Very important to avoid possible problems of
  subversion
• Who do you trust?
• Need independent allocation, third party, need to
  be convincing.

18
Selection bias after randomisation

• Selection bias is avoided if ALL participants who
  are randomised are completely followed up.
• Often there is some attrition after
  randomisation some refuse to continue to take
  part.
• Or some may refuse the intervention but can still
  be tracked IMPORTANT to distinguish between
  these.

19
What is wrong here?
20
Selection bias

• POSSIBLE selection bias has been introduced in
  this trial.
• About 10 of the sample have been non-randomly
  allocated to the interventions.
• What should have happened?
• The authors should have retained the 17 refusers
  in an intention to teach (treat) analysis.

21
Ascertainment bias

• This occurs when the person reporting the outcome
  can be biased.
• A particular problem when outcomes are not
  objective and there is uncertainty as to
  whether an event has occurred.
• Example, of homeopathy study of histamine, showed
  an effect when researchers were not blind to the
  allocation but no effect when they were.
• Multiple sclerosis treatment appeared to be
  effective when clinicians unblinded but
  ineffective when blinded.

22
Avoiding ascertainment bias

• Post-tests should be administered by someone who
  is unaware of the group allocation. Pre-tests
  before allocation is made.
• Record searches (e.g., arrest records) should be
  done by a researcher blind to the participants
  group allocation.

23
Resentful Demoralisation

• This can occur when participants are randomised
  to treatment they do not want.
• This may lead to them reporting outcomes badly in
  revenge.
• This can lead to bias.

24
Resentful Demoralisation

• One solution is to use a participant preference
  design where only participants who are
  indifferent to the treatment they receive are
  allocated.
• This should remove its effects.

25
Example of preference interaction

• SPRINTER was an RCT of treatments for neckpain.
• Two treatments a Brief Intervention (1-2
  sessions with a physio using CBT) vs usual care
  (5 sessions).
• BEFORE randomisation we asked patients their
  treatment preference.

26
SPRINTER Preferences

• In SPRINTER preferences were mixed
• 53 did not have a preference
• 16 wanted brief intervention
• 31 wanted usual care.
• ALL patients were randomised IRRESPECTIVE of
  their preference.

27
Patient Flow and 12 month Results - SPRINTER
Overall 12 month improvement -0.840
Overall 12 month improvement -2.825
28
SPRINTER Results
29
Examples of educational trials

• As part of a collaboration between educational
  researchers and trial methodologists we have
  completed two RCTs in education.
• We tried to conduct them according to health care
  standards (i.e., CONSORT).
• Computers to teach spelling to children
• Incentives to retain adult learners in evening
  classes.

30
Computer trial

• Since 1997 the Government has boasted that gt 1.7
  billion of taxpayers money has been put into
  equipping schools with computer technology. Very
  few RCTs have been undertaken to evaluate and
  substantiate this investment.
• We supported the largest RCT of computer
  technology ever undertaken in the UK.

31
Methodological challenges

• Contamination children in the control group may
  access the technology.
• Deliver intervention on a lap top.
• Secure randomisation.
• Randomisation done by York Trials Unit series
  of blocked allocation to match availability of
  laptops (pre-test data did not inform
  allocation).
• Avoiding ascertainment bias.
• Post-tests given to all children in same room by
  invigilators blind to allocation, test marked by
  marker blind to group allocation. Pre-tests done
  blindly.
• Resentful demoralisation.
• Waiting list control control children to get
  intervention at end of term.

32
Methods

• Based in one school. Lack of funding meant that
  it couldnt be extended elsewhere. All year 7
  pupils were randomised to receive literacy
  instruction via computer or not.
• Sample size (n 157) allowed us to have gt 80
  power to show 0.5 of effect size between groups.
  Sample size fixed we couldnt increase it, if we
  had wanted.
• Waiting list design all pupils received
  intervention half at the beginning of term half
  later in the term (normal practice was to
  arbitrarily assign pupils).

33
Participant Flow in Trial
34
Results
Brooks et al, Ed Studies 2006
35
Computers and Literacy

• In this study we found no benefit of using
  computers for teaching literacy skills.
• A larger recently completed trial in the USA (n
  512) found similar results no benefit.
• gt 90 million had been spent implementing Fast
  forword before it was found to be ineffective.

Rouse et al, 2004 NBER Working Paper Series,
10315.
36
Incentives

• Poor attendance to adult literacy and numeracy
  classes is a major problem.
• Recent government policy has offered either
  financial incentives to attend or financial
  penalties (reduction in benefit) if attendance is
  poor.
• Neither approach has been tested in an RCT.
• With colleagues in Sheffield we undertook a RCT
  of incentives.

37
Methodological challenges

• Incentive may lead to resentful demoralisation if
  we randomised individuals.
• Cluster or class allocation of 28 classes.
• Small number of units of allocation may result in
  chance bias or numerical imbalance.
• Used a minimisation algorithm to achieve balance
  on observed variables.

38
Design

• Cluster or class based RCT.
• 28 classes were randomised to either receive 5
  per class attended or no incentive.
• Outcome was number of sessions attended.

39
Significant reduction of about 1.5 sessions (95
confidence interval 0.28, 2.79 p 0.019)
(adjusted for clustering, cluster size and
pre-test literacy scores
40
Incentives

• Small 5 incentives are counter productive and
  reduce attendance.
• Larger incentives may work but we need to
  evaluate these in a large RCT with educational
  achievement as main outcome.

41
Differences between health and education

• Education trials are much easier and quicker.
• Pre-test variables strongly predict post-test,
  which usually isnt the case in health care
  trials, consequently pre-test measurement is
  particularly useful to gain more statistical
  power.

42
Trial designs

• Unequal allocation
• Good to use when resources limit us to the amount
  of one intervention (e.g., mentoring sessions).
• Cluster designs
• Avoiding contamination, care needs to be taken on
  using ITT and recruitment bias.
• Factorial designs
• Two trials for the price of one good value!
• Balanced design
• Controls for Hawthorne effect (e.g., Intervention
  group gets maths, control group gets English and
  2 trials for the price of one).

43
Factorial Design
44
Challenges

• Follow-up this is crucial to maximise this.
• Schools are good children are there and apart
  from (random?) absences most will be present for
  post-test
• Adults may need encouragement (e.g., possible
  incentives to complete follow-up) or statutory
  data collection (e.g., arrests) allows ITT to be
  completed.
• Convincing other researchers that RCTs are
  feasible and worth doing.

45
A few fallacies

• We cant accurately measure an important variable
  (e.g., social class) that really predicts outcome
  our trial will be biased.
• No it wont, good measurement of confounders may
  be helpful but randomisation will cancel their
  effects out.
• Non compliance means randomisation is not
  possible.
• Poor compliance will dilute the treatment
  effects, however, using ITT analysis will give a
  public policy estimate of effect and other
  statistical techniques (e.g., CACE) may adjust
  for non-compliance.

46
Can trials overcome strong beliefs?

• Often not.
• RCT of 25 schools to be given an enhanced sex
  education programme aim to reduce abortions
  among teenage women. New package cost 900 per
  teacher compared with 20 for conventional
  package.

47
Results

• Termination rate among intervention schools was
  higher (15.7 per 1,000 women 95 CI -10.7 to
  42.1, p 0.26).
• Difference not statistically significant, but
  point estimate is a 14 increase risk of
  terminations.

48
What did the authors conclude?

• High quality sex education should be continued,
  but to reduce unwanted pregancies complementary,
  longer term interventions that address
  socioeconomic inequalities and the influence of
  parents should be developed and rigorously
  evaluated.
• How does this recommendation derive from the data?

49
Summary

• Trials have challenges some simple mistakes can
  result in a trial losing its internal validity.
• Many of the problems of trials can be overcome
  with careful thought and preparation.