Randomized Controlled Trials: Allocation Concealment

1
Randomized Controlled Trials Allocation
Concealment

• Kenneth F. Schulz, PhD

2
Fixation on Sequence Generation

• Traditionally, many medical researchers
  mistakenly consider the sequence generation
  process as randomization
• They frequently ignore allocation concealment
• Without adequate allocation concealment, proper
  random sequences can be subverted
• For example, a properly generated sequence
  posted on a bulletin board

3
Allocation Concealment

• Prevents selection bias by concealing the
  allocation sequence from those assigning
  participants to intervention groups
• Until the moment of assignment

4
Mechanisms for Selection Biasin RCTs

• If those responsible for entering participants
  know, or can detect, the upcoming treatment
  allocations
• Decide entrance based on that knowledge
• Channel participants with a better prognosis to
  the experimental group and those with a poorer
  prognosis to the control group, or vice versa
• In any case, they introduce bias into the
  treatment comparison

5
Do Not Confuse Allocation Concealment with
Blinding

• Allocation concealment seeks to prevent selection
  bias, protects assignment sequence before and
  until allocation, and can always be successfully
  implemented
• In contrast, blinding seeks to prevent
  ascertainment bias, protects sequence after
  allocation, and cannot always be successfully
  implemented

6
Importance of Allocation Concealment

• Unclearly concealed and inadequately concealed
  trials, compared to adequately concealed trials,
  exaggerated the estimates of an interventions
  effectiveness by 30 to 40, on average

Schulz KF, Chalmers I, Hayes RJ, Altman DG.
Empirical evidence of bias dimensions of
methodological quality associated with estimates
of treatment effects in controlled trials. JAMA
1995273408-412.
7
Similar Results on Allocation Concealment in a
Separate Study

• Replicated our methods in different subject areas
  (digestive circulatory diseases and mental
  health)
• Examined 11 meta-analyses
• Inadequately concealed trials exaggerated
  estimates of effectiveness by 37

Moher D, et al. Does quality of reports of
randomised trials affect estimates of
intervention efficacy reported in meta-analyses?
Lancet 1998 352 609-13.
8
Allocation Concealment Summary

• Adequate allocation concealment emerges from our
  analyses as crucial to reducing bias
• Without it, the whole point of randomisation
  vanishes and bias may distort results
• Our results support Mosteller and his colleagues

When the randomization leaks, the trials
guarantee of lack of bias runs down the drain
9
Method of allocation of treatment in 208
controlled trials in head injury Dickinson K, et
al., BMJ 20003201308-1311.
Method of allocation
No. of trials
Adequate
Centralised randomisation by telephone 1
Numbered/coded identical containers administered sequentially 11
Randomisation scheme controlled by pharmacy 8
Sequentially numbered, sealed, opaque envelopes 2

Not adequate
Other 18
Date of birth 1
Day of week 3
Alternation 3

Not stated 161
10
Actual allocation method with the method just
specified as random

• Difficult (Peto and I. Chalmers experience)
• MA of OCs
• David G. knew the PI who put him in contact with
  the statistician
• Described a great method . . . Post-it notes!
• Seemed indignant . . . We asked for clarification
• Perfect also for stratificationdifferent colors!!

11
Personal Accounts of Deciphering Assignment
Sequences

• Conducted over 20 epidemiological workshops for
  medical residents and medical junior faculty
• Each included 20-25 participants
• Asked how many of the participants had
  deciphered, or had witnessed someone else
  decipher, an assignment sequence

12
Personal Accounts of Deciphering Assignment
Sequences (cont.)

• With assurance of anonymity, more than half
  related at least one instance of deciphering
• Not half of all trials
• No accurate denominator
• Nevertheless, not a rare occurrence

13
Decipherings Run the Gamut From Simple to
Intricate

• Simple the most frequent and usually take
  advantage of inadequate allocation concealment
  schemes
• Posting sequence on a bulletin board
• Opening unsealed envelopes
• Translucent envelopes to a light bulb
• Opening many envelopes that were not sequentially
  numbered

14
Intricate Decipherings Less Frequent

• Needed to circumvent more adequate allocation
  schemes
• With sequentially numbered drug containers
• Based on appearance of tablets in unsealed
  containers
• Appearance of the label
• With central allocation, obtaining the next few
  allocations at once
• Envelopes to a hot light in radiology

15
Attempted to Decipher a Numbered Container Scheme

• Gave up
• Found attending physician rifling the P.I.s
  files for the assignment sequence
• Horrified? No, impressed with his brilliance and
  proceeded to help
• Should be kept in a locked location

16
Ignorance Is Not Bliss
At the annual meeting of the Society for
Clinical Trials, held in Houston, Schulz
reported data showing that the experts fears
about ignorance are justified. Schulz examined
250 reports of clinical trials and came up with
an intriguing finding
Rachel Nowak. Problems in clinical trials go far
beyond misconduct. Science 1994 264 1539.
17
Ignorance Is Not Bliss
Most physicians are not trained in basic
scientific principles, let alone clinical trials
Stephen George, Duke University, Chair of the
Statistics Committee for NCI, NIH Cancer Clinical
Cooperative Groups
I fantasize that in the future (training) will
become a requirement for running clinical trials
John Gallin, Director of NIHs Clinical Center
18
Rigorous Trials Annoy Humans

• Investigators
• Certain Ps to benefit
• May want the results of study to reveal the
  truth
• Proper trial procedures attempt to impede human
  inclinations
• Hence, aspects of properly conducted RCTs annoy
  investigators

19
Without Methodological Rigour

• The challenge of deciphering may frequently
  become too great a temptation to resist
• Even without intent to bias
• The only way to get rid of a temptation is to
  yield to it. Oscar Wilde
• Deciphering may just reflect human
  inquisitiveness and ingenuity
• Does not necessarily indicate scientific
  malevolence

20
Without Methodological Rigour

• Or deciphering may reflect deliberate acts to
  alter findings
• Whatever the motivations, innocent or naïve or
  deliberate, those actions undermine the validity
  of the trial
• Investigators must devote diligent, persistent
  attention to randomization

21
Minimal Standards Common Allocation Concealment
Approaches

• Sequentially numbered, opaque, sealed envelopes
  (SNOSE)
• Pharmacy control
• Numbered or coded containers
• Central randomization
• Realistically, these standards should be exceeded

22
Envelopes for Allocation Concealment

• More susceptible to manipulation through human
  ingenuity
• Less than ideal method of allocation concealment
• If used, investigators must diligently develop
  and monitor the process

23
Envelopes (Cont.)

• SNOSE
• Ensure that the envelopes are opened sequentially

• only after the Ps name and other details are
  written on the appropriate envelope

24
Envelopes (Contd)

• Pressure-sensitive or carbon paper inside
• Supports enrolment
• Encourages orderly opening
• Promotes proper assignment
• Creates valuable audit trail
• Cardboard or aluminum foil inside envelope

25
Pharmacy Controlled

• Compliance with randomization/allocation
  concealment methods questionable
• Aware of gross distortions
• Pharmacy ran out of one drug . . .
• Alternate assignment
• Should not assume pharmacists knowledge
• Ensure they, and all research partners, follow
  proper trial procedures

26
Numbered Containers

• Excellent prevents foreknowledge
• Must take proper precautions
• Appearance exactly the same
• Weight exactly the same
• Sound when shaken exactly the same
• Lettering exactly the same
• Odor exactly the same
• Sealed
• Difference between concealment and blinding

27
Central Randomization

• Frequently telephone
• Fax
• Email
• Internet
• Computer
• Assignments only provided after participants
  irrevocably enrolled
• Must check procedures for leaks
• Monitor process for adherence

28
RCTs Anathema to the Human Spirit

• Must acknowledge the vagaries of human nature
• Must establish methodological safeguards that
  thwart attempts to contaminate trials with bias

29
Subversions of Sequences

• Conscious or subconscious?
• Direct or compensatory?
• Doesnt matter
• Biases the trial

30
Participants
Randomize
Placebo
New Oral Drug
60 Compliance
25 Non-compliance
75 Compliance
40 Non-compliance
Group representing the policy of no treatment
Group representing the policy of oral treatment
Outcome Outcome
31
RCT Compared the Effectiveness of Clofibrate in
Preventing Cardiac Deaths in Men Who Had Survived
a Myocardial Infarction
Clofibrate Placebo
5 Year mortality 20.2 20.9 (p .55)
Eliminating deviates from clofibrate (80 adherence) 15.0 20.9 (p lt .05)
Eliminating deviates from both groups 15.0 15.1

• Authors state that
• One can justify almost any conclusion, dependent
  upon the analysis chosen
• Manipulating deviates leads to severe bias
• Can you ever do so?

32
Exclusions After Randomization

• Can introduce bias and should be carefully
  scrutinized
• All randomized patients should be analyzed, and
  analyzed as part of the group to which they were
  initially assigned
• ITT (Intention-to-treat)

33
Exclusions of LFU Damage Internal Validity

• Without outcomes from those lost to follow-up,
  investigators have little choice but to exclude
  them from the analysis
• Any losses damage internal validity
• However, differential rates of loss among
  comparison groups cause major damage
• Investigators must minimize their losses to
  follow-up

34
Retention of Trial Participants

• Minimizing losses exudes difficulties
• Eliminating losses may be impossible
• But investigators too frequently profess
  insurmountable difficulties
• Most should work harder to obtain higher
  follow-up rates
• Investigators must commit adequate attention and
  resources to develop and implement procedures to
  minimize losses

35
Innovative Twists that Cultivate High Follow-up
Rates

• Establish many conveniently placed FU facilities
  
• Too often investigators expect participants to
• visit a single, inconvenient location
• Shortening data collection form caters to the
  participants wishes
• Long sessions lead them to vote with their feet
• Be considerate
• Foster follow-up by not overburdening
• Also may engender higher quality data on the
• main items

36
END
37
Meta-analysis of Polyglycolic
Meta-analysis of polyglycolic acid versus catgut
for perineal repair on short term pain
1 adequately concealed trial OR .89 4
unclearly concealed trials OR .44
Protective effect of polyglycotic acid was 51
lower in the unclearly concealed trial
(ROR .49 95 CI of .35 - .69)
38
Meta-analysis of Corticosteriods
Meta-analysis of corticosteroids vs. none, after
preterm rupture of membranes, on respiratory
distress syndrome (RDS)
3 adequately concealed trials OR 0.72 4
unclearly concealed trials OR 0.53
Protective effect of corticosteroids was
exaggerated by 27 in the unclearly concealed
trials
ROR 0.73 95 CI of 0.35 - 1.50
39
Randomization Process (P 138B)
Proper approach to 4 OB/GYN Journals 4 General Medical Journals
Method of generation of random numbers 32 49
Randomization concealment 23 26
Both 9 15

• In the 4 OB/GYN Journals, found the 2 treatment
  group sizes to be much too similar

Expected Observed
5 ( 5) 0
48 (50) 8
40
Separation of Executer and Generator

• Person(s) who prepare the randomization scheme
  should not determine eligibility, administer
  treatment, or assess outcome
• Neglected critical element
• Obviously importantregardless of methodological
  quality, access to sequence and thus opportunity
  to introduce bias
• Faults in this element may be the crack though
  which much of the bias seeps into controlled
  trials

41
Not quite as random as I pretended
Martyn C. Not quite as random as I pretended.
Lancet 1996 34770.
Bias in the way treatments are assigned can be
a more powerful determinant of the outcome of the
trial than the treatments that are being
investigated.
Trialists often find it hard to prevent their
hopes that a new treatment might prove to be
beneficial from evolving into a belief that it
really is so.
42
RCT in Norway

• Examined the effect of Chlorhexidine compared to
  placebo on postpartum infection and Strep B
  infections in neonates
• Reported as a randomized controlled trial in the
  published article

Insert published article
43
RCT in Norway

• Midwives knew on which shelf the Chlorhexidine
  and placebo were kept in the cabinet and just
  proceeded to administer whatever they deemed
  appropriate
• Even if they forgot the shelf, they could smell
  the difference
• No monitoring of the allocation process

44
Letter from a Physical Therapist

• A friend of mine conducted an RCT comparing 2
  training regimens following surgical repair of
  the knee for anterior cruciate ligament damage
• randomization method used was alternating
  assignment
• Referred to one patient as difficult and asked
  the friend to which group she was assigned
• Friend responded that she was in the group that
  required slightly less contact and that her
  inclusion was by his choice

45
Not so blind, after all

• Sometimes subverting can work against a
  treatments apparent effectiveness

Medical staff may try to help their sickest
patients by sneaking them into the treatment
group instead of the control group.
The drug would then have to be significantly
better than conventional treatment just to appear
equal in efficacy
Wallich P. Not so blind after all. Scientific
American May 1996.
46
RCTs Anathema to the Human Spirit

• Scientific need to obtain unbiased data from an
  inherently biased source human beings
• Many trialists are unaware of the rationale
  behind RCTs
• And the need for trial rigor
• Some may intellectually grasp the need, but once
  immersed in a trial, have many contradictory
  interests (e.g. episiotomy trial in Canada)

47
Anathema A Psychologists View

• Fundamental conflict between scientific and
  human imperatives Many probably think they
  already know what treatment is best.
• Geoffrey Scobie, psychologist at the University
  of Glasgow Doctors cheat because they know
  whats best
• New Scientist, 16 December 95, p. 10
• Of course, what we think we know is frequently
  wrong

48
Reporting of Exclusions, Double-blinding, and
Schedule Generation Related to the Level of
Allocation Concealment for 250 Trials
Authors Reported Adequately Concealed (n79) Unclearly Concealed (n150) Inadequately Concealed (n21)
No Exclusions (Apparent) 53 67 76
Double-blinding 73 39 14
Adequate generation of schedule 29 15 0