External Validity of Trials

1
External Validity of Trials
2
Background

• External or ecological validity refers to whether
  the results of the trial can be generalised to
  the general clinical population.
• Randomisation per se does NOT make a study
  externally valid.

3
Validity Issues

• Are the correct patients being included in the
  trials?
• Are the correct practitioners taking part in
  the study?
• Are the correct facilities being used?

4
Trial Problems

• Many if not most trials are undertaken in
  circumstances that are far removed from usual
  clinical practice.
• Usually clinicians running trials are content
  experts yet the results are to be applied by
  non-experts.

5
Osteoporosis Trials

• Most of the large osteoporosis trials with
  fracture as an outcome recruited participants
  from bone clinics.
• Patients under care of clinical experts but
  results are expected to be applied by clinicians
  who are not experts.

6
Participants

• Usually participants who go into trials are
  different from those who do not.
• These storm troopers of patients usually comply
  with instructions better than average.
• Combined with different clinicians and patients
  results can be different from the real world.

7
Trial participants v non
 
 
8
Crude risk of fracture
 
9
Risk of fracture (adjusted)
 
10
Alendronate

• Alendronate is a bisphosphonate for osteoporosis
  treatment. Large trials conducted in expert
  clinical centres showed it reduced fractures with
  few or no side-effects.
• Things were different when it was licensed and
  real clinicians used the drug.

11
Alendronate problems

• To take a bisphosphonate one needs to take it on
  an empty stomach with a large glass of water and
  remain upright for at least 2 hours.
• Otherwise the drug can get affect the oesophagus
  and cause ulceration.
• In real life this is what happened and led to a
  warning. Trial data gave to evidence this was a
  problem.

12
The Wrong Solution

• Often once the phase III trial has been finished
  companies finance a real world non-randomised
  study.
• This is usually nothing more than a marketing
  exercise and produces worthless data on real
  world experience of the problem.

13
A correct solution

• Involves some form of randomisation to eliminate
  confounding.
• For example one might undertake a cluster trial
  of giving the drug non-specialist physicians to
  use of their patients BEFORE the drug is widely
  available.
• This will produce more robust data.

14
Alternatively

• We should try and design our trials to be as
  pragmatic as possible so that the results are
  widely generalisable.
• This means recruiting non-expert clinicians and
  their patients.
• Using a pragmatic design rather than an
  explanatory design.

15
Back Pain Trials

• An early RCT of chiropractic manipulation for
  back pain was criticised by physiotherapists that
  the chiropractic was undertaken in private
  practice.
• May have been a Bach effect.

16
MRC back pain trial

• To address the issue of nice premises the MRC
  back pain trial included an arm where patients
  were randomised to be treated on private premises
  or NHS premises.
• Increased the costs of the trial hugely.

17
Surgical Trials

• Often surgical trials have poor external validity
  because the new technique is usually developed in
  a teaching hospital among expert surgeons (e.g.
  laparoscopic methods).
• It is unlikely the results of surgery undertaken
  in top hospitals would be the same as a
  bog-standard DGH.

18
Big and Simple

• Try and make the trial large, as it allows for
  various PRE-SPECIFIED sub-group analyses
  (specialists vs generalists).
• Simple which means one can get as many in as
  possible.

19
Summary

• Most trials will NEVER be able to recruit exactly
  the same types of participants as those who
  receive treatment in clinical practice.
• PRAGMATIC trials have the strongest external
  validity and usually one of these should be done
  before implementation.