Cluster Randomised Trials Of Schools Based Health Interventions

1
Cluster Randomised Trials Of Schools Based Health
Interventions

• What are the barriers to greater use of RCTs in
  educational research?
• Possibilities for progress or Mission impossible?
• Demonstrate by example that RCTs of complex
  educational interventions are feasible

2
Projects

• Trial of fruit tuck shops in primary schools
  (FSA)
• Trial of emergency contraception lessons (NHS
  RD)
• ASSIST Trial of peer-led intervention to reduce
  adolescent smoking (MRC)
• Free breakfast initiative in primary schools in
  Wales (Welsh Assembly Government)

3

• What are the barriers to greater use of RCTs in
  educational research?

4
Challenges in applying RCTs to evaluation of
educational interventions

• Ethical concerns
• Randomisation
• Recruitment and retention
• Scale and Cost
• Variability in delivery
• Context dependent
• Generalisability

5
Ethical concerns

• Often thought unethical to deprive one group of
  people of the innovative intervention, which is
  believed or assumed to be beneficial
• Contrast to medicine where exposure to untested
  new treatments often considered unethical
• Very common in education, despite
• In medicine, target audience is sick and the
  moral imperative to do something (must be better
  then nothing) is great. eg. AIDS, cancer
• Frequent examples of new interventions being
  ineffective or even harmful

6
Ethical concerns (2)

• Is randomisation less fair / ethical than
  postcode lottery or local policy / bid success?
• Only if we are certain that the intervention can
  do no harm should we
• Implement without strong evidence of effect
• Begin to think that randomisation might be
  unethical
• How do we define harm?
• Cost / opportunity cost
• Raised expectations

7
Ethical concerns (3)

• Not wise / moral / prudent / ethical to conduct a
  trial unless one has good reason to believe that
  the intervention may be effective
• Theory
• Formative evaluation
• Principle of equipoise remains

8
Randomisation

• Often impossible / impractical to randomly assign
  individuals to intervention / control groups
• Within one cluster, control subjects liable to be
  contaminated by exposure to some/all
  intervention activities
• Many interventions act explicitly at the cluster
  level (e.g. class, school)
• Randomisation to intervention / control may be
  undertaken at group level (cluster randomisation)
• Usually stratified randomisation or minimisation
  to ensure reasonable baseline balance

9
Cluster randomised trials

• ASSIST Peer-led smoking intervention
• 59 schools randomised
• Fruit tuck shops
• 43 schools randomised
• Free Breakfast Initiative
• 57 schools randomised
• Emergency contraception
• 25 schools randomised

10
Recruitment and retention

• Those recruited to trial should be representative
  of target population
• Participants need to consent to having their
  treatment determined by randomisation
• Thought to be particularly difficult (unethical)
  in cluster randomised trials
• In some cluster trials, those randomised to
  control may then not maintain their commitment to
  study
• Major threat of differential drop-out

11
Recruitment and retention

• Recruit all schools on basis of equal probability
  of being in intervention or control group
• Clear, honest detailed description of research
  activities
• School research contract
• Offer equal reward to both groups
• eg. Control schools given cash buy-out time
• Control schools offered intervention at end of
  measurement period
• Maintain motivation
• briefings, personal contact
• newsletters, prize draws

12
Experience with recruitment and retention of
schools

• Recruitment
• School recruitment easier than anticipated
• Refusal to participate more often due to strong
  preference regarding intervention than objection
  to randomisation or data collection requirements
• Retention
• 5 school cluster randomised trials
• 196 schools
• 1 3 years fieldwork duration
• No school drop-outs
• 2 closures

13
Scale and Cost

• Co-ordination and timeliness
• Major challenge in large scale trials
• Requirement for
• Communication between researchers and
  policy/practice
• Research networks
• Natural experiments
• Innovations in policy / practice introduced in an
  experimental manner, ideally through randomised
  roll-out

14
Scale and Cost (2)

• Trials, particularly cluster randomised trials,
  can be large and expensive
• Intervention costs
• Outcome data collection costs
• Natural experiment no extra intervention costs
• e.g. Free Breakfast Initiative
• Use of routinely collected outcome data
• Education has unexploited resource
• Frequently, trials can be very low-cost

15
Variability in delivery

• RCTs traditionally require that interventions are
  standardised and uniformly delivered
• (efficacy trial)
• Educational interventions highly dependent on
  quality of delivery
• Value of efficacy trials limited
• eg. school smoking education
• Results of efficacy trials involving enthused
  teachers not replicated in roll-out

16
Efficacy and effectiveness

• Efficacy trial
• To test whether the treatment does more good than
  harm when delivered under optimal conditions
• Effectiveness trial
• To test whether the treatment does more good than
  harm when delivered via a real-world program in
  realistic conditions
• Pragmatic, allowing variability in delivery as
  would be experienced in real world

17
Context dependent

• Educational interventions often highly dependent
  on the context within which they are delivered
• Argued therefore that RCTs not suited to their
  evaluation
• However, RCT design has the advantage that
  randomisation process ensures that systematic
  differences in external influences between groups
  do not occur
• Will achieve unbiased estimate of average effect

18
Generalisability

• Efficacy trials may demonstrate that intervention
  has active ingredients that work
• Effect unlikely to be reproduced in real world
• Attenuated by context and implementation
• Generalisability of small trials with one
  educator in one school will be limited

19

• Possibilities for progress or Mission Impossible?

20
Public Health Improvement Evidence base conundrum

• Good quality trials successfully conducted,
  evaluating weak interventions. Small or zero
  effect sizes.
• Good quality complex interventions evaluated
  using weak research designs. Biased effect
  estimates.

21
When do we do RCTs?

• In medicine, there are distinct phases in the
  development evaluation of new interventions
  (eg. drugs)
• Basic research (eg. molecular, genetic)
• Applied research development (eg.
  pharmacological)
• Trials to determine efficacy
• Trials to determine effectiveness
• Post-marketing surveillance

22
Phases of RCTs of complex interventions MRC
April 2000
23
MRC Assist TrialPeer-led smoking intervention

• Theory based (Diffusion of innovations)
• Developed from similar approach used in sex
  education
• Extensively piloted
• Feasibility trial conducted in 6 schools
• Funding for main trial (59 schools) sought and
  obtained from MRC

24
Effectiveness trials with embedded process
evaluation

• Effectiveness trials, implementing interventions
  in a manner reproducible in real world
• Crucial to conduct a comprehensive process
  evaluation (largely qualitative) within such a
  trial
• Monitor variability in context and delivery
• Identify barriers / facilitators
• Relate variability in these factors to
  variability in intervention impact

25
Fruit tuck shop trial

• Minimisation used to ensure balance in terms of
• School size
• School policy on snacks
• Schools given minimal support in setting up tuck
  shops, with wide variability in detailed
  operation
• Detailed process evaluation
• Environment of school and locality
• Operation of fruit tuck shops
• Detailed case studies of 8 selected schools
• Observation, interview, focus groups

26
ASSIST Trial

• Intervention led by specialists, as would be the
  case if rolled out in the real world
• Not to be implemented by untrained, unmotivated
  teachers
• Process evaluation in all 30 intervention
  schools, with parallel measures in the 29 control
  schools
• In-depth process evaluation in sub-sample
• Observations, field notes, diaries, records,
  interviews with pupils, teachers, staff

27
Free Breakfast Initiative Trial

• 111 schools across 9 LEAs
• Variable models of staffing and delivery
• Trial powered to identify overall mean effect on
  dietary and behavioural outcomes
• Process evaluation to monitor variation in
  delivery and identify strengths and weaknesses

28
A role for RCTs in evaluating health education
interventions?

• RCTs not always possible!
• Difficult to do well, and can be expensive
• Take opportunity of natural experiments
• Theory-driven development, formative evaluation
  and feasibility studies essential prerequisites
  prior to trial
• Get the intervention right

29
Research design

• Cluster randomised design
• Pragmatic, effectiveness trials
• Unbiased estimate of overall intervention effect
• Additional qualitative and quantitative data
  collection to measure variation in context,
  process, delivery and outcome
• Identifies issues for further development of
  intervention / further testing of its (variable)
  effect
• Hypothesis generation, not testing

30
The end.

• Stanley (1957)
• Expert opinions, pooled judgements, brilliant
  intuitions and shrewd hunches are frequently
  misleading

• MacIntyre Petticrew (2000)
• Good intentions and received wisdom are not
  enough

Laurence Moore Cardiff Institute of Society,
Health and Ethics Email MooreL1_at_cf.ac.uk Tel
02920 875387