Maintaining research rigour in evaluations of complex interventions

1
Maintaining research rigour in evaluations of
complex interventions
Laurence Moore

2
Learning Objectives (1)

• To be aware of frameworks for the development and
  evaluation of complex interventions
• To be aware of value (and added value) of
  complementary mixed methods
• To understand value of pragmatic CRTs with
  embedded process evaluation
• To be aware of examples of successful CRTs of
  complex interventions

3
Learning Objectives (2)

• To understand why cluster randomised trials can
  be the preferred design
• Situations when individual randomisation is not
  feasible
• Contamination
• Limitations of Quasi-experimental designs
• To be aware of specific issues in the conduct and
  analysis of CRTs
• Risk of baseline imbalance
• Intra cluster correlation and the design effect
• To be aware of design and analysis strategies to
  respond to these issues
• Sample size calculations
• Randomisation methods
• Analysis of cluster randomised trials

4
Mixed methods

• Quantitative
• Methods
• Outcome / summative
• Intermediate outcomes / process measures
• Research questions
• What works?
• What is effect?
• How many, how much?

5
Qualitative

• Methods
• Observations, field notes, diaries, records,
  videos, interviews, focus groups
• Process
• Research questions
• Why?
• How?
• Barriers / facilitators

6
Complementary use of mixed methods frameworks in
health

• Fit for purpose
• Match method to question
• Staged, series of studies
• PRECEDE / PROCEED
• Nutbeam model
• MRC framework

7
PRECEDE-PROCEED Framework
Formative evaluation baselines for outcome
evaluation
Intervention Mapping Tailoring
Phase 6 Implementation
Phase 7 Process evaluation
Phase 8 Impact evaluation
Phase 9 Outcome evaluation
New in 4th ed., Green Kreuter, Health
Promotion Planning, in press.
8
Stages of Research and Evaluation for Health
Promotion Programs
Problem Solution Innovation
Intervention Intervention Program definition
Generation Testing
Demonstration Dissemination Monitoring
Epidemiology and demography Social, behavioural
and organisational research Community needs
analysis
Intervention theory development
Intervention literature search, meta- analysis
Assessment of cost and benefits (financial, social
., political) Performance monitoring
Assessment of outcome
Pre- Testing methods and materials
Understanding of Process
What is the How might it Did
the Can the Can the Can the problem?
be solved? solution program
be program be program be work?
repeated/ widely sustained?
refined? reproduced?
Key Research Questions
9
Phases of RCTs of complex interventions MRC
April 2000
10
Complementary use of mixed methods

• Fit for purpose
• Match method to question
• Staged, series of studies
• PRECEDE / PROCEED
• Nutbeam model
• MRC framework
• In combination within one study
• What works?, how?, for whom? and in what
  circumstances?

11
Public Health Improvement Evidence base conundrum

• Good quality trials successfully conducted,
  evaluating weak interventions. Small or zero
  effect sizes.
• Good quality complex interventions evaluated
  using weak research designs. Biased effect
  estimates.

12
Challenges in applying RCTs to evaluation of
complex social interventions

• Recruitment and retention
• Scale and Cost
• Ethics
• Research and Policy Timescales
• Implementation

13
Challenges in applying RCTs to evaluation of
complex social interventions

• Variability in delivery
• Context dependence
• Generalisability / implementation

14
Types of intervention in which individual
randomisation is difficult or impossible

• Interventions that entail changing the
  organisation of services in a given unit or area
• Interventions targeted at changing the behaviour
  of professionals
• Community programmes
• Settings based interventions, such as workplace
  or school interventions
• Interventions targeted at individuals but based
  on social processes

15

• Risk of contamination
• (though see Puffer Torgerson)

16
Problems with quasi-experimental designs

• Selection bias external internal validity
• Imbalance at baseline, often not measurable
• Different trends at baseline
• Ethical considerations

17
Cluster (group) randomised trials

• ASSIST Peer-led smoking intervention -MRC
• 59 schools randomised
• Fruit tuck shops - FSA
• 43 schools randomised
• Free Breakfast Initiative - WAG
• 111 schools randomised
• Emergency contraception - DH
• 25 schools randomised

18
Variability in delivery

• RCTs traditionally require that interventions are
  standardised and uniformly delivered
• (efficacy trial)
• Social interventions highly dependent on quality
  of delivery
• Value of efficacy trials limited
• eg. school smoking education
• Results of efficacy trials involving enthused
  teachers not replicated in roll-out

19
Efficacy and effectiveness

• Efficacy trial
• To test whether the treatment does more good than
  harm when delivered under optimal conditions
• Effectiveness trial
• To test whether the treatment does more good than
  harm when delivered via a real-world program in
  realistic conditions
• Pragmatic, allowing variability in delivery as
  would be experienced in real world

20
Context dependent

• Social interventions often highly dependent on
  the context within which they are delivered
• Argued therefore that RCTs not suited to their
  evaluation
• However, RCT design has the advantage that
  randomisation process ensures that systematic
  differences in external influences between groups
  do not occur
• Generally use stratification or minimisation to
  minimise imbalance due to small no. of units
• Will achieve unbiased estimate of average effect

21
Generalisability

• Efficacy trials may demonstrate that intervention
  has active ingredients that work
• Effect unlikely to be reproduced in real world
• Attenuated by context and implementation
• Generalisability of small trials with (e.g.) one
  educator will be limited

22
Effectiveness trials with embedded process
evaluation

• Effectiveness trials, implementing interventions
  in a manner reproducible in real world
• Realistic level of flexibility allowed, but not
  adaptation or reinvention
• Crucial to conduct a comprehensive process
  evaluation (largely qualitative) within such a
  trial
• Monitor variability in context and delivery
• Identify barriers / facilitators
• Relate variability in these factors to
  variability in intervention impact

23
MRC Assist TrialPeer-led smoking intervention

• Theory based (Diffusion of innovations)
• Developed from similar approach used in sex
  education
• Extensively piloted
• Feasibility trial conducted in 6 schools
• Funding for main trial (59 schools) sought and
  obtained from MRC

24
ASSIST Trial

• Intervention led by specialists, as would be the
  case if rolled out in the real world
• Not to be implemented by untrained, unmotivated
  teachers
• Process evaluation in all 30 intervention
  schools, with parallel measures in the 29 control
  schools
• In-depth process evaluation in sub-sample
• Observations, field notes, diaries, records,
  interviews with pupils, teachers, staff

25
Challenges in embedding process evaluation within
trial

• Hawthorne effects
• Distinguishing team roles
• Differentiating intervention and evaluation
  activities
• Volume of data
• Sampling
• Analysis plan
• Power balance

26
Randomised trials of health promotion
interventions feasible? valuable?

• Not always!
• Cluster randomised design
• Pragmatic, effectiveness trials
• Unbiased estimate of overall intervention effect
• Additional qualitative and quantitative data
  collection to measure variation in context,
  process, delivery and outcome
• Identifies issues for further development of
  intervention / further testing of its (variable)
  effect
• Crucial for implementation stage
• Hypothesis generation, not testing

27
Workshop Analysis of trials (cluster randomised)

• Statistical issues
• Design effect, context, implementer, cluster
  effects
• Multilevel analysis
• Synthesis of qual/quant data

28
Cluster randomisation

• Randomise the cluster rather than the individual
• Generally a small number of clusters
• Four per group an absolute minimum
• Use restricted randomisation to ensure balance in
  number of clusters per group
• Use stratification or minimisation to minimise
  imbalance in group characteristics
• Matched pair design popular, but some drawbacks

29
Standard statistical methods, when applied to
cluster randomised trials, will (usually) lead to

• Sample size calculations that are too small
• Confidence intervals that are too narrow
• P-values that are too small

30
Intra-cluster correlation

• The proportion of the true total variation in the
  outcome that can be attributed to differences
  between the clusters

31
Design effect

• The ratio of the variance of the outcome under
  the cluster sampling strategy to the variance
  that would be expected for a study of the same
  size using simple random sampling
• deff 1 (n-1)?

32
Sample size inflation?0.01, m20

• n deff n
• n 360 (or 18 classes) per group
• deff 1(m-1)?
• 1(20-1).01 1.19
• n 1.19 360 428.4.
• i.e. 429 pupils per group
• Number of classes required 429/20 21.4
• i.e. 22 classes per group

33
Sample size inflation?0.02, m200

• n deff n
• n 360 (or 2 schools) per group
• deff 1(m-1)?
• 1(200-1).02 4.98
• n 4.98 360 1792.8.
• i.e. 1793 pupils per group
• Number of schools required 1793/200 9.0
• i.e. 9 schools per group

34
Further reading

• Donner A, Klar N. Design and analysis of cluster
  randomization trials in health research. London
  Arnold, 2000.
• Murray DM. Design and analysis of
  group-randomized trials. Oxford OUP, 1998.
• Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC,
  Burney GJ, Donner A. Evaluation of health
  interventions at area and organization level. BMJ
  1999319376-379.
• Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC,
  Burney GJ. Methods for evaluating area-wide and
  organisation-based interventions in health and
  health care a systematic review. Health Technol
  Assess 19993(5). (http//www.hta.nhsweb.nhs.uk/).
  
• Elbourne DR, Campbell MK. Extending the CONSORT
  statement to cluster randomised trials for
  discussion. Stats Med 200120489-496.
• Puffer S, Torgerson D, Watson J. Evidence for
  risk of bias in cluster randomised trials review
  of recent trials published in three general
  medical journals. BMJ, Oct 2003 327 785 - 789

35
www.cardiff.ac.uk/schoolsanddivisions/academicscho
ols/socsi/staff/acad/moore/

• F. Starkey, L. Moore, R. Campbell, M. Sidaway,
  M. Bloor. Rationale, design and conduct of a
  comprehensive evaluation of a school-based
  peer-led anti-smoking intervention in the UK the
  ASSIST cluster randomised trial ISRCTN55572965.
  BMC Public Health 2005, 543. 22nd April 2005.
  http//www.biomedcentral.com/1471-2458/5/43
• L. Moore, A. Graham, I. Diamond. On the
  feasibility of conducting randomised trials in
  education case study of a sex education
  intervention. British Education Research Journal
  200329673-689.
• L. Moore, R. Campbell, A. Whelan, N. Mills, P.
  Lupton, E. Misslebrook, J. Frohlich. Self-help
  smoking cessation in pregnancy a cluster
  randomised controlled trial. British Medical
  Journal 20023251383-1386.
• A. Graham, L. Moore, D. Sharp I. Diamond.
  Improving teenagers knowledge of emergency
  contraception results of a cluster randomised
  trial. British Medical Journal 20023241179-1183.
  
• L. Moore, C. Paisley, A. Dennehy (2000) Are fruit
  tuck shops in primary schools effective in
  increasing pupils fruit consumption? A
  randomised controlled trial, Nutrition and Food
  Science 30(1) 35-38.

36
Analysis of trials (cluster randomised)

• Analysis plan
• Multiple outcomes
• Primary and secondary analyses adjust for
  baseline / stratifiers
• A priori plan (register and publish) not
  baseline testing
• To include effect modifiers
• Design effect
• Clustering
• Practitioner effect
• Even in individual RCTs across clusters (context
  effect)
• Multilevel analysis
• Hypothesis generation / informing implementation
• Synthesis
• Triangulation, discordance

37

• Nothing worse than a poorly conducted trial
• Complex intervention trials very challenging
• DO IT WELL GET ADVICE!!

38
Cluster randomised trials

• Laurence Moore
• Cardiff Institute of Society, Health and Ethics
• Email MooreL1_at_cf.ac.uk
• Tel [phone]