On the interpretation of responder analyses and NNTs

1
On the interpretation of responder analyses and
NNTs

• Stephen Senn

2
An apology

• I will talk mainly about responder analysis
• I have little to say about numbers needed to treat

3
Genes, Means and Screens
It will soon be possible for patients in clinical
trials to undergo genetic tests to identify those
individuals who will respond favourably to the
drug candidate, based on their genotype. This
will translate into smaller, more effective
clinical trials with corresponding cost savings
and ultimately better treatment in general
practice. individual patients will be targeted
with specific treatment and personalised dosing
regimens to maximise efficacy and minimise
pharmacokinetic problems and other side-effects.

Sir Richard Sykes, FRS, 1997
4
Soon?
5
(No Transcript)
6
(No Transcript)
7
The Pharmacogenomic Revolution?

• Clinical trials
• Cleaner signal
• Non-responders eliminated
• Treatment strategies
• Theranostics
• Markets
• Lower volume
• Higher price per patient day

8
Implicit Assumptions

• Most variability seen in clinical trials is
  genetic
• Furthermore it is not revealed in obvious
  phenotypes
• Example height and forced expiratory volume
  (FEV1) in one second
• Height predicts FEV1 and height is partly
  genetically determined but you dont need
  pharmacogenetics to measure height
• We are going to be able to find it
• Small number of genes responsible
• Low (or no) interactive effects (genes act
  singly)
• We will know where to look
• In fact we simply dont know if most variation in
  clinical trials is due to individual response let
  alone genetic variability

9
My Opinion

• Most of the hype is due to a failure to
  understand response
• Responder analysis is to blame
• And related to this is an obsession with Numbers
  Needed to Treat
• Which is increasing the pressure to use
  dichotomies

10
A Thought Experiment

• Imagine a cross-over trial in hypertension
• Patients randomised to receive ACE II inhibitor
  or placebo in random order
• Then we do it again
• Each patient does the cross-over twice
• We can compare each patients response under ACE
  II to placebo twice

11
(No Transcript)
12
NB These are conditional probabilities of
response on the second occasion. They are not
conditional probabilities of being a true
responder.
13
(No Transcript)
14
(No Transcript)
15
?
16
NOTE FOR GUIDANCE ONCLINICAL INVESTIGATION OF
MEDICINAL PRODUCTSIN THE TREATMENT OF
HYPERTENSION 1998 P2 Arbitrarily, response
criteria for antihypertensive therapy include the
percentage of patients with a normalisation of
blood pressure (reduction SBP lt 140 mmHg and DBP
lt 90 mmHg) and/or reduction of SBP 20 mmHg
and/or DBP 10 mmHg. Results obtained should be
discussed in terms of statistical significance
and in relation to their clinical relevance.
The first word in this paragraph is the most
important
17
Dichotomania

• Continuous measurements taken and referred to
  baseline
• Patients dichotomised as responder/non- responder
• Inefficient
• Arbitrary
• Sheep versus goats
• Ignores geep and shoats
• Analysis on risk difference scale to calculate NNT

18
Y outcome, X baseline If (Y lt 90 ? X gt 95) ?
(Y lt 0.9X) patient responds
19
Why I mistrust the NNT

• It has very poor properties as a scale
• Reciprocal of risk difference
• It is theoretically unlikely to be stable from
  study to study
• And this theoretical instability has been
  demonstrated by empirical research
• It is an impatient measure
• It tries to shortcut the steps from study to
  practice
• It is an illusion that this can be done
• Those who advocate it are preferring an easy lie
  to a difficult truth

20
Pharmacogenetics A cutting-edge science that
will start delivering miracle cures the year
after next.
21
Moerman and Placebos

• Paper of 1984
• Investigated 31 placebo-controlled trials of
  cimetidine in ulcer
• Found considerable variation in response
• Considered placebo response rate was an important
  factor
• Has been cited by others as proof of variation in
  treatment effect from trial to trial

22
(No Transcript)
23
Lessons from Moerman

• There is no evidence of variation in the
  treatment effect from trial to trial
• We should be wary about concluding that apparent
  variation signals true variation
• We need to be cautious and think carefully about
  analysis
• Of courseit is always possible that there was
  exactly the same genetic mix in each trial
• in which case gene by treatment would not
  manifest itself as trial by treatment interaction
• We need to understand components of variation

24
Pharmacogenomics A subject with great promise.
25
What you learn in your first ANOVA course

• Completely randomised design
• One way ANOVA
• Randomised blocks design
• Two way ANOVA
• Randomised blocks design with replication
• Two way ANOVA with interaction
• No replication, no interaction

26
1. Senn SJ. Individual Therapy New Dawn or False
Dawn. Drug Information Journal 200135(4)1479-149
4.
27
(No Transcript)
28
A Word of Caution

• What is additive on one scale is not additive on
  another
• The Moerman example suggests a constant effect on
  the log-odds ratio scale
• If the background risk varies this translates
  into a varying effect on the risk-difference
  scale
• The biological interpretation of this is then
  moot
• However the practical implication of this is
  summarise on the additive scale

29
(No Transcript)
30
The Mottos

• Additive at the point of study
• Relevant at the point of application
• If NNTs have their place it is in decision making
  for individual patients
• Not in reporting results from individual trials
• The additive scale has to be transformed into the
  relevant scale at the point of treatment
• The fact that NNTs might be relevant when making
  an individual decision is not an excuse for
  summarising results this way

31
Tiotropium v Placebo in Chronic Obstructive
Pulmonary Disease
From the UPLIFT Study, NEJM, 2008
Significant differences in favor of tiotropium
were observed at all time points for the mean
absolute change in the SGRQ total score (ranging
from 2.3 to 3.3 units, Plt0.001), although the
differences on average were below what is
considered to have clinical significance (Fig.
2D). The overall mean between-group difference in
the SGRQ total score at any time point was 2.7
(95 confidence interval CI, 2.0 to 3.3) in
favor of tiotropium (Plt0.001). A higher
proportion of patients in the tiotropium group
than in the placebo group had an improvement of 4
units or more in the SGRQ total scores from
baseline at 1 year (49 vs. 41), 2 years (48
vs. 39), 3 years (46 vs. 37), and 4 years (45
vs. 36) (Plt0.001 for all comparisons). (My
emphasis)
32
Two Normal distributions with the same spread but
the Active treatment has a mean 2.7 higher. If
this applies every patient under active can be
matched to a corresponding patient under placebo
who is 2.7 worse off
33
A cumulative plot corresponding to the previous
diagram. If 4 is the threshold, placebo response
probability is 0.36, active response probability
is 0.45.
34
In summarythis is rather silly

• If there is sufficient measurement error even if
  the true improvement is identically 2.7, some
  will show an improvement of 4
• The conclusion that there is a higher proportion
  of true responders by the standard of 4 points
  under treatment than under placebo is quite
  unwarranted
• So what is the point of analysing responders?

35
Who are the authors?
1. Tashkin, DP, Celli, B, Senn, S, Burkhart, D,
Kesten, S, Menjoge, S, Decramer, M. A 4-Year
Trial of Tiotropium in Chronic Obstructive
Pulmonary Disease, N Engl J Med 2008.
Personal note. I am proud to have been involved
in this important study and have nothing but
respect for my collaborators. The fact that,
despite the fact that two of us are
statisticians, we have ended up publishing
something like this shows how deeply ingrained
the practice of responder analysis is in medical
research. We must do something to change this.
36
In conclusion

• Responder analysis is the source of much
  confusion
• It is leading trialists to overestimate the
  individual element of response to treatment
• The key to understanding response is replication
  and careful analysis
• Stupid dichotomies do not help this understanding
• NNTs may be relevant at the point of application
  but they are not relevant at the point of study
• Personalised medicine may be about to happen
  soon for quite a few years to come yet