What is wrong with What is said?

1
What is wrong with What is said?

• Hasan Yazici, MD
• University of Istanbul

2
Vitamin-E in goutthe Hypothesis

• Several recent reports hint that vitamin E can
  lower serum uric acid levels (1-3).
• These are either single case reports or
  uncontrolled, open studies in small groups of
  patients.
• Thus we tested the hypothesis to see whether
  vitamin E was effective treatment for gout in a 6
  month, double blind, placebo controlled study.

3
What is wrong?

• A hypothesis should not be formulated as a
  question.

4
Methods - 1

• The patients were randomized into the vitamin E
  (n 60) and placebo (n 60) groups.
• After randomization each patient signed a written
  informed consent.
• The patients were assessed monthly with serum
  uric acid, hematocrit, serum creatinine,
  cholesterol and testosterone levels.
• The number of gout attacks were also noted.

5
What is wrong?

• The randomization follows, not precedes, the
  informed consent.

6
Methods -2

• 30 patients had to leave the study for various
  reasons.
• 20 patients from the active drug group and 10
  from the placebo left the study before its
  completion (ns).
• Thus the final analysis for drug efficiency was
  made among 90 patients.
• Among the 20 patients who dropped out from the
  treatment group 3 moved to another city.
• 15 complained of severe indigestion and no
  information was available for the remaining 2.
• Dyspepsia was the reason for withdrawal in 7
  patients in the placebo group while headaches
  were the main reason for discontinuation of
  treatment in the remaining 3.

7
What is wrong?

• The intention to treat principle is neglected.

8
Intention to treatanother study

• 90 patients were initially randomized into 30
  patients each of placebo, QXY 2mg qd and QXY 5mg
  qd groups.
• One patient in the QXY 5mg group developed
  pneumonia and had to be hospitalized 10 days
  after treatment started.
• The patient was withdrawn and, according to the
  protocol, another patient was recruited.
• Thus, the intention to treat analysis brought the
  number analyzed to 31 in the QXY 5mg group and
  the total number to 91.

9
What is wrong?

• How was randomization achieved in the newcomer?
• The newcomer inflates the denominator when
  looking at harm.

10
Methods-3

• The allocation of 60 patients to the active drug
  and the placeo arms ensured a 86 power to detect
  a difference rate of 25 in the uric acid levels
  between the 2 groups using a 2-sided Fishers
  exact test with a set at 0.05

11
What is wrong?

• In a power calculation it is not enough to state
  the magnitude of change, the anticapated event
  rates should also be given.
• Ensured 86 power?

12
Sample size calculations 4 components

• I. Type I (alpha) error
• II. Type II (beta) error
• III. Event rate in the control group
• IV. Event rate in the treatment group

13
Power calculations for efficacy and harm

• The primary end point was the achievement of at
  least 25 improvement according to the ACR
  criteria (ACR20) at week 20.
• The sample size of 180 patients per group was
  chosen to ensure an adequate safety evaluation.
• The sample size also ensured that there was 90
  power to detect a significant difference in the
  proportion of ACR20 responders between the
  treatment groups using a significance level of
  0.05, assuming 20 and 42 of the patients in the
  control and the treatment groups respectively
  achieved ACR 20 responses.

14
What is wrong?

• No basis for the assumptions related to harm
• Post hoc power calculations

15
Powering for safety

• A sample size of 300 patients each in the study
  drug and the control groups was determined to
  demonstrate a specific adverse event rate of 1
  or less with 95 confidence.

16
What is wrong?

• The probability of an event not happening does
  not give us information about the likelihood of
  events happening in the different arms of the
  study.

17
The zero patient method

• If one screens n individuals and does not find an
  attribute y among this group
• Then one can conclude that the frequency of y is
  less than 1/0.33n among the n individuals
  surveyed, with 95 confidence.
• This approximation is true for prevalances lt
  0.02 of the attribute surveyed.
• H Yazici et al
  Rheumatology Oxford (2001)

18
Efficacya subgroup analysis

• The uric acid levels decreased to 5.2 1.5mg/dL
  from a baseline of 9.3 2.3 mg/dL in the
  treatment and from 9.7 1.9 mg/dL in the placebo
  groups.
• There were no differences in the lowering of uric
  acid between the 2 groups (pgt0.05).
• There were also no differences in the number of
  gout attacks between the 2 groups of patients.
  However a subgroup analysis was also done.
• Among those patients in the treatment group who
  gave a history of more than 5 gouty attacks per
  year, as compared to those who had less than 3,
  the uric acid levels were significantly lower
  after treatment (plt 0.05).

19
Reviewer comments

• The findings in the subgroup analysis, conducted
  among a small number of patients, should be
  interpreted with caution.
• Even though the authors found a statistically
  significant difference in efficacy the study was
  not primarily planned to assess this difference.
• This lessens the external validity of these
  results.

20
What is wrong?

• The main problem with this subgroup analysis
  is not that the subgroup is small. It is an
  effort in the direction of proving that the new
  drug at hand is effecacious.

21
Harm another subgroup analysis

• Table 3 gives the rate of observed adverse
  events. Indigestion was rather frequent in either
  group 15/45 patients in the treatment and 14/45
  in the placebo groups.
• A subgroup analysis was also done after the
  patients who participated in the trial were
  questioned about a pre-trial history of
  indigestion. It turned out that 20/45 patients in
  the treatment and 18/45 in the placebo group had
  pre-trial indigestion.
• A further analysis among these patients revealed
  that 14/20 in the treatment and 2/18 in the
  placebo group (plt 0.03) with a history of
  pre-trial indigeston also reported indigestion
  during the trial.

22
Reviewer comments

• The findings in the subgroup analysis, conducted
  among a small number of patients, should be
  interpreted with caution.
• Even though the authors found a statistically
  significant difference in harm the study was not
  primarily planned to assess this difference. This
  lessens the external validity of these results.

23
What is wrong?

• Again the main issue is not the size of the
  subgroup. This subgroup analysis looks at the
  possibility that this new drug might not be all
  that harmless. Thus the exercise is in the
  direction of falsification and therefore is
  justified.

24
Dr. Pincus study

• Recently, in a placebo controlled withdrawal
  study, Dr. Pincus showed that small doses of
  prednisone (1- 4 mg/day) were significantly
  effective in the management of rheumatoid
  arthritis.
• The study was conducted among 31 patients

25
Reviewer comments

• This beneficial effect of prednisone described
  among a small number of patients should be
  interpreted with extreme caution even if the
  authors found a statistically significant
  difference. The number of patients studied, thus
  the study power, was simply too small.

26
What is wrong?

• The reviewer is partially right.
• The issue of power applies only if we are missing
  a more important outcome that would have been
  more evident in a larger group.
• External validity? The small number of patients
  might be quite different from the real life RA
  patients.
• However (!) this is not simply an issue of
  numbers but of patient selection.

27
The coxib study

• 6000 patients with osteoarthritis of the knee
  were randomized to recieve either the new coxib
  (NCB) or the traditional (TCB).
• 40 of the NCB and 35 of the TCB patients found
  total pain relief.
• Hypertension was a problem in 2.5 of the NCB
  and 4.5 of the TCB patients.
• It was concluded
• a. NCB decreased pain by 13
• b. There was 45 less hypertension with NCB

28
What is wrong?

• Nothing. Needs more interpretation.
• A NNT analysis will tell you that you have to
  treat 10 patients to see the superiority of NCB
  over TCB in 1 patient.
• A NNH analysis will say that you have to treat 50
  patients with TCB to harm 1 more patient with
  hypertension as compared to using NCB.

29
NNT NNH

• Relative risk (RR) outcome ratio in the control
  group/ outcome ratio in the control group
• Absolute risk reduction (ARR) difference in the
  ratios
• Relative risk reduction (RRR) ARR/outcome in the
  control group OR 1-RR
• NNH 1/ARR (frequency of hypertension in the
  control group 0.045, frequency of hypertension in
  the study group 0.025, ARR 0.020 NNT 50.)

30
A withdrawal study

• Two previous double blind studies of colchicine
  in BS had shown no superiroity of this agent over
  placebo in treating the oral ulcers in this
  condition.
• Recently a withdrawal study was done among those
  patients who claimed benefit from colchicine.
  They were randomized to contiune to receive the
  active drug or placebo.
• After 3 months those who stopped taking
  colchicine had significantly more ulcers (sign
  test, p 0.02).

31
Reviewer comments

• This study is interesting but is of limited use.
  A problem with all withdrawal trials, they
  seldom represent real life use.
• The authors used the sign test to analyze the
  differences in oral ulcers between the 2 groups.
  Is this a new test? I suspect it is not quite
  powerful. Why not use the more powerful tests of
  significance to show the real differences, if
  any?

32
What is wrong? (I)

• The main problem with withdrawal studies is that
  they are not done often enough.
• They provide excellent information about possible
  a type II error in previous, traditional RCTs.
• An important issue is that they do not give a
  fair picture of drug associated harm.

33
What is wrong? (II)

• The sign test is a time honored, conservative
  tool.
• Significance in a conservative test gives more
  validity to the significant differences observed.
  

34
The mighty p

• Among the 96 patients allocated to the new
  medication there were 3 cases of myocardial
  infarction while the same was true for 2/94
  patients allocated to placebo (p0.86).

35
What is wrong?

• Better give it with the statistic used (sign
  test, p 0.02).
• Do not use it to mesmerize the reader.

36
The Commandments

• Hypothesis as an affirmative statement
• Intention to treat
• Components of the power calculation
• Post hoc power analyses
• Powering for harm, no short cuts
• Justifications for subgroup analyses
• Small but significant
• Meaning of treatment effects
• Withdrawal studies to be cherished
• The mighty p

37
Summary

• In a RCT, like in all scientific endeavor, all
  efforts aimed at proving the hypotheses are
  wrong.
• The aim is falcification.

38
Vitamin-E in goutthe Conclusion

• Vitamin E is an innocuous agent and with the
  possible additional benefit of an
  antihypertensive effect, as was also
  serendipitously noted in this study, such studies
  are surely warranted. We are about to start a
  study along those lines at our institution.

39
What is wrong?

• Do not preempt!