OTC-TFM Monograph: Statistical Issues of Study Design and Analyses

1
OTC-TFM Monograph Statistical Issues of Study
Design and Analyses
Thamban Valappil, Ph.D.Mathematical
StatisticianOPSS/OB/DBIII
Nonprescription Drugs AC Meeting March 23, 2005
2
Outline

• Introduction
• Summary of Statistical Issues
• Current TFM trial Design and Analyses with
  surrogate endpoints
• Statistical Issues of Study Design and Analyses
• Options for Trial Design and Efficacy Criteria
  using surrogate endpoints

3
Introduction

• Previous presentations on issues involved in
  validating surrogate endpoints.
• In the absence of clinical trials data, FDA still
  needs to address current products under review.
• This talk discusses issues related to analysis of
  data obtained on surrogate endpoints.
• Does not address clinical relevance of
  statistical findings or differences in analysis
  of data based on surrogate endpoints.

4
Summary of Statistical Issues

• Primary Endpoint of Log reduction in Bacterial
  Counts from baseline.
• Data Analyses Variability Issues
• Binary Outcomes
• Log Reduction (Mean vs. Median) outcomes
• Variability in methodology
• Study Design and Controls
• Active
• Vehicle

5
Current TFM Recommendations
Indication Controls Endpoints Statistical tests Sample size /arm
Surgical HandScrub Active, Vehicle or Placebo Log Reduction, Binomial t-tests 66
PreOperative Skin Prep Active Control Log Reduction, Binomial Paired t-test 96
Healthcare Personnel Handwash Active control Log Reduction t-tests 54
6
Current TFM Recommendations Issues

• TFM Recommends
• Randomized
• Blinded (to persons determining counts only)
• Use of Active Control
• Use of Vehicle or Placebo Control (role not
  clearly specified)
• However, in the current TFM
• A non-comparative study design is used in which
  the test product is not directly compared to the
  Active Control.
• Mean log reduction meeting the threshold log
  reduction has been used to demonstrate efficacy.

7
Current TFM Recommendations Issues

• Although vehicle and placebo controls are
  mentioned in the TFM , majority of the NDAs only
  have test product and active control arms.
• Active controls have only been used for internal
  validation of study methods.
• Efficacy assessment does not include a direct
  comparison of Test product performance to Active
  control, vehicle or placebo.

8
Statistical Issues of Study Design and Analyses

• Primary Endpoint Log Reduction
• Mean Log Reduction can be influenced by few
  extreme observations.
• ( Suggestion Median log reduction may be
  another option. Median Log Reduction is less
  sensitive to few subjects with extreme log
  reductions or outliers. )

9
Statistical Issues of Study Design and Analyses

• The efficacy criteria in the current TFM are
  based on point estimates and do not include
  confidence intervals to evaluate variability.
• Consequently, a few extreme observations can
  potentially drive the efficacy results.

10
Example
Log Reduction Mean 2.0, Median 1.7 Below
2-log 78
11
Example Log Reduction in Bacterial Counts
Active Control
Threshold
Test
.
Vehicle
12
Upper Limit Mean Lower Limit
Threshold
Active Control
Test
.
Vehicle
13
Threshold
Test
Active Control
.
14
Primary Endpoint Using Binary Response

• Subject will be classified as a success or a
  failure based on meeting the threshold
  reduction.
• Advantages
• Outcome centered on number of subjects and not
  on organisms may be more clinically relevant.
• Effect of Variability is reduced.
• Disadvantage
• This method does not differentiate the magnitude
  of log reductions among those who meet the
  criteria for success.

15
Example
16
Study 1 (Surgical Hand Scrub)

• Treatment Day Mean
  Median Success ()
• Test 1 (1 log) 2.63 2.95
  20/21 (95)
• 2 (2 log) 3.25
  3.17 21/21 (100)
• 5 (3 log) 3.51
  3.88 13/20 (65)
• Active 1 1 (1 log) 1.51
  1.69 9/12 (75)
• 2 (2 log) 2.37
  2.36 8/12 (67)
• 5 (3 log) 3.47
  3.59 9/12 (75)
• Active 2 1 (1 log) 1.17
  1.40 14/22 (64)
• 2 (2 log) 2.02
  1.80 10/22 (45)
• 5 (3 log) 1.66
  1.58 2/21 (10)

17
Sample Size Issues

• In current TFM, sample size is estimated based on
  allowing a test product to be as much as 20
  worse than active control in the mean log
  reductions. However, the basis for 20 margin is
  not clearly stated.
• Majority of the current submissions do not follow
  the recommended sample size as specified in the
  TFM and only use a sample size of 30 subjects
  per treatment arm.

18
Options for Trial Design and Efficacy criteria

• Issue 1
• How to analyze the data obtained on the surrogate
  endpoint of log reductions in bacteria?
• Issue 2
• How to take into account the variability in the
  data collected, when measuring effect of the
  product?
• Issue 3
• How to take into account the variability in the
  test methodology?

19
Options for Trial Design and Efficacy criteria

• Issue 1 How to analyze the data obtained on the
  surrogate endpoint of log reductions in bacteria?
• Mean log reduction
• It can be influenced by few extreme observations.
• Median log reduction
• Less sensitive to outliers or extreme
  observations.
• Percentage of subjects who meet log reduction
  criteria
• Outcome centered on number of subjects who meet
  the threshold and may provide incentive to study
  conditions of use that provides highest success
  rates.

20
Options for Trial Design and Efficacy criteria

• Issue 2 How to take into account the variability
  in the data collected?
• Examine confidence intervals around the
  outcomes as defined on previous slide with a
  threshold for lower bound of confidence interval
• PRO Improvement over examination of point
  estimates alone.
• CON Does not take into account the variability
  in test method.
• Examine confidence intervals around treatment
  difference between test product and some control
• PRO Allows for examination of variability in
  methodology across treatment arms.
• CON May require larger sample size for products
  with lower success rates.

21
Options for Trial Design and Efficacy criteria

• Issue 3 How to take into account variability in
  the test methodology?
• Equivalence/non-inferiority showing test
  product is no worse than active control by some
  margin
• PRO Allows comparison with an active treatment
  to rule out loss of effect relative to active
  control
• CON Lack of constancy of effect of active
  control in previous studies, possible overlap of
  effect of active and test product with vehicle,
  hence no basis to select a non-inferiority margin
• Superiority of test product to vehicle AND
  superiority of active control to vehicle
• PRO Given lack of constancy of effect with both
  active and vehicle controls, allows internal
  validity of comparisons
• CON May require larger sample size than current
  TFM standards (how much larger depends on product
  efficacy over vehicle)

22
Controlling Variability in Test Methodology
?
Test Product
Active Control
Vehicle
S
S
S Superiority
23
Sample Size Superiority Test
24
Thank you!