FDA and Pharmaceutical Manufacturing Research Projects

1
FDA and Pharmaceutical Manufacturing Research
Projects

• Jeffrey T. Macher Jackson A. NickersonCo-Princi
  pal Investigators

2
Presentation Overview

• Executive summary
• Project goals
• Data collection and synthesis
• Analysis methodology
• Findings
• Development opportunities and constraints

3
Executive Summary

• We develop statistical models that predict the
• Probability of a facility being chosen for
  inspection.
• Effect of investigator training, experience, and
  individual effects on the probability of
  investigational outcomes.
• Characteristics and identities of facilities that
  correlate with the probability of non-compliance.
• We present initial results for each of these
  analyses.
• We identify additional opportunities and next
  steps to create value along with some
  constraints.

4
FDA Research Project History

• Research project idea emerged in Fall, 2001.
• Approached FDA in late Spring, 2002.
• Formalized relationship with FDA in Fall, 2003.
• Began receiving data September, 2004.

5
FDA Research Project Goals

• Risk-based assessment of FDA cGMP outcomes.
• Identify underlying ability of investigators and
  their training.
• Identify underlying compliance of each facility.
• Identify attributes (currently recorded by the
  FDA) that impact inspection outcomes.
• Transfer learning to FDA.

6
Progress to Date

• Just as new drugs go through
• Discovery
• Development and
• Commercialization.
• Our model and this presentation concludes the
  discovery phase of our project.
• Please think of our model as a platform that
  can be developed to assess a variety of
  compliance issues.

7
FDA Project Approach

• Compile and link FDA databases.
• Estimate the likelihood of various outcomes
• NAI, VAI, OAI Warning Letters Field Alerts
  Product Recalls.
• based on
• compound/product, facility, firm, FDA district,
  investigator and training derived factors.
• in order to
• evaluate the allocation of investigational
  resources.
• inform effectiveness of investigator training and
  management.

8
FDA Databases

• DQRS (Field alerts)
• EES
• FACTS (Inspections) CDER only
• Product Listing
• Product Recalls
• Product Shortages
• Facility Registration (DRLS)
• ORA Training database
• Warning letter database

9
Data Preparation

• Started with FACTS (1990-2003).
• Manufacturing facilities only.
• Assembled investigator training database
• Identified corporate ownership by plant by year
  and firms operating at a specific facility each
  year.
• Constructed facility-year data
• Added observations for years NOT inspected.
• Corrected FEI/CFN mismatches.
• Constructed numerous other variables.

10
Some basic facts about the FDA data

• Years covered FY 1990-2003
• Total number of facilities inspected 3753
• Total number of Pac codes 38,341
• Total number of Inspections 14,162
• Total number of investigators 783

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Empirical Methodology

• Inspection
• Probability of choosing a facility to inspect.
• Detection
• Probability of a non-compliance inspection
  outcome.
• Noncompliance
• Probability of noncompliance, inspection, and
  detection.
• Detection control estimation.

15
Inspection

• Groups of variables
• Technology variables
• Rx Prompt Release Ext or Delayed Rel
• Gel Cap Soft Gel Cap Ointment
• Liquid Powder Gas
• Parenteral Lg. Vol. Parent. Aerosol
• Bulk Sterile Suppositories
• Industry variables
• Vitamins (IC 54) Necessities (IC 55)
• Antibiotics (IC 56) Biologics (IC 57)
• Inspection decision variables
• Ln(Days between inspections)
• Surveillance reason for inspection (0
  Compliance)
• Last inspection outcome (1 OAI, 0 NAI, VAI)
• Years 1992-2003 (binary variables for each year)

16
Inspection Explained Variance

• Probit analysis of decision to inspect.

• D R2 Cumulative R2
• Technology variables 12 12
• Industry variables 9 21
• Inspection Decision variables 20 51
• Year dummy variables 0 51

Omitted categories Human Drugs (IC 60-66),
select technologies, Year dummies
1990-91. Foreign inspection included in analysis
but uniquely identifies many inspections and is
dropped from the analysis.
17
Technology VariablesChange in Probability of
Inspection
Gas -0.68
Parenteral -0.32
Lg Vol Parent. -0.08
Aerosol -0.26
Bulk -0.37
Sterile -0.07
Suppositories -0.23
Rx 0.13
Promp Rel. -0.19
Ext/del Rel. -0.19
Gel Cap -0.25
Soft Gel Cap -0.36
Ointment -0.32
Liquid -0.30
Powder -0.37
99 confidence interval 95 confidence
interval 90 confidence interval
Omitted categories Not Classified, Bacterial
antigens, Bacterial vaccines, Modified bacterial
vaccines, Blood serum, Immune serum.
18
Industry and Inspection VariablesChange in
Probability of Inspection
Industry Variables
Inspection Variables
Ln(Days btwn Insp) -0.28
Surveillance -0.84
Last outcome 0.13
Antibiotics (IC 56) 0.19
Vitamins (IC 54) 0.11
Necessities (IC 55) -0.06
Biologics (IC 57) -0.07
Omitted category Human drugs
99 confidence interval 95 confidence
interval 90 confidence interval
19
Days Between Inspections
20
Detection

• Groups of variables
• Technology
• Industry
• Training
• Total training days prior to inspection (other
  than 5 main drug courses)
• Drug course 1 Basic drug school
• Drug course 2 Advanced drug school
• Drug course 3 Pre-approval inspections
• Drug course 4 Active Pharmaceutical Ingrediant
  Mfg.
• Drug course 5 Industrial sterilization
• Investigator Experience
• Number of inspections in the prior 12 months
• Number of inspections in the prior 12-24 months
• ORA District Office
• Investigator Classification
• A consolidation of position classifications

21
Detection Explained Variance

• Probit analysis of decision to inspect.

• DR2 Cumulative R2
• Technology variables 0.9
  0.9
• Industry variables 0.3
  1.2
• Training and Experience vars. 0.3
  1.5
• Office and Position variables 1.4
  2.9
• Investigator effect 4.2 7.1

22
Training and Experience Variables Change in
Probability of Detection1
Total training days prior to inspection (less 1-5) -2.2E-03
Drug course 1 Basic drug school 0.07
Drug course 2 Advanced drug school -0.05
Drug course 3 Pre-approval inspections -0.23
Drug course 4 Activ. Ingred. Mfg. -0.15
Drug course 5 Industrial sterilization 0.08
No. of inspections in the prior 12 months 4.8E-03
No. of inspections in the prior 12-24 months -1.4E-03
1Without investigator fixed effects.
23
ORA Office and Classification Variables Change
in Probability of Detection2
ORA Office Variables
Position Variables
Compliance 0.04
Microbiologist -0.02
Investigator -0.04
Chemist -0.05
Eng/Sci -0.07
Dist/Reg. Admin. -0.10
FDA Bureau -0.15
Technician -0.18
ORA LOS 0.07
ORA KAN -0.06
ORA NYK -0.07
ORA SJN -0.09
ORA SRL -0.10
ORA ATL -0.10
ORA DAL -0.10
ORA SAN -0.11
ORA DET -0.13
ORA NWE -0.15
All other ORA off. insignificant. All other ORA off. insignificant. All other ORA off. insignificant.
2With investigator fixed effects.
24
425 Investigators
25
Non-compliance

• Detection Control Estimation
• Relatively new procedure used in academic
  literature.
• Used for assessing tax evasion, EPA compliance,
  and other applications.
• FDA application more complicated than other
  applications.
• Assume three actors
• Facility decides level of compliance.
• Inspection decision-maker chooses when to
  inspect.
• Investigator chooses detection or not.
• Estimate all three processes simultaneously.

26
Non-compliance model

• Assume inspection decisions are non-random.
• Assumption is different from other applications.
• Construct a likelihood function that models the
  probabilities of
• a plant being selected for inspection and
• the outcome of the inspection.

27
Constructing a Likelihood Function
The likelihood that facility i is inspected
The likelihood that facility i is not inspected
L2i 0
L1i 1
L2i 1
L3i 1
The likelihood that facility i is non-compliant
The likelihood that facility i is found
non-compliant
L3i 0
L1i 0
The likelihood that facility i is found compliant
The likelihood that facility i is compliant
28
Likelihood Function

• Three probabilities are combined to form the
  function
• Probability that a non-compliant facility is
  inspected and detected
• L1i1, L2i1, L3i1
• Probability of inspecting and not detecting
  noncompliance
• probability that the facility is compliant
• L1i0, L2i1
• probability that noncompliance goes undetected
• L1i1, L2i1, L3i0
• Probability that a facility is not inspected in a
  given year
• L2i0

29
Simple Likelihood Function

• LL log F(x1ib1) G(x2ib2) H(x3ib3)
• log G(x2ib2) F(-x1ib1)
  F(x1ib1) H(-x3ib3)
• log G(-x2ib2)

Where A facilities inspected and found
noncompliant B facilities inspected
and found compliant C facilities
not chosen for inspection
30
Estimating the Likelihood Function

• Select covariates associated with non-compliance,
  selection, and detection.
• Non-compliance facility-related
  characteristics.
• Selection factors currently used in selecting
  facilities.
• Detection investigator-related factors.
• Use a maximum likelihood estimation to find
  coefficient estimates that maximize the function.
• Initialize parameter estimates with results from
  inspection and detection analyses.

31
Change in Probability of Non-compliance
Rx -0.10 -0.09 -0.05 -0.04
Prompt rel. 0.07 0.08 -0.13 -0.13
Ext/Del rel. 0.17 0.21 0.13 0.14
Gel cap 0.20 0.19 0.05 0.06
Soft gel cap -7.E-05 0.02 -0.04 -0.04
Ointment 0.11 0.08 -0.18 -0.15
Liquid 0.21 0.22 -0.04 -0.03
Powder 4.E-03 -0.01 -0.26 -0.22
Gas -0.24 0.15 0.41 0.36
Parenteral 0.14 0.14 -0.04 -0.01
Lg. vol Parent. -0.24 -0.25 -0.26 -0.27
Aerosol 0.08 0.08 0.11 -0.07
Bulk -0.18 -0.15 -0.24 -0.27
Sterile 0.09 0.09 0.03 0.01
Suppositories 0.12 0.12 -0.26 -0.27
Number of obs. 81570 55371 22456 17499
32
Vitamins 0.07 0.17
Necessary 0.13 0.12
Antibiotics 0.23 0.22
Biologics -0.05 0.06
No. Thera. Classes/Plant No. Thera. Classes/Plant No. Thera. Classes/Plant 2.E-03 -3.E-03
No. Products/Plant No. Products/Plant No. Products/Plant -2.E-03 -1.E-03
No. Dose forms/Plant No. Dose forms/Plant No. Dose forms/Plant -4.E-03 -0.01
No. D.F. Routes/Plant No. D.F. Routes/Plant No. D.F. Routes/Plant -3.E-04 0.00
No. Sponsor Appl./Plant No. Sponsor Appl./Plant No. Sponsor Appl./Plant 0.02 0.02
Ownership change (t0) Ownership change (t0) Ownership change (t0) 0.16
Ownership change (t1) Ownership change (t1) Ownership change (t1) -0.13
Ownership change (t2) Ownership change (t2) Ownership change (t2) -0.09
Ownership change (t3) Ownership change (t3) Ownership change (t3) 0.34
Firms per plant Firms per plant Firms per plant -0.07
Inspection Technology Yes Yes Yes Yes
Plant Select No Yes Yes No
Detection Training Yes Yes Yes Yes
No. of obs 81570 55371 22456 17499
33
Facility-fixed Effects

• Construct binary variables for the facilities
  with the Greatest number of inspections.
• Re-estimate non-compliance model using binary
  variables for these 50 facilities.
• Identify those facility more or less likely than
  average to be non-compliant.

34
Predicted Level of Facility Non-compliance For 50
Most Inspected Facilities
1 34 26 47 8 36 21 32 18 23 2 44
3 5 19 16 9 29 20 15 7 27
28 41 45 25 35 4 33 38 13 42 14 43 30
37 10 39 50 49 46 22 17 31 12 40
Statistically more noncompliant than the mean
facility.
Statistically not different from the mean
facility.
Statistically more compliant than the mean
facility.
35
Immediate Implications

• Inspection and Non-compliance
• New suggestions for inspection choices.
• Use non-compliance analysis to assess risk of any
  given facility, firm, or technology.
• Increase focus on particular facilities and
  attributes.
• Ownership changes.
• Mixed strategy inspection plan.
• Detection
• Use detection analysis to assess quality of
  investigators and their training.
• Focus investigator activities to build and
  maintain short-run experience.

36
Broader Implications

• Our statistical methods provide a test-bed for
  asking and answering management and oversight
  questions.
• Further development is needed.
• DCE has potentially broad applicability to CDER
  and other centers at the FDA including CBER,
  food, etc..
• What facilities are most at risk of
  non-compliance?
• Base-line non-compliance
• Technology
• Ownership changes, etc.
• What manufacturers are more/less prone to
  non-compliance.
• DCE has implications for the type, format, and
  processing of data to be collected and analyzed.

37
Development Opportunities

• Additional variables can and are being
  constructed to examine additional issues.
• Recall, shortages, supplement filings.
• More fine-grain information on technology,
  manufacturing knowledge, organizational
  capabilities.
• Evaluate manufacturer data collected in our
  study.
• More heavily weight more recent investigations.
• Expand to full set of investigators and
  facilities (requires additional computational
  resources).
• Evaluate endogeneity concerns.

38
Development Constraints

• Software/computer limitation.
• Data preparation/man-power.
• Funding resources are nearly exhausted.
• Teaching.

39
Current Plan

• Document current progress in a white paper.
• Further develop data in hand (EES, Shortages,
  etc.).
• We received cooperation from the gold sheets.
• Work with you to develop plan for transferring
  results to FDA.
• Look for additional funding sources.