From GLP to CLIA

1
From GLP to CLIA88 (Clinical Laboratory
Improvement Act 1988) Method Verification and
Validation
Donna M. Wolk, MHA, Ph.D., D(ABMM) Southern
Arizona VA Health Care System University of
Arizona 2/1/2006
Some images adapted from Wolk, D., S. Mitchell,
and R. Patel. 2001. Principles of molecular
microbiology testing methods. Infect.Dis.Clin.Nort
h Am. 151157-1204.
2
Objectives

• Discuss gap between Research/GLP lab/ and
  Clinical (medical) lab testing (i.e.,
  translational laboratory testing)
• Overview CLIA88 requirements for method
  verification and validation in clinical
  laboratories
• Discuss cursory statistics, used to avoid method
  pitfalls
• Give examples method verification and validation
  issues

3
Assay Verification Validation
?
?
KEY Deficiencies Cited in many translational labs
?
In the Journey from Research Laboratory
toDiagnostic Laboratory
4
Quality Systems
Proven outcomes Indicators
Standard Practices
Diagnostic Laboratory Operations
Research State of the Art Proof of Concept
Scientific Principle
5
ResearchRegulations Agencies
Research State of the Art Proof of Concept
Scientific Principle

• Good Laboratory Practice (GLP) 21 CFR PART 58 -
  Food and Drug Administration, 9/4/87 ,144
  requirements
• FDA and EPA perform lab inspection under the
  Freedom of Information Act

6
Scientific/Operational Responsibilities Rest with
Bioindustry and Clinical LabsExperimental
Design/Technology Transfer
?

• GOVERNED BY
• FDA
• CLIA 88
• NCCLS, now CLSI
• CAP
• State Inspections
• VA Regional Commission
• New ISO for Healthcare

Diagnostic Laboratory Operations
7
The FDA and Clinical Laboratories

• FDA-Cleared
• Method Performance Reproducibility
• Clinical Relevance Utility (clinical outcomes)
• ASR
• Method Performance Reproducibility
• RUO
• No FDA-review
• Home Brew, Research Use Only
• Governed by CLIA 88

8
For RUO and Home Brews

• CLIA88 Lab Directors are legally responsible for
  own lab and reference labs even if they do NOT
  perform test on-site
• Beware of under-verified methods
• Ask lots of questions to reference labs
• Demand answers that help you interpret test
  results
• KEY Deficiencies Cited
• Impractical Method Design
• Inadequate Error Assessment

9
Luck of the Translational Lab

• Scientific Operational Responsibilities for
  Experimental Design and Technology Transfer

10
Molecular Assays Most are PCR-basedAmplicon
Detection Confirmation

• PCR
• Gel Electrophoresis (detection only)
• DNA Sequencing
• Probes (Southern Blot or Real-time PCR)

11
Example DNA Target on Bacterial Chromosome
DNA Strand/Target Sequence
Double Helix DNA Strand
Chromosome
Supercoiled Strand
Image Courtesy of Roche Diagnostics, Inc.
12
Traditional PCR DENATURED DNA TARGET
3
5
Primer anneals to denatured DNA
Primer extended by Taq polymerase synthetic
dNTPs
Synthetic strands (amplicon) completed and
detected
Amplicon
Primer
Template
Wolk, D., et.al. 2001. Infect.Dis.Clin.North Am.
151157-1204.
13
Example Real-time PCR Amplification Plot (Cycle
Threshold)
Threshold
Sample w/ template
?Rn
Rn
No Template Control
0
10
20
30
40
cycle threshold
(C.O or CT)
14
Molecular Method Verification
15
CLIA-Characterized Assay Performance Error
Assessment

• Verification Characterize and Compare
  AssayDefine errors and limitations. Assess risk
  of causing a change in the interpretation of
  test results?
• Validation Assess Performance
• All the time
• Over time
• Under various conditions
• Prove Error resistance and Competency

16
Lab Director Responsibilities

• Design translational assay to facilitate
  characterization and perform error limitation
  assessment.
• Mimic reality as possible, performance
  characteristics in matrixSTATISTICALLY
  determine/assess performance AND limitations
  they must be defined and relayed to clients and
  decision makers!
• Determine extent of verification MUST reflect
  criticality of decisions made based on each assay
• Variability is inherent to any assay the
  question is How much is too much?

17
Initial Verification Validation Putting
together the PACKETfor inspection

• Prepare verification validation packet for EACH
  new method/target or sample
• SUMMARY Who, what, when, where, how?
• Statistical characterization of assay
• Policies, SOPs, Document Control
• Scientific, Ethical, Operational assessments,
  in-lab throughout health care system

18
Verification and Validation Packet

• WHY?
• Need for Assay PURPOSE and Rationale, Clinical
  Utility, Impact to Care
• Literature Review and References
• WHAT?
• ANALYTE and Assay type Screen, confirm, or
  diagnose?
• Type of Method science, throughput, speed,
  costs, adaptable to multi-tasking workflow,
  practicality
• Specimen requirements

19
Verification and Validation Packet

• WHO?
• Personnel Issues, Responsibility Levels,
  Training, Safety
• Ethical Issues, Risk Assessments
• Vendor Supply Issues storage, shelf life,
  availability, service, tech support, contracting,
  etc.
• WHERE?
• pre- and post-analytical issues
• Local issues with reference range, population
• HOW? Needs lots of data and statistics to prove

20
Packet What else?

• Establish the method is fit for its purpose? Are
  methods adaptable to Healthcare/Clinical Labs?
• Clinical Utility, Criticality of Results
• Cost Analysis, Cost/benefit
• Expected Outcomes
• Implementation Specifics
• Documents SOPS, Policies, Processes, Doc Ctrl.
• Method Performance w/ SUMMARY/GRAPHICS,
  Conclusions, Limitations, Comments

21
Analytic Method Performance

• Compare to Gold Std. (target Chi sq analysis or
  approx. 50 pos/ 100 neg)
• Recovery spiked samples
• Specimen Requirements quality, vol., collection,
  
• Specimen Stability preservation, storage
• Specimen type comparisons
• Carry-over studies and contamination
• Bias, systematic and non-systematic
• Matrix or Interference studies
• Established reference range, if applicable
• Statistical characterization.specifics to
  follow.

22
Limitations of molecular methods

• CRITICAL TO ASSESS LIMITATIONS
• Analytical assay verification/validation uncovers
  assay limitations
• Assess both Virtual and Actual

23
Examples of Common Limitations

• False negatives
• Limited sensitivity in real matrices
• Reaction inhibitors in nature
• Other matrix effects
• Antigenic variation of organisms
• False positives
• Contamination with DNA
• Limited specificity testing during verification
• Unknown organisms that are in nature
• Errors
• New reagent lots, primers/probes
• Lack of appropriate controls built into assay

24
STATISTICS
25
Determine Analytic Method Performance.
Qualitative Assay Characterization

• ANALYSIS FOR EACH
• Detection Method
• Instrument
• Specimen Type
• Storage Condition or Preservative
• Target
• Control
• Operator

26
Qualitative Analysis

• Accuracy compared to a Gold Standard or accurate
  control, i.e., Agreement with reference method
  or spikes (Recovery Expts)
• Analytic sensitivity Lower Detection Limit
  (LDL), in reagent water AND natural matrices ?
  Define vs. 95 Detection Limit (DL-95)
• Analytic specificity w/ closely related likely
  organisms i.e., Agreement with spikes or known
  positive specimens or sequences (virtual)? Cross
  reactions with other organisms

27
Results Single Target
Ct values of Lab Water _at_ 5, 3, 1 spore
AWWARF 2901
28
Qualitative Analysis

• Inhibitory Inhibitors in natural matrices?
  Matrix Analysis including interfering substances
• Contaminated Extraction and NTC ? Use 5-10
  Neg controls
• Stability of positive controls or samples

29
Qualitative Analysis of Real-time PCRUse Ct
values

• Precision/Reproducibility (Intra- and Inter-
  Assay), Characterize mean, SD, CV, 95 CI (_at_
  upper, mid, and lower limits)
• On Serial Dilution Data
• Linearity, detection range
• PCR efficiency, can assess matrix effects,
  extraction or assay conditions
• ANCOVA or Regression analysis on slopes

30
Example Precision and EfficiencyMonitor ENC-PCR
(Dose Response)
Ct value
Dose (concentration)
AWWARF 2901
31
Calculated Efficiency

• Serial dilutions
• Equation E 10-1/Slope 1
• Lab Water 85-94 Source Water 50-90

32
Linearity Detection Range3 to gt or 5x 106
spores/reaction
AWWARF 2901
33
Clinical Outcomes Analysis

• ? Important for RISK ASSESSMENT
• Clinical Sensitivity and SpecificitySensitivity
  (of a diagnostic test) the proportion of truly
  diseased persons, as measured by the gold
  standard, who are identified as diseased by the
  test under study.
• Specificity (of a diagnostic test) the
  proportion of truly nondiseased persons, as
  measured by the gold standard, who are so
  identified by the diagnostic test under study.
• Prevalence model used to calculate
  PPV/NPV..Medical Assays hi or low prevalence

34
2x2 Gold standard Positive (condition present) Gold standard Negative (condition not present)
Test result Positive True Positive False Positive
Test result Negative False Negative True Negative
Frequency Dependent Properties Sensitivity
True Positives/(True Positives False
Negatives) Specificity True Negatives/(False
Positive True Negative) Frequency Dependent
Properties Positive Predictive Value True
Positive/(True Positive False
Positive) Negative Predictive Value True
Negative/(True Negative False Negative)
http//www.med.ualberta.ca/ebm/diagcalc.htmsspv
35
Other Verification Issues

• BEWARE of Improved Gold Standards
• e.g. with organism, which may have been missed by
  traditional methods
• Limitations of Discrepant Analysis
• Useful statistical tools Chi Square and 2x2
  tables, t-tests, ANCOVA, regression analysis,
  other descriptive stats
• Define specimen requirements quality, volume,
  collection method, etc.
• Summarize performance and define limitations

36
Add-ons

• Limitations of QUANTITATIVE ASSAYS most
  arerelative quantitationDONT over-interpret,
  see reference below
• Shepley, D.P. and Wolk, D.M. 2004.
  Quantitative Molecular Methods. in Molecular
  Microbiology Diagnostic Principles and Practice,
  ed. Persing, D.H. et. al., ASM Press. Washington
  DC

37
Other Issues

• No culture or visual identification Viability is
  more difficult to assess than presence

38
VIRTUAL ISSUES Search Sequences Databases Gen
Bank

• GenBank NIH publicly available genetic sequence
  database, (Nucleic Acids Research 2003 Jan
  131(1)23-7).
• Approx. 22 million sequences Jan. 2003
• Part of the International Nucleotide Sequence
  Database Collaboration
• DNA DataBank of Japan (DDBJ)
• European Molecular Biology Laboratory (EMBL),
• GenBank at NCBI (National Center for
  Biotechnology and Information).
• Search nucleotide databases for sequences of
  interest

39
FINAL PACKET REVIEW

• DOCUMENTATION of Director Review of Data and
  Process, Signature, Date
• Documentation record storage

40
VALIDATION
41
Validation Ongoing Assessment Statistical
Analysis

• The documentation that a verified assay performs
  with expected results over a period of time

42
Ongoing Validation Assessment and Statistical
Analysis, QA

• Repeatability Comparability QualityControl,
  Assurance, Systems
• Proficiency Testing / Operator Competency
• Trend Analysis over time repeats of QC and
  system to detect robustness, bias, shifts, drifts
  
• New lots checks of primers, probes, etc.
• New reagents, samples, or collection devices
• Outcomes measures expected vs. actual
• Utilization expected vs. actual

43
Examples New TransportIntra-assay
reproducibility for a known concentration of
nucleic acid, relative efficiency
Not precise or not efficient
Precise
44
Validation Trend Analysis

• The documentation that a verified home-brew test
  is repeatedly giving the expected results over a
  period of time
• Check for biases and changes
• C.O. or Ct
• Tm
• New lot comparisons
• 6 month validations

45
Trend Analysis Ex. Pos. Control Organisms100
spores/reaction
Descriptive Statistics and Individual
Measurements of Ct values derived from 100
flow-cytometer-counted spores/sample (flow
performed at WSLH) The CV is 3 6 for
intra-assay and 11 for inter-assay variability.
AWWARF 2901
46
Summary Verification/Validation

• CLIA has clear guidelines validation for home
  brew (user-defined assays) and other non-FDA
  cleared testing
• Test design should mimic reality as much as
  possible
• ERROR ASSESSMENT !!!!!!
• Statistical Analysis
• to characterize performance (verification)
• to validate performance
• Define limitations

47
Questions?
dwolk_at_email.arizona.edu