PCI Risk Model Comparisons

1
PCI Risk Model Comparisons

• An alternative model for case level estimation of
  pre-procedure PCI Mortality Risk
• Michael Blechner, M.D.
• Michael Matheny, M.D.

2
Goal

• Explore alternative models for pre-intervention
  risk assessment in patients being considered for
  a percutaneous coronary intervention (PCI)

3
PCI Background

• A myocardial infarction is typically due to a
  chronic narrowing in one or more of the blood
  vessels supplying the heart combined with an
  acute obstruction at that site
• Treatment options
• surgical bypass of the region or
• PCI in which a catheter is fed through the vessel
  and the temporary inflation of a small balloon
  widens the vessel lumen
• Both techniques can also be performed on patients
  with evidence of chronic narrowing but who have
  not yet had an MI

4
Pre-intervention Risk Assessment

• Risk of death in PCI varies widely based on
  co-morbidities
• Providing case level estimations can greatly aid
  patient and physician decision-making
• Estimates by physician experts are inaccurate at
  the high and low ends of the probability spectrum

5
History of PCI Risk Assessment

• PCI is a high volume procedure with significant
  morbidity mortality
• Early attempts to develop statistical models of
  risk were limited by non-standardized data
• The American College of Cardiologists (ACC) has
  since mandated that accredited centers maintain
  detailed data on all PCI patients
• Track outcomes with respect to predictor variables

6
(No Transcript)
7
Current Risk Model StandardLogistical Regression
(LR)

• Type of generalized non-linear model
• Used in analysis of a binary outcome
• Bounded by 0 and 1
• Produces Coefficients/Odds Ratios and an
  intercept
• Variable selection
• From All Available Data
• Known Risk Factors from Prior Studies
• Selected Subset of data based on Study Design

8
SummaryLogistic Regression

• Advantages
• Straightforward
• Intuitive results in the form of odds ratios
• Disadvantages
• Presumes independence between variables
• Difficulty in applying model to different
  geographies and time periods
• Missing data points assumed to be negative
  findings

9
(No Transcript)
10
Alternative Risk ModelBayesian Network (BN)

• Advantages
• Can incorporate variable co-dependencies
• Provides a method for the estimation of unknown
  variables, i.e., reasoning under uncertainty
• Easy to retrain
• Provides a graphical representation of variable
  relationships
• Disadvantages
• Accuracy of network is dependent on nodal
  connections

11
Bayesian Network Methodology

• Directed acyclic graph (DAG) consisting of
• Nodes
• Directed links between nodes
• Conditional probability tables (CPT)
• Assumptions of conditional dependence and
  independence based on expert opinion or machine
  learning algorithms
• Prior and conditional probabilities are developed
  using existing data or expert opinion

12
Study Hypothesis

• A BN will provide a better case level estimation
  of risk than a model developed using standard LR
  techniques

13
(No Transcript)
14
Source Data

• Brigham Womens Hospital
• Interventional Cardiology Database
• January 1, 2002 October 30, 2004
• 5383 Cases
• 2/3 Training Cases (3588)
• 1/3 Test Cases (1795) beginning October 27, 2003

15
Sample DemographicsOverview

Age
0-49 590 10.96
50-59 1167 21.68
60-69 1497 27.81
70-79 1398 25.98
80 652 12.22
Diabetic 1721 31.98
Hypertensive 4083 75.86
Hyperlipidemia 3737 69.44
Prior PCI 1822 33.85
Salvage Procedure 24 0.45
Cardiogenic Shock 98 1.82
Hemodynamic Instability 265 4.92
Death 78 1.45
16
Variable Selection
Age Hyperlipidemia Hx COPD
Gender HTN Hx CVD
Race Diabetes Hx PVD
Cardiogenic Shock Creatinine Thrombolytic
Cardiac arrest Hx CHF BMI
Hemodynamic instability CHF EF
Procedure urgency Prior MI AMI
IABP Prior PCI
Smoker Prior CABG
17
Logistic Regression ModelDevelopment

• Backwards Stepwise Technique
• Exclusion Threshold P gt 0.10
• Inclusion Threshold P gt 0.05
• Variables Evaluated 35
• Continuous Variables Discretized
• STATA 8.2 (College Station, Texas)

18
(No Transcript)
19
Methods BNNaïve BN
Netica Release 2.17 (Norsys Software Corp.,
Vancouver, BC, Canada)
20
Methods BNNaïve Hidden BN
21
Methods BNNon-Naïve BN
22
Discrimination

• A models ability to distinguish between patients
  who die and those who survive
• Although a Model calculates an outcome
  probability, the classification of a case into
  death vs. survival is based on an arbitrary
  threshold
• This threshold determines the sensitivity and
  specificity of the prediction
• ROC curves graph the sensitivity vs.
  1-specificity at different thresholds
• The discriminatory performance of the model is
  estimated by the area under the ROC curve

23
Calibration

• Measures how close the models estimates are to
  the true probability
• The true probability is the probability of
  death for a similar patient population
• Provides an estimation of case level accuracy
• Accuracy of the statement The risk of death from
  PCI in patients like you is 1 in 1,000.
• Hosmer-Lemeshows Goodness-of-Fit Test
• Ranks population by probability estimate
• Divides population into equal subsegments
• Calculates how well the observed and expected
  frequencies match

24
ResultsBN
25
(No Transcript)
26
Results Logistic Regression Model
OR 95 CI P
Age 60-69 21.79 1.88-252.83 0.014
Age 70-79 57.61 4.76-697.21 0.001
Age gt 80 161.36 13.28-1960.76 lt0.001
Prior PCI 0.29 0.09-0.93 0.037
Cardiogenic Shock 25.03 9.50-65.97 lt0.001
Cardiac Arrest 7.12 1.68-30.13 0.008
Hemodynamic Instability 3.69 1.49-9.17 0.005
Salvage Procedure 201.56 13.54-3001.07 lt0.001
Diabetic 2.43 1.15-5.10 0.019
Hyperlipidemia 0.18 0.08-0.41 lt0.001
27
All Model Test ROCSummary
Training Set Training Set Training Set
Model ROC 95 CI
Logistic Regression 0.94 0.91-0.98
Naïve Bayesian Network 0.93 0.88-0.97
Naïve Hidden Bayesian Network 0.91 0.86-0.96
Non-Naïve Bayesian Network 0.97 0.95-0.98
Test Set Test Set Test Set
Model ROC 95 CI
Logistic Regression 0.86 0.80-0.93
Naïve Bayesian Network 0.89 0.82-0.97
Naïve Hidden Bayesian Network 0.89 0.82-0.96
Non-Naïve Bayesian Network 0.85 0.76-0.93
28
All ModelsTraining ROC Comparison
29
All ModelsTest ROC Comparison
30
All ModelsPair-wise ROC Evaluation
Diff P
Logistic Regression vs Naïve Bayes -0.031 0.373
Logistic Regression vs Naïve Hidden Bayes -0.026 0.277
Logistic Regression vs Non-Naïve Bayes 0.014 0.686
Naïve Bayes vs Naïve Hidden Bayes 0.005 0.877
Naïve Bayes vs Non-Naïve Bayes 0.045 0.277
Naïve Hidden Bayes vs Non-Naïve Bayes 0.040 0.178
31
All Model HL Good-FitSummary
Training Set Training Set Training Set
Model HL Chi2 Prob gt chi2
Logistic Regression 9.48 0.219
Naïve Bayesian Network 11.94 0.154
Naïve Hidden Bayesian Network 20.40 0.009
Non-Naïve Bayesian Network 24.97 0.002
Test Set Test Set Test Set
Model HL Chi2 Prob gt chi2
Logistic Regression 18.16 0.011
Naïve Bayesian Network 4.91 0.768
Naïve Hidden Bayesian Network 16.3 0.038
Non-Naïve Bayesian Network 18.85 0.016
32
Calibration Plot
33
(No Transcript)
34
Discussion

• Discrimination
• All Models had excellent performance
• None of the models was significantly different in
  performance
• Calibration
• Two models achieved calibration on the training
  set Logistic Regression Naïve Bayes
• The only model to retain calibration on the test
  set was the Naïve Bayes Model

35
Limitations

• The CPTs for hidden nodes within our BNs were
  built using a machine learning algorithm
• Data reporting and database quality would be
  expected to improve over time
• Ambiguity between absent and negative values
  for some database fields
• Attempts to develop more realistic Bayesian
  Networks were limited by software failures

36
Causal BN
37
Conclusions

• Calibration is essential for any test where case
  level accuracy is important
• The only model that retained calibration with the
  test set was the naïve BN
• This study supports the use of a Naïve Bayesian
  Network for case level estimation in
  pre-procedural PCI risk assessment as an
  alternative to logistic regression

38
The end