Lessons Learned in Clinical Decision Support: Over-alerting

1
Lessons Learned in Clinical Decision Support
Over-alerting

• Tejal K. Gandhi, MD MPH
• Director of Patient Safety
• Brigham and Womens Hospital
• Assistant Professor in Medicine
• Harvard Medical School

2
Overriding of Alerts

• Studies have shown that MDs override clinical
  decision support alerts a large percent of the
  time
• 88 of inpatient DDI alerts overridden (Payne et
  al. Proc AMIA 2002)
• 83 of inpatient drug-allergy alerts (Abookire et
  al. Proc AMIA 2000)
• 89 of outpatient high severity DDI alerts and
  91 of outpatient drug-allergy alerts (Weingart
  et al. Arch Intern Med 2003)

3
Are Overrides Appropriate?

• 7,761 drug-allergy alerts in BWH inpatients,
  Aug-Oct 2002
• Alerts were overridden 80 of the time
• In chart review of 300 overrides, all were
  clinically justified
• Evidence that we are over-alerting
• Only 6 of alerts were triggered by an exact
  match between drug ordered and drug in allergy
  list (lots of potential noise)

Hsieh et al. JAMIA 2004
4
Alerts Triggered by Exact Drug-Allergy Matches
are Rare

• Ordered drug
• Hydromorphone (DILAUDID) (N18)

• Drug in allergy list
• Codeine (39)
• Oxycodone (22)
• Meperidine (22)
• Morphine (17)
• Hydromorphone (0)

5
Alerts Triggered by Exact Drug-Allergy Matches
are Rare

• Ordered drug
• Furosemide (LASIX) (N18)
• Hydrochlorothiazide (N4)

• Drug in allergy list
• Sulfa (95)
• Furosemide (5)
• Sulfa (100)
• Hydrochlorothiazide (0)

6
Allergy Alerting Recommendations

• Need better specificity of alerts to avoid false
  positives
• E.g. requiring exact matches for certain classes
• Good news FDB recently removed lasix/HCTZ
  sulfa interaction

7
Overall Alerting Issues

• Need more studies to maximize effectiveness of
  alerts/ minimize over-alerting
• Issue of how best to display the messages
• Need to learn from other industries (industrial
  engineering)

8
Drug-Pregnancy Level 1
9
(No Transcript)
10
Potential Strategies to Improve Alerting

• Creation of streamlined knowledge bases
• Only essential content
• Balance between sensitivity and specificity
• Tiering of alerts is also a possibility
• Hard stop
• Interruptive
• Non-interruptive
• Minimizing interruptions

11
Impact of Reduced Alerting on Override Rates

• Study in the ambulatory setting
• Decision support included
• Duplicate drug
• Drug-disease
• Drug-drug
• Drug-lab
• Drug-pregnancy

Shah et al. JAMIA 2006
12
Knowledge base streamlining

• Expert panel
• Physicians, pharmacists, informaticians
• Reviewed sources
• Vendor knowledge-bases, pre-existing locally
  created KBs, literature
• Removed certain alerts and tiered the rest

13
Alert tiers

• Level 1 Potentially life-threatening
• E.g., erythromycin - diltiazem -gt V-fib
• Hard stop couldnt proceed
• Level 2 Potential for serious injury
• Rizatriptan - linezolid -gt serotonin syndrome
• Interruptive, required a reason
• Level 3 Use w/ caution
• Warfarin levofloxacin -gt increased PT
• Noninterruptive

14
(No Transcript)
15

16
Actions w/ level 2 alerts

• Cancel
• Do not proceed with order
• Modify, examples include
• Alter the problem list
• Discontinue the pre-existing drug
• Hold the other medication
• Accept defined as cancel or modify
• Override defined as not accept

17
Results

• Final knowledge base
• 2 level 1 63 level 2, 35 level 3
• 18,115 alerts
• 12,933 non-interruptive (71)
• 5,182 interruptive (29)
• Of 5,182 interruptive alerts
• 3475 (67) accepted

18
Interruptive alerts
Alert N Accepted Overridden
Duplicate class 3,875 2,695 (77) 910 (23)
Drug-drug 1078 451 (42) 627 (58)
Drug-disease 19 10 (53) 9 (47)
Drug-lab 92 37 (40) 55 (60)
Drug-pregnancy 118 12 (10) 106 (90)
Total 5182 3,475 (67) 1,707 (33)
19
Summary of Reduced Alerting Study

• Can reduce alert burden by streamlining and
  tiering the knowledge base
• Concept of a non-interruptive alert may be
  helpful
• Still need more research on what is optimal level
  of alerting
• Are we missing things is always the worry

20
Non-interruptive alerts
Alert N ( of all alerts in that category)
Duplicate drug 0 (0)
Drug-drug 3547 (77)
Drug-disease 24 (56)
Drug-lab 4,444 (98)
Drug-pregnancy 4,918 (98)
Total 12,933 (71)
21
(No Transcript)
22
Impact of Non-Interruptive Alerts

• Drilled down into the drug-lab alerts
• No difference in lab ordering between control and
  intervention groups
• Is it a user interface issue or are
  non-interruptive alerts just not effective?
• Cost- benefit issue
• Minimize interruptions so higher risk alerts are
  accepted more often yet little benefit seen with
  non-interruptive alerts

23
Impact of Tiering on Inpatient DDI Alerts

• Two academic medical centers
• Same knowledge base
• Site A used 3 tiers
• Site B had all of the alerts as interruptive
  (Level 2)
• Overall alert acceptance higher at tiered site
  (29 vs 10, plt.001)

Paterno, et al. Unpublished data.
24
Tiered Inpatient DDI Acceptance Rates

• Level 1 Acceptance rates
• 100 (hard stop) vs 34 (not a hard stop)
• Level 2 Acceptance rates
• 29 vs 11
• Likely higher at tiered site since less alert
  fatigue because fewer interruptive alerts

25
Conclusions

• Streamlined knowledge bases and tiered alerting
  have higher acceptance rates
• Especially for very high risk alerts
• Non-interruptive alerts may have little value
• What is our ideal acceptance rate??
  Sensitivity/specificity? Best way to display?
• More work needs to be done to maximize the
  clinical benefits
• Sharing of streamlined knowledge should be
  widespread
• No need to reinvent the wheel