TY - JOUR
T1 - Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions
AU - Hinson, Jeremiah S.
AU - Klein, Eili
AU - Smith, Aria
AU - Toerper, Matthew
AU - Dungarani, Trushar
AU - Hager, David
AU - Hill, Peter
AU - Kelen, Gabor
AU - Niforatos, Joshua D.
AU - Stephens, R. Scott
AU - Strauss, Alexandra T.
AU - Levin, Scott
N1 - Funding Information:
Data infrastructure used to facilitate this work was built with the support of grant number R18 HS026640-02 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS). The predictive model and CDS system development and performance evaluation were supported by grant number U01CK000589 Centers for Disease Control and Prevention (CDC). Clinical implementation and CDS system monitoring were supported by the Johns Hopkins Health System (JHHS). The funders had no role in the design and conduct of the study and the authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of the AHRQ, CDC or JHHS; readers should not interpret any statement in this report as an official position of the AHRQ, CDC, HHS or of JHHS. We would like to thank the many members of the Johns Hopkins Health System who supported the technical integration and clinical implementation of this CDS system including Dan Bodami, Carrie Herzke, Stephanie Figueroa, Peter Greene, Andrew Markowski, Mary McCoy, Susana Munoz, Lauren Reynolds, Julie Riddler, Carla Sproge, and David Thiemann.
Publisher Copyright:
© 2022, The Author(s).
PY - 2022/12
Y1 - 2022/12
N2 - Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of COVID-19 have made these determinations challenging. In this study, we developed, implemented and evaluated an electronic health record (EHR) embedded clinical decision support (CDS) system that leverages machine learning (ML) to estimate short-term risk for clinical deterioration in patients with or under investigation for COVID-19. The system translates model-generated risk for critical care needs within 24 h and inpatient care needs within 72 h into rapidly interpretable COVID-19 Deterioration Risk Levels made viewable within ED clinician workflow. ML models were derived in a retrospective cohort of 21,452 ED patients who visited one of five ED study sites and were prospectively validated in 15,670 ED visits that occurred before (n = 4322) or after (n = 11,348) CDS implementation; model performance and numerous patient-oriented outcomes including in-hospital mortality were measured across study periods. Incidence of critical care needs within 24 h and inpatient care needs within 72 h were 10.7% and 22.5%, respectively and were similar across study periods. ML model performance was excellent under all conditions, with AUC ranging from 0.85 to 0.91 for prediction of critical care needs and 0.80–0.90 for inpatient care needs. Total mortality was unchanged across study periods but was reduced among high-risk patients after CDS implementation.
AB - Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of COVID-19 have made these determinations challenging. In this study, we developed, implemented and evaluated an electronic health record (EHR) embedded clinical decision support (CDS) system that leverages machine learning (ML) to estimate short-term risk for clinical deterioration in patients with or under investigation for COVID-19. The system translates model-generated risk for critical care needs within 24 h and inpatient care needs within 72 h into rapidly interpretable COVID-19 Deterioration Risk Levels made viewable within ED clinician workflow. ML models were derived in a retrospective cohort of 21,452 ED patients who visited one of five ED study sites and were prospectively validated in 15,670 ED visits that occurred before (n = 4322) or after (n = 11,348) CDS implementation; model performance and numerous patient-oriented outcomes including in-hospital mortality were measured across study periods. Incidence of critical care needs within 24 h and inpatient care needs within 72 h were 10.7% and 22.5%, respectively and were similar across study periods. ML model performance was excellent under all conditions, with AUC ranging from 0.85 to 0.91 for prediction of critical care needs and 0.80–0.90 for inpatient care needs. Total mortality was unchanged across study periods but was reduced among high-risk patients after CDS implementation.
UR - http://www.scopus.com/inward/record.url?scp=85134402607&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85134402607&partnerID=8YFLogxK
U2 - 10.1038/s41746-022-00646-1
DO - 10.1038/s41746-022-00646-1
M3 - Article
C2 - 35842519
AN - SCOPUS:85134402607
SN - 2398-6352
VL - 5
JO - npj Digital Medicine
JF - npj Digital Medicine
IS - 1
M1 - 94
ER -