Unlocking the Clinical Potential: Maximizing Value with Epic Sepsis Model Predictions

Researchers at the University of Michigan have demonstrated that the risk stratification accuracy of the Epic Sepsis model was significantly impacted by the limitation that its predictions were confined to data collected prior to fulfilling a patient’s sepsis criteria.In the latest research article featured this month in the esteemed pages of the New England Journal of Medicine (NEJM), the group of researchers hinted at the potential of advanced artificial intelligence (AI) technologies adept at forecasting the initiation of sepsis could improve treatment and outcomes. Tools like the Epic Sepsis Model (ESM) for identifying sepsis risk have been developed and deployed in healthcare systems nationwide, but questions have recently been raised about their accuracy and clinical utility.

The extensive use of Epic’s electronic health record (EHR) system, within which ESM is embedded, has led to numerous evaluations of the performance of the sepsis tool. A study published in JAMA Internal Medicine in 2021 found that the model correctly identified sepsis in only 63% of cases, which is significantly lower than the accuracy suggested by the tool’s information sheet. Another study published in JAMA Network Open last year found that Epic’s model outperformed tools like quick Sequential Organ Failure Assessment (qSOFA), Sequential Organ Failure Assessment (SOFA), and Systemic Inflammatory Response Syndrome (SIRS) in predicting sepsis-related organ failure, but lacked timely responsiveness necessary for effective diagnostic support.

Researchers emphasized the potential of physicians to identify sepsis before fully meeting patient criteria and initiating treatment, raising additional concerns about the potential clinical benefits of ESM and other models. To address this, the research team evaluated ESM predictions during hospital admission and treatment phases.

Data analysis was conducted on 77,000 adult patients admitted to Michigan University hospitals from 2018 to 2020, with approximately five percent of the group having sepsis. To assess the performance of ESM’s pre-admission predictions, researchers compared the likelihood that the tool assigned high-risk scores to patients diagnosed with sepsis against those who were never diagnosed with the condition.

The capability of the tool to indicate sepsis before fulfilling sepsis criteria and to provide indications for sepsis treatment plans was measured within the Receiver Operating Characteristic (AUROC) curve framework. ESM achieved an AUROC of 0.62, including predictions made before completing sepsis criteria, and incorporating predictions made after sepsis diagnosis in some cases resulted in an AUROC drop to 0.47.

The large number of patients identified with high-risk scores by ESM during all phases of hospitalization highlights the challenge of distinguishing patients at high risk. While ESM accurately flagged high-risk patients in 87% of cases, this accuracy dropped to 62% when predictions were limited before completing patients’ sepsis criteria. When further restricted to analyzing only data before orders for blood culture were given, the model’s predictive accuracy dropped to around 53%.

These findings prompted the research team to conclude that the tool could leverage information indicating the presence of sepsis to support clinical decision-making. “We suspect that some health data dependent on the Epic Sepsis model’s encoding may inadvertently cause physicians to doubt whether a patient has sepsis,” said study co-author Jenna Vience, PhD, Associate Professor of Computer Science and Engineering at the University of Michigan, in a press release.

If the tool is leveraging physicians’ uncertainty about the presence of sepsis, its predictions may not provide significant value in supporting clinical decision-making. “It’s important to consider when the model’s evaluation in clinical workflows is being done, when it’s determined whether it’s helpful for clinicians or not,” said Donna Tandra, PhD, doctoral student in computer science and engineering and co-author of the study. “Evaluating the model’s performance with data collected after physicians already suspect sepsis may show stronger performance, but this practice is not integrated with clinician assistance in practice.”

While the success of sepsis prediction models has been mixed, researchers are working on improving common shortcomings in tools like ESM. In January, a research team from the University of California (UC) San Diego School of Medicine shared that their AI model significantly reduced sepsis mortality rates following deployment in emergency departments. Known as a compositor, the tool utilizes intensive training to continuously monitor patients in emergency departments. It then utilizes 150 real-time feature vectors related to sepsis to predict the onset of the condition. Since implementation, this approach has led to a 17% reduction in mortality rates.