Epic Sepsis Model, version 1, disputed by external validation
Epic's proprietary sepsis prediction model, deployed at hundreds of US hospitals with a publicly reported area under the curve of 0.76 to 0.83, scored 0.63 in its first external validation, at Michigan Medicine, missing 67% of sepsis cases while alerting on 18% of all hospitalised patients.
- Lab
- Epic Systems
- Model
- Epic Sepsis Model v1 (proprietary penalized logistic regression)
- Field
- Medicine
- Date
- 2021-06-21
Sources
Media coverage
Challenge
What was found
Epic released the model in 2018 and it was implemented at hundreds of hospitals; it computes a score every 15 minutes from data already in the record. The Michigan study covered 27,697 patients and 38,455 hospitalisations from December 2018 to October 2019, 7% of them with sepsis. At the alerting threshold of 6, the model fired for 18% of all hospitalised patients and missed 1,709 of the 2,552 sepsis cases; it flagged only 183 sepsis patients (7%) whom clinicians had not already treated with timely antibiotics, the cases where an alert could add something. A 2023 study across nine hospitals of one network found discrimination as low as 0.55 at the largest hospital and no higher than 0.73, worse where sepsis, comorbidity and cancer were more common. Epic disputed the Michigan analysis, saying the investigators' approach cast a wide net that produced more false positives, and that it was continually improving implementations of the model. In 2022 Epic released a retrained version 2, recorded as a separate entry.
Novelty check
Not a discovery claim, so the question is what had been checked before. Europe PMC, "Epic Sepsis Model": no external validation precedes the Michigan study, whose authors say the model had not been adequately evaluated despite widespread use. Later validations at other systems are listed under independent checks.
Caveats and known objections
Epic's own validation figures were never published in a peer-reviewed venue, so the claim under dispute is known through vendor documentation as the study authors and the press reported it. Epic's objection is a real methodological point: a fixed-threshold evaluation measures the model as configured, not as it could be tuned. The later version 2 validation attributes part of the gap between internal and external figures to version 1's use of antibiotic orders as predictors and to internal validation against non-clinical outcome labels. Graded disputed rather than refuted because no common analysis has reconciled the developer's and the independent figures, and Epic replaced version 1 with a retrained model rather than settling the question; more than 100 organisations still used version 1 in 2025. Penalized logistic regression is a modest machine-learning model; it is in the registry because it is among the most widely deployed clinical prediction tools in the US and the clearest public record of the gap between vendor-reported and real-world performance.
Independent checks
Wong et al., Michigan Medicine (JAMA Internal Medicine, 2021): Retrospective cohort of 38,455 hospitalisations: hospitalisation-level area under the curve 0.63 (95% CI 0.62 to 0.64), poor calibration, 67% of sepsis cases missed at the default threshold and alerts on 18% of all patients. · link ↗
Lyons et al., Washington University School of Medicine and BJC HealthCare (JAMA Internal Medicine, 2023): 806,696 encounters at nine networked hospitals: discrimination varied from 0.55 at the largest hospital to at most 0.73, and fell with higher sepsis incidence, comorbidity burden and cancer prevalence. · link ↗
Disagree with these grades?
Bring a citation: a grade moves on evidence, not on argument.
Or on GitHub: submit a check challenge the grade send a correction or send a pull request
Flag this for triage
Signals order the review queue and nothing else. They are never published, and they never move a grade: that takes a citation.
Entry history (1 event)
- AddedEntered the registry graded Disputed and AI-assisted.
Entries are never deleted. A grade that does not hold up is downgraded on the record, with the reason beside it.
Graded disputed for verification and ai-assisted for autonomy. What these mean.
Cite this entry
whataifound.org. (2021). Epic Sepsis Model, version 1, disputed by external validation. whataifound.org: A Registry of AI Scientific and Mathematical Discoveries. https://whataifound.org/finding/2021-06-21-epic-sepsis-model-v1BibTeX
@misc{whataifound-epicsystems-2021-v1,
title = {Epic Sepsis Model, version 1, disputed by external validation},
author = {{whataifound.org}},
year = {2021},
howpublished = {whataifound.org: A Registry of AI Scientific and Mathematical Discoveries},
note = {Result by Epic Systems. Verification: Disputed. Autonomy: AI-assisted.},
url = {https://whataifound.org/finding/2021-06-21-epic-sepsis-model-v1}
}