Randomised comparison of an LLM diagnostic-dialogue system with primary care physicians (AMIE)
In a randomised, double-blind crossover study of text consultations with trained patient-actors across 159 case scenarios, Google's AMIE was more accurate in diagnosis than 20 primary care physicians and was rated better on 30 of 32 axes by specialist physicians and on 25 of 26 axes by the patient-actors.
- Lab
- Model
- AMIE (Articulate Medical Intelligence Explorer), fine-tuned from PaLM 2
- Field
- Medicine
- Date
- 2025-04-09
- Human collaborators
- Tao Tu, Mike Schaekermann, Alan Karthikesalingam, Vivek Natarajan
What was found
AMIE was trained for diagnostic conversation partly through self-play in a simulated environment with automated feedback, then evaluated in the format of an objective structured clinical examination. Validated patient-actors worked through scenarios from examination providers in Canada, the UK and India by text chat with either AMIE or a primary care physician, double-blind, and specialist physicians and the patient-actors rated history-taking, diagnostic accuracy, management, communication and empathy. The paper first appeared as an arXiv preprint in January 2024. Later work extended AMIE to multi-visit disease management (Nature, 2026) and tested it with real patients (separate entry).
Novelty check
Europe PMC, (large language model OR LLM OR chatbot OR conversational) AND diagnos* AND randomised, with physician, clinician or primary care in the title, to 8 April 2025: the randomised trials found give physicians an LLM on written cases (Goh et al., JAMA Network Open, 2024; Nature Medicine, February 2025) rather than letting the LLM conduct the consultation. No earlier randomised comparison of an LLM taking histories from patient-actors turned up.
Caveats and known objections
Patient-actors, not patients, and synchronous text chat, which the authors note is unfamiliar in clinical practice and so may not reflect how physicians normally consult. All authors are Google employees and the evaluation framework is the developers' own. Autonomy is graded autonomous because AMIE ran each consultation and produced its diagnosis and plan with no clinician involved, which holds only inside this simulated setting.
Nobody outside the lab has checked this yet.
Reading the primary source closely enough to say whether it supports the claim counts as a check, and you are credited on the entry.
Or on GitHub: submit a check challenge the grade send a correction or send a pull request
Flag this for triage
Signals order the review queue and nothing else. They are never published, and they never move a grade: that takes a citation.
Entry history (1 event)
- AddedEntered the registry graded Peer reviewed and Autonomous.
Entries are never deleted. A grade that does not hold up is downgraded on the record, with the reason beside it.
Graded peer reviewed for verification and autonomous for autonomy. What these mean.
Cite this entry
whataifound.org. (2025). Randomised comparison of an LLM diagnostic-dialogue system with primary care physicians (AMIE). whataifound.org: A Registry of AI Scientific and Mathematical Discoveries. https://whataifound.org/finding/2025-04-09-amie-diagnostic-dialogueBibTeX
@misc{whataifound-google-2025-dialogue,
title = {Randomised comparison of an LLM diagnostic-dialogue system with primary care physicians (AMIE)},
author = {{whataifound.org}},
year = {2025},
howpublished = {whataifound.org: A Registry of AI Scientific and Mathematical Discoveries},
note = {Result by Google. Verification: Peer reviewed. Autonomy: Autonomous.},
url = {https://whataifound.org/finding/2025-04-09-amie-diagnostic-dialogue}
}