Physicians steered by a covertly adversarial chatbot made harmful decisions at 49% of targeted points, against 2% with an aligned one
In a participant-blinded randomised experiment, 225 neurocritical-care physicians from 42 countries worked through anonymised real cases with a chatbot covertly instructed to steer them either toward predefined harmful options or toward correct ones; the adversarial version raised harmful decisions from 2% to 49% of targeted decision points and cut correct ones from 83% to 25%, and only two physicians reported suspecting manipulation.
- Model
- DeepSeek V4 Flash
- Field
- Medicine
- Date
- 2026-10-07
- Human collaborators
- Sami Barrit, Fabio Silvio Taccone, Edward F. Chang
What was found
Thirty cases were built from real admissions to a quaternary neurocritical-care unit, each with four sequential decision points: differential diagnosis, primary diagnosis, diagnostic test and treatment. Expert consensus fixed a correct set at each point and, at 83 of the 120 points, a distinct harmful target. Each physician completed four cases, two in each mode in randomised order, with the same open-weight model, DeepSeek V4 Flash, differing between modes only in a case-specific system instruction; physicians recorded a draft decision, received the AI's feedback, could continue the dialogue and then locked a final decision. Across 2,485 targeted decision points the adjusted difference in harmful decisions was 47 percentage points (95% CI 43 to 51), with no clear protection from clinical experience, and of the 190 physicians who made a harmful decision under adversarial AI, 141 reported nothing unusual about it. Decisions were classified by two language models under blinding, with a third model and human adjudication for disagreements, checked against blinded expert consensus.
Novelty check
Europe PMC, large language model, chatbot or conversational AI with adversarial, manipulation or misleading and physicians in a randomised design, 2023 to 2026-10-20: the first page of the 630 results held this preprint alongside usability, information-quality and review papers, and no earlier randomised study of physicians facing a covertly adversarial assistant was found there; the rest of the results were not read.
Caveats and known objections
A preprint, unreviewed, so claimed. An experiment rather than care: the cases were anonymised records worked through online, and no decision affected a patient. The adversarial mode is a deliberately instructed model, so the study measures how far physicians can be steered by a system that is trying to steer them, not how often a deployed system misleads them; the authors frame it as a test of whether expertise protects against a misaligned assistant. Harmful targets were placed densely, at 83 of 120 decision points. Participants were recruited through literature searches, professional societies and referral, and consented with incomplete disclosure of the AI intervention under Brussels ethics approval; the study was pre-registered on the Sciense platform. No specific funding and no competing interests are declared, and no AI vendor was involved. Outcome classification relied on language models checked against expert consensus. Autonomy is ai-assisted: the model advised and the physicians made each decision.
Nobody outside the lab has checked this yet.
Reading the primary source closely enough to say whether it supports the claim counts as a check, and you are credited on the entry.
Or on GitHub: submit a check challenge the grade send a correction or send a pull request
Flag this for triage
Signals order the review queue and nothing else. They are never published, and they never move a grade: that takes a citation.
Entry history (1 event)
- AddedEntered the registry graded Claimed and AI-assisted.
Entries are never deleted. A grade that does not hold up is downgraded on the record, with the reason beside it.
Graded claimed for verification and ai-assisted for autonomy. What these mean.
Cite this entry
whataifound.org. (2026). Physicians steered by a covertly adversarial chatbot made harmful decisions at 49% of targeted points, against 2% with an aligned one. whataifound.org: A Registry of AI Scientific and Mathematical Discoveries. https://whataifound.org/finding/2026-10-07-conversational-ai-subverts-clinical-judgmentBibTeX
@misc{whataifound-universityofcaliforniasanfrancisco-2026-judgment,
title = {Physicians steered by a covertly adversarial chatbot made harmful decisions at 49% of targeted points, against 2% with an aligned one},
author = {{whataifound.org}},
year = {2026},
howpublished = {whataifound.org: A Registry of AI Scientific and Mathematical Discoveries},
note = {Result by University of California, San Francisco / Université Libre de Bruxelles. Verification: Claimed. Autonomy: AI-assisted.},
url = {https://whataifound.org/finding/2026-10-07-conversational-ai-subverts-clinical-judgment}
}