A fully machine-generated paper passed workshop peer review
One of three end-to-end AI-generated manuscripts submitted to an ICLR 2025 workshop averaged 6.33 from reviewers and would have been accepted; the authors withdrew it, and the bar cleared was a workshop track, not the main conference.
- Lab
- Sakana AI
- Model
- The AI Scientist-v2
- Field
- Computer science
- Date
- 2025-03-12
What was found
The AI Scientist-v2 generates a hypothesis, writes and runs the experiments through agentic tree search, and produces the manuscript, figures and related work without human editing. Sakana submitted three such papers to the ICLR 2025 workshop "I Can't Believe It's Not Better", with the knowledge and cooperation of the workshop organisers and ICLR leadership, and agreed in advance to withdraw anything accepted. One paper, on whether an explicit compositional regularisation term improves compositional generalisation, scored 6.33 and cleared the bar.
Novelty check
Machine-written text had passed review before in the form of nonsense-generator stings (SCIgen, 2005), which tested review rather than produced research. The claim here is different: an autonomous pipeline produced a real experimental paper that reviewers rated acceptable. Sakana's own write-up is the source for the process, and the code and submitted manuscripts are public.
Caveats and known objections
Workshop tracks accept a far higher share of submissions than the main conference, and Sakana says so itself; the company also notes the accepted paper contains a citation error and that its own reviewers judged the work below the main-conference bar. The paper's scientific content is a negative result: the proposed regularisation did not help. Reviewers were not told which submissions were machine-generated, but the experiment ran with organiser consent, so this is not a blind test of peer review at scale. Humans chose the venue and selected which generated papers to submit, which caps autonomy at ai-led.
Nobody outside the lab has checked this yet.
Reading the primary source closely enough to say whether it supports the claim counts as a check, and you are credited on the entry.
Or on GitHub: submit a check challenge the grade send a correction or send a pull request
Flag this for triage
Signals order the review queue and nothing else. They are never published, and they never move a grade: that takes a citation.
Entry history (1 event)
- AddedEntered the registry graded Author verified and AI-led.
Entries are never deleted. A grade that does not hold up is downgraded on the record, with the reason beside it.
Graded author verified for verification and ai-led for autonomy. What these mean.
Cite this entry
whataifound.org. (2025). A fully machine-generated paper passed workshop peer review. whataifound.org: A Registry of AI Scientific and Mathematical Discoveries. https://whataifound.org/finding/2025-03-12-ai-scientist-workshop-paperBibTeX
@misc{whataifound-sakanaai-2025-paper,
title = {A fully machine-generated paper passed workshop peer review},
author = {{whataifound.org}},
year = {2025},
howpublished = {whataifound.org: A Registry of AI Scientific and Mathematical Discoveries},
note = {Result by Sakana AI. Verification: Author verified. Autonomy: AI-led.},
url = {https://whataifound.org/finding/2025-03-12-ai-scientist-workshop-paper}
}