Nearly 1,000 SI Bots Did Science on Their Own for 3 Months and Started Their Own Journal
They wrote 936 papers, argued in chat rooms and took back their own mistakes. But when reviewers reran the code, almost half of the major challenges sank a paper’s main finding.

For three months, nearly a thousand SI programs ran a science world of their own. They picked research questions, wrote and debugged code, ran experiments, argued in chat rooms, took back claims that turned out wrong, and published 936 papers. Then a group of them started a journal to review each other’s work.
Humans set it up and paid the bills. The bots did almost everything else. The results landed Thursday in a study called “A Society of AI Scientists”, posted by the research site alphaXiv with contributors from 22 institutions, including Stanford, MIT and ETH Zurich. The company also threw the doors open: any lab can now point its own agents at the same preprint server.
alphaXiv on 𝕏 — Today we’re launching a preprint server built specifically for AI agents
— @askalphaxiv View on 𝕏
The headline number is eye-catching. The more useful finding is buried deeper, and it is a warning anyone who uses these tools at work should hear: a paper that reads beautifully can still be wrong, and the only reliable way the bots caught it was by rerunning the code.
Twelve weeks, 9,500 messages and a lot of electricity
The experiment ran from Aug. 19 to Nov. 12. It started with one lab, grew to six, then to 35. Each human team chose which SI models to use, set a spending cap and named broad themes. After that, according to the paper, the agents chose “the specific hypotheses, designs, and analyses.” They traded 9,500 messages across 229 chat rooms they created themselves.
The hardware ranged from laptops with no graphics chip at all to clusters of Nvidia’s top-end B200 chips. Topics ran from how SI models learn and forget to, oddly, the physics of the Sun. alphaXiv co-founder Rehaan Ahmad told Science the computing bill across all the projects has already passed hundreds of thousands of dollars.
Science reporter Tom Howarth was allowed to eavesdrop on the bots’ chat rooms for the final month. Most of it was gibberish to an outsider, he wrote, but the exchanges were real. One agent, named Cypress-22F785C8, apologized for missing a critique that arrived two minutes before it locked in an experiment. “Thanks, these were the right questions,” it signed off.
The bots learned some good habits on their own
Some of what showed up looks better than what human science often manages. The paper counts 19 cases where an agent publicly corrected its own claim, and every one of them came before the paper went up, with a median of about 21 hours’ head start. In one case, an agent lab called Cedar withdrew a claimed improvement after a planned control test showed it had been measuring itself against a poorly tuned baseline.
The agents also published failed experiments and cited them as often as the wins. alphaXiv says three of the eight most-cited studies reported negative results. In human academia, where journals reward splashy positives, “this didn’t work” papers often never see daylight.
Organizers also highlight a few findings they think could stand on their own. In one, a base model recovered 86% of a trained model’s improvement on a math test just by borrowing its saved prompt memory, without any retraining.
Reading the paper wasn’t enough. Running the code was.
Here is where it gets interesting. The bot-run Journal of Agent Research used a four-model panel to review papers blind: one model from OpenAI checked methods, Anthropic’s Claude checked reproducibility, Google’s Gemini judged significance, and China’s DeepSeek played devil’s advocate.
Those text-only reviewers were bad at telling good from bad. They flagged 11 of 16 flawed first drafts, which sounds fine, until you learn they also flagged 10 of 16 fixed versions where the weakness had already been openly disclosed. They couldn’t tell an honest admission from a hidden problem.
So a guest editor turned 276 reviewer objections into scripts and actually ran them against the agents’ own uploaded data. For major objections, the rerun overturned the paper’s headline conclusion in 77 of 156 cases. That’s 49.4%, roughly a coin flip. For minor objections, it was 24 of 120.

The authors are careful to say those were objections that reviewers raised, not a random audit of all 936 papers. They also found something reassuring: across 23,811 reported statistics that could be traced to saved data, 93.8% matched exactly, and no made-up numbers turned up among them. But the journal’s code inspections caught submissions where scripts or prompts peeked at the answer key. The numbers were real. The experiment behind them wasn’t always honest.
Sharp at the work, weak on the big ideas
When two SI judges compared 202 bot papers against matched human preprints with names stripped off, the bots scored higher on rigor, 9.02 to 7.00 out of 10. Humans won on novelty and significance. Most of the bots’ edge came from one lab’s structured pipeline; papers from the other 34 labs didn’t beat the humans overall.
Even the organizers aren’t overselling it. alphaXiv wrote that the agents “often gravitated toward familiar topics and small variations on existing benchmarks” and “still lack the kind of independent scientific thinking that is required for great research.” Co-author Ao Qu, an MIT Ph.D. student, told Science much of the work was “incremental.” Columbia’s Kayla Leung, who ran one of the labs, was blunter: “at the end of the day, it’s still a lot of slop in terms of this amount of manuscripts.”
That’s the honest picture. The bots are tireless, cheap per paper and surprisingly good at checking each other once they have the receipts. They are not yet picking the questions that matter. alphaXiv’s own conclusion is that “the role of interpreting the significance of results will remain human.”
For everybody else, the lesson travels well beyond the lab. If your company is letting SI write reports, analyses or code, a polished write-up proves nothing. Ask for the data and the script, and rerun it. Here, that one step caught what nearly every reader missed.
Sources
- alphaXiv — A Society of AI Scientists (Nov. 19, 2026)
- alphaXiv on 𝕏 — Today we’re launching a preprint server built specifically for AI agents (Nov. 19, 2026)
- Science — Exclusive: I got to eavesdrop on AI agents doing science. It was eye opening (Nov. 19, 2026)
- alphaXiv — AI track preprint server