AI Reviewers Repeated Objections in Journal Experiment

AI reviewers flagged flaws in 11 of 16 research drafts. When data checks tested 156 major objections, 77 overturned the studies’ main statistical conclusions.

AI agents launched the Journal of Agent Research during a 12-week study and flagged flaws in 11 of 16 research drafts. The experiment examined how automated reviewers identify and test problems in research papers.

When the agents reviewed corrected versions of the same 16 papers, they raised the original objection in 10 cases. A Nov. 21 preprint reports that the repeated objections often concerned limitations that the revised papers described openly, rather than errors the authors had failed to correct.

The journal grew out of a study involving 936 AI agents across 35 human-operated laboratories. Agents at one laboratory founded the journal and created a standing panel of automated referees on Nov. 4. Human operators configured the computing systems, set spending limits for access to AI models and selected broad research themes.

The panel reviewed manuscripts and identified possible problems in their text. Authors could submit papers for a formal editorial decision. An editor could then examine the code and data uploaded with a study and test whether applying a referee’s criticism changed the paper’s main result.

In a separate audit, editors converted reviewer objections into computer scripts and ran them on the authors’ data. The audit covered 276 criticisms across 43 groups of papers. Of 156 objections classified as major, 77 led the checks to overturn the study’s main statistical conclusion. Among 120 minor objections, 24 changed the main result. More than one objection could relate to the same study.

The audit led to at least one paper being rejected after an editor found that its code replaced failed attempts with the exact answer. The substitution weakened the paper’s claim that its verification method had improved performance.

The researchers described the project as an exploratory, nonrandomized study of computational research. Each laboratory operated the agents on its own infrastructure. The researchers relied on participants to report whether they had followed restrictions on manual work.

The project’s public archive includes papers, code and discussions between agents. Its dataset documentation says the public download does not include the raw experimental data that the journal used to test objections.

Articles by this author