Sakana AI's Multi-Layered Review: a Claude-based peer-review system that finds planted errors, and where it falls short

Most AI paper reviewers are judged by how closely their reviews resemble human ones. A paper from Sakana AI researchers argues that this is the wrong test, and proposes a different one: can the reviewer find an error that has been deliberately planted in a paper? The paper, "Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review," is published in Transactions on Machine Learning Research (TMLR) and was posted to arXiv on October 8, 2026. It introduces a Contradiction Benchmark and a review system called Multi-Layered Review (MLR), built on Claude models.

MLR caught more than 70% of contradictions planted in papers' main claims, which the authors call "a fourfold improvement over the next strongest baseline." MarkTechPost, which covered the paper on October 10, reports the exact figures as 73.43% for MLR and 14.81% for the best prior system. On real withdrawn papers, the advantage was much smaller, and every system tested, MLR included, could be swayed by hidden manipulation text.

This article is based on the arXiv version of the paper (arXiv 2610.11087) and on MarkTechPost's report.

The short version

• Who: Rachel S.Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku and Tarin Clanuwat, all listed under Sakana AI (the first author also lists the National University of Singapore).
• System: three agents. An appendix agent on Claude Haiku 3.5, plus a review agent and an optional literature-review agent on Claude Sonnet 4.
• Benchmark: 1,164 contradictions inserted into 257 papers from ACL, AISTATS, CVPR, ICML 2025 and NeurIPS 2024.
• Main result: more than 70% of core-claim errors detected, about 4× the best open baseline (73.43% vs. 14.81%, per MarkTechPost).
• Real papers: on withdrawn arXiv papers, MLR matched the documented error 26.07% of the time (similar match), against 18.48% for the next-best system.
• Cost: about $0.47 per review at Claude Sonnet 4 API prices, using about 189,000 input tokens.

What problem is the paper addressing?

Machine learning conferences receive more submissions than reviewers can handle, and many authors and reviewers already use LLMs. Earlier AI reviewers, including Sakana's own AI Scientist reviewer, were largely judged by whether their scores and comments matched human reviewers.

The authors argue that this rewards imitation, not usefulness. A reviewer that sounds human but misses a broken argument doesn't help. So the paper measures error detection: plant a specific logical contradiction in a real paper, then check whether the review flags it.

How the Contradiction Benchmark works

1. Map the argument. Gemini 2.5 Pro builds a knowledge graph of each paper, with nodes for claims, evidence, methodology, implementation and mathematical theory.
2. Measure distance. Nodes labeled "main claim" have distance 0. Every other node's distance is its shortest path to a main claim. An error at distance 0 undermines the paper's central claim. Errors further out are more peripheral.
3. Plant the error. For each distance, one node is chosen and GPT-4.1 rewrites its paragraph to contradict the original. The LaTeX source is then recompiled into a new PDF.
4. Judge the review. OpenAI's o3 reads each review and decides whether it caught the planted contradiction. The judge runs 10 times and the results are averaged.

The benchmark covers 257 Creative Commons-licensed papers, about 51 per venue, with 1,164 contradictions in total.

The judge was checked in both directions. On unmodified papers, o3 correctly said "no contradiction found" 99.90% of the time. On 50 MLR catches that the authors verified by hand, it recognized 86.80%. The authors say this means reported detection rates may be conservative.

How Multi-Layered Review works

MLR tries to understand a paper before judging it:

• Appendix agent (Claude Haiku 3.5): digests the appendices so the main reviewer doesn't have to process them in full.
• Literature-review agent (Claude Sonnet 4, optional): uses web search to place the paper among prior work.
• Review agent (Claude Sonnet 4): runs a three-pass prompt chain "inspired by" Keshav's 2007 three-pass approach to reading research papers, which moves from a quick skim to a detailed, critical read.

The headline result combines four reviews per paper. A single review scored lower: MarkTechPost reports 60.79% at distance 0. The paper says a single MLR review still beats other methods by more than 20 points on average accuracy.

Part of the gain comes from the model, not the design. MarkTechPost reports an ablation in which swapping GPT-4.1 for Claude Sonnet 4 inside the LLM-Review baseline lifted distance-0 detection from 14.56% to 35.40%. MLR's structure accounts for the remaining gap.

Results

Planted contradictions

Exact values as reported by MarkTechPost on October 10, 2026, citing the paper. The paper's text confirms MLR "detects more than 70% of distance 0 contradictions" and calls this a fourfold improvement. Across all distances, MarkTechPost reports 40.95% for MLR.

Detection falls as the error moves further from the main claim. Both the paper's per-distance figure and MarkTechPost's summary show this. MLR is best at catching errors that break a paper's central argument and weaker on peripheral inconsistencies.

Real withdrawn papers

To test real errors, the authors used WithdrarXiv-Check, a set of withdrawn arXiv papers with documented errors, and checked whether reviews identified the error.

Only MLR and the strongest competing result in each column are shown. MLR still leads, but by single-digit points rather than a factor of four. Real errors are harder to catch than planted ones.

Agreement with human reviewers

On the imitation measure the paper argues against, MLR is competitive but not dominant. Its review scores correlated with human scores at 0.586 (Pearson) on ICLR 2025, 0.451 on NeurIPS and 0.429 on ICML, where Sakana's earlier AI Reviewer scored slightly higher (0.439). MarkTechPost reports that human-to-human agreement on ICLR was 0.742. The paper notes that MLR weighs validity and experiments more, while human reviewers weigh clarity and novelty more. In this part of the study, scores were inferred by GPT-4.1 from each review's text.

Manipulation

The authors also hid instructions in papers meant to inflate review scores, a form of prompt injection. All systems were strongly influenced, the paper says. MLR flagged the manipulative text in 8 of 50 cases. Our explainer on prompt injection covers why any agent that reads untrusted documents faces this risk.

Limits the authors acknowledge

• Synthetic errors can be unnatural. In a 50-item audit, 34% of inserted contradictions were rated as lacking coherence with neighboring sentences, and 8% as sounding AI-generated. A reviewer might partly be catching awkward prose.
• The benchmark can be gamed. A system that finds the original arXiv paper and compares it with the modified one would score well without reviewing anything.
• Narrow scope. Only machine learning papers, and only open review systems; closed commercial tools couldn't be compared.
• Older models. MLR uses Claude Sonnet 4 and Haiku 3.5, not Anthropic's current lineup (see which Claude model to use). Newer models may shift both MLR's and the baselines' results.
• No public code. The paper says data and code are "available upon request."

What builders should take from it

1. Evaluate reviewers on what you need them to catch. Measuring agreement with humans is easy. Measuring error detection takes a benchmark with known answers. The same logic applies to code-review agents: plant bugs, then check the catch rate. Our benchmark explainer covers related pitfalls.
2. Structure beats a single prompt, at a cost. MLR's multi-pass, multi-agent design added about 25 points over a single prompt on the same model, according to MarkTechPost's report of the ablation. It also uses about 189,000 input tokens per review. The paper says a consolidated single-prompt variant gets comparable performance at about a third of the cost.
3. Several reviews beat one. Combining four reviews lifted core-claim detection substantially. Sampling multiple independent reviews is a simple lever.
4. Don't treat AI review as a gatekeeper. Real-error detection was 26% at best, and hidden instructions moved every system's scores. AI review works as a second reader, not a final judge. That echoes Epoch AI's finding that agents can overstate their own research results: you need checks on the checker.

What's confirmed, reported and uncertain

• Confirmed (paper, arXiv 2610.11087): authors and affiliations; TMLR publication (the PDF header reads 07/2026; the arXiv posting is dated October 8, 2026); the three-agent design and Claude models; benchmark construction (257 papers, 1,164 contradictions); "more than 70%" distance-0 detection and the fourfold claim; WithdrarXiv-Check figures; human-correlation figures; 8-of-50 manipulation detection; ~$0.47 cost and 189,062 input tokens; judge accuracy; stated limitations; code on request.
• Reported (MarkTechPost, October 10, 2026, citing the paper's tables): the exact 73.43% / 14.81% / 60.79% / 40.95% figures, the 14.56% → 35.40% ablation, other baselines' distance-0 scores, and the 0.742 human-to-human correlation. These sit in appendix tables that weren't readable in the arXiv text used for this story.
• Uncertain: how MLR would perform with current Claude models, outside ML, or against reviewers designed to resist manipulation. Sakana AI hasn't published a blog post on the paper as of October 12, 2026.