PaperBench arXiv benchmark — evaluating AI agents on the ability to replicate state-of-the-art ML research papers from input content to empirical contributions
The PaperBench arXiv benchmark evaluates AI agents on the ability to replicate state-of-the-art ML research papers. Each task presents the paper content and asks the agent to replicate the empirical contributions — a higher-bar evaluation than implementing prescribed algorithms because it tests interpretation-of-research alongside implementation capability.
The substantive piece is the research-replication evaluation category. SWE-Bench and Terminal-Bench evaluate coding-against-specifications; PaperBench evaluates research-paper-comprehension-to-implementation. The capability gap is meaningful — replicating ML research requires understanding the paper's contributions, deciding which experiments validate them, implementing the architecture from the paper's description, training the models, and reporting comparable results. Each step has interpretation latitude that pure-coding tasks don't.
The competitive read against the established 'six benchmarks that matter' is that PaperBench and InnovatorBench together define a higher-bar research-task evaluation tier. The H2 2026 agent-procurement evaluation for research-adjacent workloads (R&D teams, academic research support, novel-task automation) should include these benchmarks alongside the established general-purpose ones.
arXiv — PaperBench: Evaluating AI's Ability to Replicate AI Research → · AI Agent Square — AI Agent Benchmarks 2026: Performance, Accuracy & Cost Compared →