PaperBench + InnovatorBench define the research-task evaluation frontier — what changes when agent benchmarks measure interpretation, not just implementation

SWE-Bench measures coding-against-specifications. GAIA measures general assistance. The new research-task benchmarks (PaperBench, InnovatorBench, AutoResearchBench) measure something different — interpretation of research papers, end-to-end research methodology, scientific literature discovery. The capability tier they evaluate is fundamentally harder than implementation.

PaperBench's research-replication evaluation and InnovatorBench's end-to-end research-task evaluation together establish a research-task evaluation tier above the H1 2026 'six benchmarks that matter' consolidation. The capability difference is meaningful — replicating an ML paper requires interpreting the contributions, deciding which experiments matter, implementing from incomplete descriptions, training, and reporting comparable results.

Why interpretation-vs-implementation matters

SWE-Bench tasks have specification-clear correctness criteria (test passes or fails). PaperBench tasks have interpretation-dependent correctness criteria — the paper may not specify every implementation detail; the agent has to make judgment calls. The capability surface tested is qualitatively different. Frontier models that excel at SWE-Bench may underperform at PaperBench because the interpretation-judgment requirement isn't a strength of agent-loop architectures optimized for spec-compliance.

The procurement implication for R&D workloads

Enterprise R&D teams considering AI-augmented research support should evaluate vendors against PaperBench-class benchmarks alongside the established production benchmarks. The H1 2026 'six benchmarks that matter' consolidation matches production-workload evaluation well; the research-task benchmarks add a higher-bar capability tier for novel-research workloads.

What stays uncertain

PaperBench and InnovatorBench are early-stage benchmarks — their methodology hasn't been stress-tested against the reward-hacking and benchmark-gaming patterns that the established six benchmarks have endured. The H2 2026 to 2027 evolution of these benchmarks (handling of adversarial agents, scoring-function-hardening, sample-set expansion) will determine whether they sustain as credible procurement-evaluation instruments or fade into research-novelty status.

arXiv — PaperBench: Evaluating AI's Ability to Replicate AI Research → · arXiv — InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research → · AI Agent Square — AI Agent Benchmarks 2026 →