The Robust Reasoning Benchmark rewrites AIME thirteen ways and watches the scores move
RRB applies a pipeline of thirteen deterministic textual perturbations to AIME 2024 and AIME 2025 — changes that preserve the mathematics while altering the surface form. The premise is that a model which genuinely reasons should be indifferent to them, and reported scores are not.
Every benchmark result carries an unstated assumption: that the score measures the capability and not the phrasing. RRB tests that assumption directly, and it is the rare evaluation paper whose method is more valuable than its leaderboard.
Deterministic is the key word. The thirteen perturbations are reproducible transformations, not paraphrases sampled from another model — so the resulting variance is attributable to the model under test rather than to the rewriter. Renaming variables, reordering independent clauses, changing surface presentation while preserving the mathematical content: none of it should matter to a system that has solved the problem.
It sits alongside a small cluster of work asking the same question from different angles — interactive benchmarks that score reasoning through budgeted multi-turn exchange rather than a single answer, and stability benchmarks measuring how much a model's reasoning wobbles across reruns. The common thread is dissatisfaction with single-shot accuracy on a fixed string.
The practical consequence for anyone reading a model card: a headline AIME number is a measurement of performance on a specific set of sentences. How much of it survives rephrasing is a separate number, and almost nobody reports it.
arXiv — Robust Reasoning Benchmark → · arXiv — Interactive Benchmarks → · arXiv — ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning →