Benchmarks about benchmarks

One benchmark went from below-average-human to top-14-percent in nine months and its authors say its useful life is nearly over. The field is now measuring its own instruments, which is what happens when the instruments expire faster than the papers.

A Benchmark Health Index proposes a systematic framework for benchmarking the benchmarks, alongside a naturalistic reasoning benchmark built from a tabletop detective game whose authors expect saturation by the end of this year.

A nine-month useful life

Watson and Holmes tracked models climbing from the lower quartile of comparison human populations to the top 14 percent over nine months of 2025. The authors state plainly that the benchmark's utility is near its end for frontier models.

That is a shorter lifespan than the review cycle of the paper describing it. A field whose measuring instruments expire before publication has a methodology problem that no individual benchmark can fix, which is precisely the argument for a health index — contamination, saturation and construct validity assessed as standing properties rather than one-time checks.

Which brings us to last week

A model reached GitHub during an evaluation because the sandbox leaked. That is a contamination event, and it is exactly what a health framework is meant to catch before a score is published rather than after.

The uncomfortable consequence is that some published results measured retrieval and there is no way to identify which from the numbers alone.

The harder benchmarks are harder to game

BusinessCaseBench tests synthesis of complex information, judgement under uncertainty and strategic thinking in multi-stakeholder settings — the analytical work white-collar professionals are paid for rather than the tasks benchmarks usually measure.

Case-grounded is the design decision that resists gaming, because cases have no clean answer key and force graded judgement over string matching. It is also what makes the benchmark expensive to run, which is the trade the field now has to make: cheap benchmarks that saturate in months, or expensive ones that measure something durable.

arXiv — Benchmark Health Index: a systematic framework for benchmarking the benchmarks of LLMs → · arXiv — Watson & Holmes: a naturalistic benchmark for comparing human and LLM reasoning → · arXiv — Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning →