2.6%, and the honesty of a hard benchmark
A benchmark where almost everything fails is worth more than a benchmark where everything passes. The interesting question is what the 2.6% is a number about.
Saturation is the enemy
Most agent evaluations now cluster in ranges where differences between systems fall within noise. A benchmark everyone passes has stopped measuring anything; it has become a certificate. A benchmark where nearly everything fails has room to register improvement for years, which is the only property that makes a benchmark useful over time.
Verifiable outcomes and expert-authored tasks are the design choices carrying the weight. Long-horizon agent evaluation usually founders on grading — human judges are expensive and inconsistent, model judges inherit the failure modes of the thing being judged. Tasks with checkable end states avoid both.
The living-benchmark trade
Continuously onboarding new workflows answers contamination: a fixed task set leaks into training data and stops meaning anything within a year or two. The cost is comparability — you cannot compare this year's score against last year's if the exam changed. That is a real trade and the design chooses correctly, but it should be stated rather than glossed.
What the number is actually about
This is where the user-simulation work becomes load-bearing. Most agent benchmarks either remove the human entirely or model them badly. Real deployments involve users who clarify, interrupt, change their minds, and supply information only when asked.
An agent optimised against a silent oracle is optimised against a counterfactual world. So a pass rate measured without a realistic user is measuring *autonomous completion* — a harder task than most real deployments actually pose. The 2.6% is honest. What it is a number about needs stating carefully, and usually is not.
The better framing
Configurable human participation, because participation is not binary. The interesting axis is how much the human supplies and when — and an agent that performs well at one level may be useless at another. A single score cannot express that, and most of the field is still reporting single scores.
Which is the same lesson arriving from a third direction this week: aggregate numbers hide the structure that determines whether the thing actually works.
Hugging Face Papers — Agents' Last Exam → · arXiv — VISTA: a versatile interactive user simulation toolkit for agent evaluation → · arXiv — HAS-Bench: evaluating LLM-based human-agent systems under configurable human participation →