You can rank agents on a fraction of the tasks and get the same order
Across eight benchmarks and 33 agent scaffolds, rank-order prediction stayed stable on small task subsets. The proposed protocol needs no optimisation: evaluate new agents only on tasks with intermediate historical pass rates.
A study across eight benchmarks and 33 agent scaffolds asks whether small task subsets preserve agent rankings, and finds that rank-order prediction remains stable. The protocol it proposes is optimisation-free: evaluate a new agent only on tasks with intermediate historical pass rates.
The selection rule is the elegant part. Tasks that everything passes carry no information, and tasks that everything fails carry no information. What discriminates is the middle, and the middle can be identified from history rather than searched for — which is why no optimisation is needed and why the protocol survives new entrants.
Thirty-three scaffolds is a serious sample. Agent evaluation is unusually sensitive to scaffolding rather than to the model, and a result that holds across that many harnesses is measuring something about the benchmarks rather than about one implementation.
The limitation is stated in what the result claims. Rank order is preserved; absolute scores are not, and the interesting question for a buyer is often absolute — can this agent do my task, not is it better than that agent. A cheap ranking is a real contribution and it is not a cheap evaluation.
Read next to the $40,000 cost of a full leaderboard pass, this is the field trying to make its own claims checkable by someone other than the claimant.
arXiv — Efficient Benchmarking of AI Agents → · arXiv — From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level →