Agents' Last Exam: a 2.6% average full pass rate on the hardest tier
A new benchmark built with more than 250 industry experts covers over 1,000 long-horizon, economically valuable tasks with verifiable outcomes across 13 industry clusters. Across mainstream harness and backbone configurations, the average full pass rate on the hardest tier is 2.6%. It is designed as a living benchmark whose task pool keeps growing.
A 2.6% pass rate is the useful kind of benchmark result, because saturated benchmarks stop carrying information. Most agent evaluations now cluster in ranges where the differences between systems are within noise. A benchmark where nearly everything fails has room to measure improvement for some time.
Verifiable outcomes and expert-authored tasks are the design choices doing the work. Long-horizon agent evaluation usually founders on grading — a human judge is expensive and inconsistent, and a model judge inherits the failure modes of the thing being judged. Tasks with checkable end states sidestep both.
The living-benchmark design is a direct response to contamination. A fixed task set leaks into training data and the score stops meaning anything within a year or two. Continuously onboarding new workflows keeps the measurement honest, at the cost of comparability across time — you cannot compare this year's score to last year's if the exam changed.
Hugging Face Papers — Agents' Last Exam → · arXiv — Evaluation and benchmarking of LLM agents: a survey → · arXiv — A survey on evaluation of LLM-based agents →