"Learning shrinks the hard tail": how much inference-time compute helps depends on training
A solvable linear model showing that the returns from inference-time scaling are training-dependent — better-trained models have a thinner tail of hard instances, so sampling many candidates and picking the best buys less than it does for a weaker model.
The result inverts a comfortable assumption. Inference-time scaling has been read as a way to buy capability at serving time, implying you could economise on training and make it up later. If the returns shrink as training improves, that trade runs the wrong way.
The mechanism is intuitive once stated. Best-of-n sampling pays off in proportion to the spread between a typical attempt and the best available one. Training compresses that spread on instances the model has genuinely learned — so the hard tail where extra samples matter gets thinner as the model gets better.
Using a solvable linear model is a deliberate methodological choice and the reason to take it seriously. A closed-form result in a tractable setting establishes a mechanism rather than a correlation; whether the scaling constants transfer to a frontier transformer is the open question, and it is a narrower one than "does this effect exist".
For anyone allocating a budget, the reading is that training and inference compute are not fungible in the direction people hoped. Paired with a theoretical ceiling on token complexity, thinking longer is looking less like a scaling axis and more like a fallback for what training left behind.
arXiv — Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model → · arXiv — Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling →