Stopping early
Stopping agent evaluations early cut a third of the tokens and moved scores by two points
EarlyEval cut up to 44.1% of input tokens on SWE-bench Verified, TerminalBench and Toolathlon by stopping agent runs before they finished. Its authors report that resolve rates moved by one to two points against the full runs.
Most of an evaluation’s cost is spent on runs whose outcome was already decided.
The method trains LightGBM classifiers on a run’s intermediate behaviour and predicts the final outcome partway through, then halts the ones it can call. The authors put prediction accuracy at 89% to 97%, which is what buys the saving. They report output tokens down as much as 29.4% against the full runs.
A one-to-two point shift is wider than the gap between many models on these leaderboards, so the saving is real and the number it produces is not the same number.
What it changes Use it for iteration, where a third off the bill is worth two points of noise. Do not use it for the run you publish.