The steps a model calls important are not the steps that changed its answer

Judges scoring which chain-of-thought steps mattered reached a PR-AUC near 0.03, against a ceiling of roughly 0.51 to 0.64. That ceiling was set by splitting the study’s own estimates in half, which puts the judges about nine times below what the method itself could reach.

Actual importance was measured by counterfactual, not by reading. The authors resampled 50 rollouts from each step’s prefix and compared the probability of reaching the correct answer with the step and without it, across roughly 1,800 responses over 180 questions.

A fine-tuned critic reached only 10 to 20 per cent of that ceiling on the steps of correct answers, and did better on wrong ones. The ceiling here is resampling variance rather than human disagreement, so this is a limit of the measurement, not of annotators.