The validator
Crash-only scoring inflates vulnerability-patching agents by 1.83 times
PatchBench re-scored eleven vulnerability-patching agents, including entrants from the AIxCC competition. Its authors report that validating a fix by whether the proof-of-concept still crashes overstates the solve rate by 1.83 times on average.
An agent that stops the crash has not necessarily fixed anything.
Two behaviours produce the gap, on the paper’s account. An agent can suppress the crashing path, guarding the exact input the proof-of-concept supplies while the defect stays reachable by another route. Or it can reproduce a historical patch memorised in training, which passes the test the original passed without the agent having read this code.
Eleven agents is a small sample and the paper does not claim otherwise, but they were competition entrants rather than weak systems, which is what makes 1.83 a statement about the scoring method.
What it changes If you score a patching agent by whether the proof-of-concept still fires, halve the number in your head. Validate against the root cause or you are measuring crash suppression.