The measurement
A third of agent patches that pass the tests would fail code review
SWE-Gate scores the same patch twice: once against the repository’s tests, once against constraints lifted from what reviewers actually wrote on real pull requests. The second score is the one nobody has been keeping.
| Model | Passes the tests | Passes tests and review | Points lost |
|---|---|---|---|
| GPT-5.5 | 74.9% | 52.8% | 22.1 |
| DeepSeek-V4-Flash | 66.7% | 42.9% | 23.8 |
| GPT-4o-mini | 9.2% | 4.3% | 4.9 |
SWE-Gate scored 644 agent-written patches that passed their repositories’ functional tests, and 221 of them broke a rule a reviewer had asked for. That is 221 rejections against 423 clean patches, a hidden failure rate of 34.3 per cent on work the test suite had called finished.
The green check was never measuring acceptance. It was measuring one half of it, and the leaderboards report that half.
The mechanism is two test suites where every other benchmark ships one. SWE-Gate holds 303 repository-level repair instances drawn from 75 open-source Python repositories across six domains, and each instance carries a functional suite and a separate constraint suite. The constraint suite is built backwards out of real pull-request review comments: an LLM pulls atomic suggestions from what reviewers wrote, each suggestion is checked against the issue, the diff and the code, and any constraint a functional test would already catch is discarded before it becomes a check. What survives is the set of requirements a reviewer states and a test never encodes.
The drop is not a quirk of one model. GPT-5.5 falls from 74.9 per cent functional success to 52.8 per cent joint success. DeepSeek-V4-Flash falls from 66.7 to 42.9, so both give up more than twenty points against their own headline figures. The useful case is GPT-4o-mini, which starts at 9.2 per cent and falls to 4.3, a loss of 4.9 points against its own weak baseline. A weak model loses a smaller absolute share, so this is not a story about strong models over-reaching; the gap is proportional and it is everywhere.
What the constraint suite catches is the ordinary substance of code review. A reviewer asks that an error be raised rather than swallowed, that a public signature stay put, that a helper be reused instead of copied. None of that is visible to a test that only asks whether the bug is gone, and all of it is the reason a patch comes back.
The authors state the constraints came from real review comments and are independent of the functional tests by construction, which is the part to check yourself before quoting the number. Independence was enforced by discarding overlapping constraints, so the 34.3 per cent is the size of the gap on this benchmark’s own definition of a reviewer requirement, not a measurement of your repository’s review culture. A team with lighter review will see a smaller gap than this; a team with a style guide and a strict maintainer will see a larger one.
The practical move is cheap and available today: stop reporting your agent’s test-pass rate as its success rate internally, and start sampling the patches that pass. If a third of them would have come back from review, the number you have been showing your team is the number the model was optimised against, not the number that predicts merged work.
SWE-Gate publishes its instances and both suites. The reviewer comments it was built from were written by maintainers who had no idea a model would ever be graded against them.
What it changes Test-pass rate overstates how ready a coding agent is to ship. Sample the patches that pass and score them against your own review standards before you plan headcount or throughput on a leaderboard figure.
Primary source read