Science Independent signal for people building with AI In the edition of Thursday, 10 September 2026
The Early Edition

The change, the consequence, the evidence. Before the churn.

The instrument

Letting a coding agent write its own tests made it worse, not better

Separating the test-writing role from the repair role and freezing the tests so the repair agent cannot edit them takes the same 35B model to 72.6 per cent, 11.4 points above the no-test baseline. Both agents are the same size, so the gain is the role split rather than scale.

By the Newsroom · From arXiv 2609.09133, submitted 8 September

Naive self-generated tests 57.3 No tests at all 61.2 GPT-generated tests 65.3 ExecCritic, both roles trained 72.6
Resolve rate on the paper’s benchmark under four test regimes. The accented bar is the one that costs you something: adding self-generated tests, untrained, is worse than adding none.

Resolve rate fell to 57.3 per cent when an untrained test-writing agent was added, against 61.2 per cent with no tests at all.

Self-testing is negative before it is positive, and most scaffolds in the wild are on the negative side of that line.

The fix in the paper is structural rather than a better prompt. The agent that writes tests is separated from the agent that writes the fix, and a fail-closed harness freezes qualified tests so the repair agent cannot edit them. Role-specific post-training then lifts the Qwen test agent’s Base-to-Gold success from 22.2 to 62.2 per cent. The two trained agents composed reach 72.6 per cent resolved, 11.4 points over the no-test baseline, with no stronger model and no oracle feedback at evaluation time.

Both agents are Qwen-3.5-35B-A3B, so the gain comes from role-specific reinforcement learning rather than from scale. Tests generated by a stronger external model reached 65.3 per cent, which is better than nothing and worse than training the role.

The baseline being beaten is the authors’ own no-test configuration, which is the fair comparison and also the easy one. The number to carry away is not 72.6 but the four points that adding tests badly costs you against adding none.

What it changes If your coding agent generates its own tests and nothing stops it editing them when they fail, that loop is costing you roughly four points of resolve rate. Either give the test role its own model and freeze its output, or turn the self-generated tests off.

The figures are read from the abstract page. The benchmark family behind the resolved rate was not confirmed on the page and the per-benchmark split is unverified.

ExecCritic: Learn to Test, Test to Improve for Coding Agents

Who is telling you this

arXiv · primary

Written by
Leitian Tao, Baolin Peng, Haorui Wang and seven co-authors
For
Microsoft Research affiliated; code at github.com/MSR-Orchard/execcritic
What they gain if this is true
The paper’s best guess: they gain from a scaffold-and-recipe result being adopted, and the negative finding about naive self-tests is the part that costs them nothing to admit and helps everyone else.

Published in the edition of Thursday, 10 September 2026 · Beat: Science

More from Science

  1. Thursday, 10 September 2026 Ten thousand agents found a Navier-Stokes singularity in 88 hours, and the verifier is the transferable part
  2. Thursday, 10 September 2026 Being listed first mattered twice as much as the applicant’s name in 40,726 audit requests
  3. Thursday, 10 September 2026 Claude Sonnet 5 conceded a false premise 42 per cent of the time at turn one and 74 per cent by turn 25
  4. Tuesday, 8 September 2026 On five molecular benchmarks, most frontier models are reciting the published number
  5. Tuesday, 8 September 2026 A 4B draft model reading a coding agent’s output cut execution errors and token spend, and task success slipped
  6. Tuesday, 8 September 2026 LLM decompilers that pass every shipped test still behave differently on 4.9 per cent of functions