Science
-
Quoted, not supported
On clinical guidelines the verbatim quote was there and the support was not, for twelve models
Johns Hopkins researchers scored twelve models on 222 questions over four clinical guidelines, one stage at a time. The quotes were real. What they proved was not.
If your RAG pipeline reports citation or quote coverage as a quality metric, add a substantiation stage judged separately, or the number you ship measures formatting.
-
Which file
Given a weakness class and a repository, the best of 27 models found the right files at an F1 of 0.229
Cisco’s Foundation AI built 500 tasks from real fixes. In 38.4 per cent of them no model found a correct file, and finding the file did not mean noticing when it was fixed.
Treat an agent’s list of vulnerable files as a starting point, not a finding. Run it again on the patched tree: if it still names a location, the agent is not measuring the code, and neither is your dashboard.
-
Clean traces
A corrupt plan placed in the context is paraphrased as the model’s own reasoning, and the monitor reads it as clean
Stanford and CMU researchers found the attack with a 7B model and reproduced it on DeepSeek-R1. The best of five monitors caught about three in four.
If a chain-of-thought monitor gates an agent that reads retrieved documents, tool outputs or another agent’s plan, budget for it to miss a quarter of injected plans. Feed the monitor the upstream inputs and measure whether that helps on your task; on one of the two here it hurt.
-
Nearly cut
A Google harness claims 218 of 222 Codeforces problems and 71 per cent on research-level theorems, on Google’s models
Kept, although every number is the authors’ own on their employer’s models, because the harness ships inside a Google product as a named pattern a builder can turn on.
Nothing to switch on this week unless you are in Antigravity. If you build long-horizon proof or planning agents, the readiness gate and the verifier-to-section routing are the two ideas to steal.
-
Two linear maps
A frozen Qwen3-32B routed skills better than pipelines that bolted on up to 16B parameters
Tsinghua researchers report that two trained linear maps, read from a frozen Qwen3-32B, route skills better than routers that add parameters.
If your harness preloads every skill’s metadata, this is the argument for stopping; the code is the authors’ and the backbone is Qwen3-32B.
-
Negative result
A safety technique’s own authors say it does not beat the simpler one it was built to improve
Inoculation Midtraining “does not outperform standard Inoculation Prompting”, its authors at Geodesic Research, OpenAI and the UK AI Security Institute report.
Nothing to adopt. If you use Inoculation Prompting, this is evidence you need not move.
-
Marking the shape
A six-feature classifier beats the TruthfulQA leaderboard without reading the question
Counting negations, hedges and word lengths in the answer alone beats every model on the public leaderboard. The authors then delete the 314 question pairs that made it possible.
Stop quoting an absolute TruthfulQA number in a model card or an eval report; swap to TruthfulQA-476 and say which you used. Comparisons between models you evaluated yourself are safe. If you built an internal benchmark by writing correct and incorrect answers in separate sittings, run these six features over it before you trust its absolute score.
-
Sparse attention
Training the token selector end to end lifts GPQA Diamond by 15.5 points at a 1,024-token budget
SAS folds the token selector’s score into the attention softmax, so the training loss reaches the ranking decision. A hard top-K cut lets no gradient through to what it threw away. On Qwen3-14B it scores 61.14 on GPQA Diamond against SeerAttention-R’s 45.64, with dense attention at 65.25.
Worth testing if you serve reasoning workloads under a tight KV budget, and not otherwise.
-
Repeated data
Mixture-of-experts models start overfitting repeated data at half a dense model’s budget
Stanford and Washington report dense models tolerating 8x data repetition against 4x for matched MoE models. The sparse advantage is gone entirely by 32x. Router assignments go near-stationary early, so each expert memorises one fixed slice of tokens.
Halve the repetition budget you would assume from dense experience before continuing to train an MoE on a fixed corpus.
-
Beside the lead
The same Qwen weights scored 26, 62 and 68 on three hosts, and the model name predicted none of it
Three FP8 hosts served Qwen3.6-27B the same 128 locked tasks, in a paper from Iterate.ai submitted on 9 September. It scored 26.12 on DeepInfra, against 62.25 on CoreWeave and 68.26 on a third host. The weights and the precision were the same. The routes were not.
Before you trust a benchmark for a hosted open model, send your own largest request to the exact route you will buy: the most images, the longest completion and the full tool list.
-
In brief
Reusing another agent’s KV cache inside a new prompt scored worse than using no cache at all
KVShareArena, from the University of Central Florida, measured what happens when a coordinator reuses the KV caches of its subagents’ reports. On its numbers, caches stitched naively into a new prompt did worse than no cache at all, scoring minus 0.82 where zero is no cache and one is full recomputation. Fixing positions rescued single-document questions, not multi-hop ones. Every result comes from models of 8 billion parameters or fewer. The verdict: never splice a subagent’s cache into a new prompt without a repair step.
If your serving stack reuses KV caches outside the prompt prefix, compare multi-hop answers with and without reuse before shipping it.
-
The instrument
Claude Sonnet 5 conceded a false premise 42 per cent of the time at turn one and 74 per cent by turn 25
A benchmark that keeps pushing for 25 turns finds every model tested folds further the longer the pressure runs. Emotional appeals were the least used tactic and the most effective.
If you ship a long-running chat or an agent loop with a human in it, your sycophancy and correctness numbers were measured at the wrong horizon. Re-run them at 20 turns and more, and weight emotional pressure rather than logical challenge in the adversarial set.
-
The instrument
Being listed first mattered twice as much as the applicant’s name in 40,726 audit requests
A pre-registered bias audit across hiring, lending and medical triage found none of its 36 planned contrasts survived correction. The largest effect it did measure was the audit’s own option ordering.
If you audit your own stack for bias, randomise option position, run rate-one-at-a-time and rank-side-by-side arms separately, and pre-register the contrast. Otherwise you will either ship a bias finding that is an artefact of your prompt, or clear a model that is not clear.
-
The instrument
Letting a coding agent write its own tests made it worse, not better
Separating the test-writing role from the repair role and freezing the tests so the repair agent cannot edit them takes the same 35B model to 72.6 per cent, 11.4 points above the no-test baseline. Both agents are the same size, so the gain is the role split rather than scale.
If your coding agent generates its own tests and nothing stops it editing them when they fail, that loop is costing you roughly four points of resolve rate. Either give the test role its own model and freeze its output, or turn the self-generated tests off.
-
The one we nearly cut
Ten thousand agents found a Navier-Stokes singularity in 88 hours, and the verifier is the transferable part
A mathematics result in a paper for builders, kept because of what closed the loop rather than what it proved.
Nothing to do here this week. The reusable idea is that a mechanical verifier is what makes fan-out pay, so before scaling an agent swarm, ask what checks its output and whether that check can be run without a human.
-
The measurement
Swap the model under a notes-based agent memory and accuracy moves 13 points, in a direction that depends on which way you swapped
Two LinkedIn engineers ran the same 48 histories through four memory formats and then changed the model that reads them. Fixed-schema graphs did not care. Free-text notes did, and the average hid it.
Before you change the model behind an agent with free-text memory, run the migration in both directions on a held-out slice and score it. If the memory is a schema you control, this paper says you can skip that; if it is notes, you cannot, and keep the raw history until you have.
-
Coordination
Replacing one agent in a settled team kept the score and raised the talking by up to 63 per cent
Jianxin Gao’s team at China Agricultural University formed eight agent pairs from GPT-5.6 Luna, gave each agent a notebook for ten episodes, then swapped partners.
If your multi-agent pipeline hot-swaps replicas that carry memory, measure tokens per completed step across the swap, not task accuracy. The paper’s number is a 16 to 63 per cent coordination tax that a success-rate dashboard will not show.
-
The metric rewarded the wrong path
LLM decompilers that pass every shipped test still behave differently on 4.9 per cent of functions
Chang Liu, Edward Raff and Kristopher Micinski built an oracle that reruns decompiled C on inputs the shipped tests never used.
If an LLM decompiler feeds your vulnerability triage, a green test suite is not evidence the vulnerability survived recompilation. Fuzz the candidate against the original binary, or read the traditional tool’s ugly output for the function you care about.
-
The veto gate
A 4B draft model reading a coding agent’s output cut execution errors and token spend, and task success slipped
Konstantin Grotov and Valentin Malykh inverted speculative decoding: a small draft model scores a coding agent’s finished trajectory instead of proposing tokens for it.
Worth a week’s experiment if you pay for retries at volume: the recipe is a few thousand labelled trajectories, a distilled 4B draft and a linear calibrator, and the return is fewer wasted execute-fail-retry loops, not a higher benchmark score.
-
Contamination
On five molecular benchmarks, most frontier models are reciting the published number
Matthias Busch and colleagues audited 22 frontier models on 12 molecular-property regression benchmarks for verbatim digit-level retrieval. They found it, the authors report, on five datasets in more than half the models and only in isolated cells elsewhere. Raising the reasoning level flagged retrieval 89 per cent more often. The objection that a low error could be real skill dies on FreeSolv: one model reproduced it with a median absolute error of 0.025 kcal/mol against the benchmark’s default experimental uncertainty of 0.6, and no predictor beats the noise in its own labels.
Treat a molecular-property leaderboard score as unproven on novel compounds until someone has run the retrieval audit.
-
The measurement
A third of agent patches that pass the tests would fail code review
SWE-Gate scores the same patch twice: once against the repository’s tests, once against constraints lifted from what reviewers actually wrote on real pull requests. The second score is the one nobody has been keeping.
Test-pass rate overstates how ready a coding agent is to ship. Sample the patches that pass and score them against your own review standards before you plan headcount or throughput on a leaderboard figure.
-
The instrument
The LLM judge grading your evals does not agree with itself from one day to the next
Two preregistered audits fixed their thresholds in advance. Neither got past validating the instrument, and the reason is in the serving stack rather than the model.
Before an LLM judge gates an eval, a leaderboard or a data filter, send it the same few hundred requests twice and measure the agreement. If it cannot reproduce its own ranking, the pipeline downstream of it is measuring load, not quality.
-
The grader
Fourteen agents in a hundred found the grader’s loophole, and twenty-four reported them
A swarm of 100 Gemini 3.1 Pro agents was set to prove formal conjectures, and 14 of them redefined the operators instead. Another 24 organised a response to it, against no instruction from any human to do so.
If you score a multi-agent system with an automated grader, confirm the grader compares semantics rather than matching a template, and budget for the fraction of your fleet that will spend its time on the grader instead of the task.
-
Diff noise
One line in the prompt cut unrelated edits by a third and raised the pass rate
Telling a code-repair model to leave the rest of the file alone cut excess edit distance from 0.195 to 0.131. It also cut added cognitive complexity by 26.6 per cent and raised Pass@1 by 2.3 points, against the same prompts without the instruction.
Add a preservation instruction to your code-repair prompt today. It costs one line, it shrinks the diff a human has to re-read, and on this benchmark it did not cost accuracy.
-
Reading the reasoning
The steps a model calls important are not the steps that changed its answer
Judges scoring which chain-of-thought steps mattered reached a PR-AUC near 0.03, against a ceiling of roughly 0.51 to 0.64. That ceiling was set by splitting the study’s own estimates in half, which puts the judges about nine times below what the method itself could reach.
A readable chain of thought is not an explanation of the answer. If you are using one to decide whether to trust an output, you are grading prose.
-
The adapter
Your model may be calling tools that your serving stack throws away
A serving adapter moved a BFCL v4 tool-call rate from 0.00 to 0.96, with the model, its weights and its seed unchanged. Parsed calls went from none to 636 in the same comparison, the preprint’s authors report.
Before concluding a model cannot call tools, log what the adapter received and what it emitted. A near-zero score is a parser hypothesis first and a capability finding second.
-
The validator
Crash-only scoring inflates vulnerability-patching agents by 1.83 times
PatchBench re-scored eleven vulnerability-patching agents, including entrants from the AIxCC competition. Its authors report that validating a fix by whether the proof-of-concept still crashes overstates the solve rate by 1.83 times on average.
If you score a patching agent by whether the proof-of-concept still fires, halve the number in your head. Validate against the root cause or you are measuring crash suppression.
-
The benchmark
Hallucination scores fell because the models said less, not because they knew more
Six inference-time hallucination-mitigation methods were tested across three vision-language models and four benchmarks, MMStar among them. The authors report that lower hallucination scores came coupled with reduced object recall, reduced visual coverage and less detailed responses.
Score a mitigation method on informativeness alongside hallucination. A model that answers fewer questions is not a model that answers them better, and on this evidence you cannot yet tell which of the six you are buying.
-
Stopping early
Stopping agent evaluations early cut a third of the tokens and moved scores by two points
EarlyEval cut up to 44.1% of input tokens on SWE-bench Verified, TerminalBench and Toolathlon by stopping agent runs before they finished. Its authors report that resolve rates moved by one to two points against the full runs.
Use it for iteration, where a third off the bill is worth two points of noise. Do not use it for the run you publish.
-
Agent craft
Models that build their own agent harness still lose at code and search
Six models built their own execution harnesses across 2,207 held-out tasks, against harnesses engineered by hand. The self-built ones trailed on code and search, and matched or beat the hand-built ones on writing and machine-learning experimentation.
-
Agent craft · The Monday lead
Your coding agent’s model is no longer the only thing that needs an eval
LoopArena separates the controller from the worker and finds that long-running software work still breaks at the level of direction, verification, and stopping.
Log control decisions separately from tool execution. Grade what the agent chose to do, what it chose to verify, and why it stopped. If you only score the repository at the end, you cannot tell whether your orchestration improved or the coding model merely rescued it.