The LLM judge grading your evals does not agree with itself from one day to the next

Haoyaun Zhu and Jie Zhang audited 52,988 requests to a hosted LLM judge and got a rank agreement of 0.400 against the 0.90 they preregistered.

A judge that cannot reproduce its own ranking is not a noisy instrument. It is not an instrument.

The thresholds were fixed before the data was collected, which is what makes the result hard to argue with. Repeats inside the same window agreed at 0.400 against a required 0.90. Byte-identical replays sent the next day agreed at 0.78 against a required 0.99. In both campaigns the execution record sat at ceiling, meaning the harness did what it was told every time. The failure is not in the plumbing.

Three mechanisms explain it. One is a label-to-meaning mapping that biased the readout. Another is a near-degenerate measurand: the score separations the study needed to resolve sat four to seven orders of magnitude below the noise floor, with 84 per cent of task groups holding candidates of identical correctness. Asking a judge to rank items that are genuinely the same produces a ranking, and it produces a different one each time.

The third is the one that touches every reader. A shared endpoint batches concurrent requests together, and the inference kernels are not batch-invariant, so the arithmetic your request goes through depends on who else was in the batch. Providers also swap deployments behind a fixed model name without changing the name. Nothing in the response metadata records either fact. Concurrent load alone raised disagreement 8.4-fold against the unloaded case, which means the same evaluation run on a busy afternoon and a quiet night is two different measurements wearing one model id.

The primary campaign audited qwen3.7-plus. A cross-provider supplement covering deepseek-v4-flash, mistral-small-2603 and gpt-4.1-2025-04-14 over two days in August 2026 ran 4,000 calls and found the same shape, so this is not one vendor’s bad week.

The authors’ own recommendation is the cheap one, and it is worth more than the finding. A pilot at roughly 2 per cent of the study’s call volume would have exposed both unreachable gates before either campaign was funded. Run the repeat first, on a few hundred calls, and check the judge agrees with itself before you let it rank anything.

This paper is a preprint dated 3 September and it audits two campaigns, not the whole practice. What it establishes is narrower than the headline temptation: not that LLM judges are worthless, but that a judge on a shared endpoint has a reproducibility ceiling nobody has been measuring, and that the ceiling can sit below the threshold a study needs. If your pipeline gates training data on a hosted judge, the 2 per cent pilot tells you which side of that ceiling you are on.