The same Qwen weights scored 26, 62 and 68 on three hosts, and the model name predicted none of it

Three FP8 hosts served Qwen3.6-27B the same 128 locked tasks, in a paper from Iterate.ai submitted on 9 September. It scored 26.12 on DeepInfra, against 62.25 on CoreWeave and 68.26 on a third host. The weights and the precision were the same. The routes were not.

DeepInfra rejected a thirteen-image request at its four-image route limit. CoreWeave, despite advertising a 262,144-token completion maximum, “stopped at exactly 32,768 tokens with finish_reason=length.” The authors audited 18 published benchmarks and found every one of them scores the advertised model identifier, so neither failure would have shown up in any of them.

Their protocol, IB2, runs ten preflight checks on a route before any task reaches it. A five-point gap they report between two other routes for Qwen3.8-27B is weaker evidence, and they say so: those arms also differed in harness and tool-call parser. The 26 and the 62 are the stronger result, because each has a named mechanism.

Iterate.ai paid the whole inference bill, $245.99 for eleven systems or about $22 each, and evaluated none of its own products. That price is the verdict: any team can afford to test its own route, and a leaderboard that names only the model has told it nothing about the host it will pay.