The first outside numbers on Google’s TPU put it a fifth cheaper than a B200 per token

SemiAnalysis published on 7 September the first third-party inference benchmarks of Google’s Ironwood TPU, a chip with only Google’s own numbers behind it until now. On those numbers Ironwood serves a million tokens for $0.181 against $0.222 on a B200. The B300 sits above both at $0.276, all at 100 tokens per second per user.

The claim is not that the TPU is faster. It is that the hourly price is low enough to make slower cheaper, and that claim rests on a cost model SemiAnalysis built itself.

The test ran the 397-billion-parameter Qwen3.5 in FP8, matching the model and precision on the Nvidia side, on a private-beta stack called TorchTPU that exposes a TPU as a native PyTorch device through the PrivateUse1 backend, lowers torch.compile graphs to StableHLO and hands them to XLA, and calls Pallas kernels for the operations that matter. It replaces TorchAX, which ran PyTorch models by intercepting every ATen operation and translating it into JAX. Ironwood also edges the B200 on raw throughput at 20 tokens per second per user, and with the lower modelled hourly cost that becomes 50.4 per cent more tokens per dollar than a B200 and 96.0 per cent more than a B300.

Those two figures are the headline and they belong to one datapoint. At the concurrency where they are measured, the TPU’s mean time to first token is 5.41 seconds against 2.40 on the B300; the table has the B200. At a twenty-second median response time the gap narrows to $0.098 per million against $0.106 on the B200, and around a thirty-second median the B200 comes out ahead.

Ironwood has no native FP4, so on FP4 Nvidia keeps the lead outright; SemiAnalysis says that changes with TPUv8i. And the comparison that matters to a large deployer, disaggregated prefill on both sides, currently favours a GB300 NVL72 by about 30 per cent in the middle of the latency range against the TPU’s aggregated serving, because the external TPU stack has no optimised disaggregated path yet.

SemiAnalysis is not a bystander. InferenceX is its benchmark, its analysts are named in the optimisation work alongside Google’s, Inferact’s and RadixArk’s engineers, and the $1.03 per chip-hour internal cost that pushes the advantage to 76.7 per cent over the B200 is its estimate, not Google’s disclosure.

Nothing changes this week. TorchTPU stays in private beta and is due to be open-sourced around mid-October at the PyTorch Conference, after which SemiAnalysis says it will move its TPU benchmarking from a fork into the public InferenceX repository. Kimi K3 and GLM 5.3 are next in Google’s queue behind Qwen3.5. The decision this changes is a 2027 capacity plan, not a September deploy.

The kernel work is already portable: most of the Pallas kernels written for the old JAX path move to TorchTPU unchanged, which is why the October date is credible in a way the FP4 one is not.