Sparse attention

Training the token selector end to end lifts GPQA Diamond by 15.5 points at a 1,024-token budget

SAS folds the token selector’s score into the attention softmax, so the training loss reaches the ranking decision. A hard top-K cut lets no gradient through to what it threw away. On Qwen3-14B it scores 61.14 on GPQA Diamond against SeerAttention-R’s 45.64, with dense attention at 65.25.

The gain does not travel. On long-context and agentic tool-use benchmarks the same method moves scores by under 4 points, against the 15.5 it adds on GPQA Diamond.

What it changes Worth testing if you serve reasoning workloads under a tight KV budget, and not otherwise.

Primary source read

Published in the edition of Tuesday, 15 September 2026 · Beat: Science