A six-feature classifier beats the TruthfulQA leaderboard without reading the question

Researchers at Utah, UMass Amherst, Thoughtworks and Martian AI scored 68.9 on TruthfulQA with a six-feature logistic classifier. That is ahead of the best model on the public leaderboard, at 66.9.

The classifier never sees the question. That is the finding: a benchmark meant to measure truthfulness can be partly solved by measuring the shape of a sentence.

What the classifier does is add up six counts and compare the total. Number of negations, whether a negation opens the answer, hedge rate, word count, average token length and type-token ratio each get a learned weight, and the weighted sum decides which of two candidate answers is the true one. The question is never read. The authors state the comparison themselves, writing that “this accuracy (68.9%) is higher than the best model score on the llm-stats TruthfulQA leaderboard (66.9%, by Granite 3.3 8B Instruct, as of May 2026)”.

Negation carries most of it. Dropping the negation features alone costs the classifier 0.064 of its accuracy, the largest single contribution of the six, which says the incorrect options in TruthfulQA are written differently from the correct ones rather than merely being wrong.

Their repair is to delete the leaky pairs. TruthfulQA-476 keeps 476 of the original 790 pairs. On it the same classifier drops to 52.2 per cent, against a chance baseline of 50, which is the number showing the leak was removed rather than merely reduced.

The obvious objection is that everyone’s score falls together, so comparisons survive. It holds, and the authors measure it: model rankings on the full and cleaned sets correlate at a Spearman rho of 0.915, where 1.0 would be an order unchanged by the cleaning. Nobody picked the wrong model because of this. What was wrong was every absolute figure, including the ones in vendor cards.

They audited fourteen benchmarks the same way, and TruthfulQA is not the worst of them. HaluEval QA separates at an AUC of 0.973 on surface features alone, against 0.715 for TruthfulQA.

The authors concede their probe is “intentionally narrow and simple”, six hand-picked statistics where embeddings would find more, and rule that the right response is to treat the audit as a floor on leakage rather than a measurement of it.

Their pruning method is greedy rather than optimal, because AUC cannot be decomposed per example, so TruthfulQA-476 is one defensible cleaned set and not the cleaned set.

So the paper’s ruling is that a published TruthfulQA figure now means less than it did last week, and nobody yet knows by how much on any given leaderboard, because none has rerun on the cleaned set.