Quoted, not supported
On clinical guidelines the verbatim quote was there and the support was not, for twelve models
Johns Hopkins researchers scored twelve models on 222 questions over four clinical guidelines, one stage at a time. The quotes were real. What they proved was not.
Johns Hopkins researchers found claude-opus-5 backs 98.0 per cent of clinical claims with a verbatim quote and fully substantiates 37.1 per cent, less than half.
Two of the four authors are in computer science and two in nursing. A quote that exists is not a quote that proves the sentence it hangs on, and the two rates were measured separately, which is why the gap is visible.
The harness works in stages. Twelve models answer 222 synthetic questions written to be answerable from four clinical practice guidelines. Each factual claim in an answer is checked for a citation, then for a verbatim quote, using a deterministic string match against the guideline, then for whether the quote supports every detail of the claim, using an LLM judge the authors validated against human raters. Most models clear the second stage on prompting alone, attaching verbatim quotes to more than 90 per cent of claims; the paper names claude-haiku-4.5 as an exception on both quoting and substantiation.
The models tested were mainstream ones from five vendors, and the questions are synthetic, so a clinician’s real query may be harder or easier; the authors release the harness and the question set so that can be checked.
The number to keep is the gap, not the level. A pipeline that reports its verbatim-quote rate as evidence of grounding is reporting the 98 and hiding the 37.
What it changes If your RAG pipeline reports citation or quote coverage as a quality metric, add a substantiation stage judged separately, or the number you ship measures formatting.
Primary source read