Given a weakness class and a repository, the best of 27 models found the right files at an F1 of 0.229

The strongest of 27 language models reaches a File F1 of 0.229 at locating the files behind a named weakness, Cisco’s Foundation AI reports.

The group’s co-authors are at Yale and Carnegie Mellon, and the benchmark covers a task the existing security benchmarks skip: not whether an agent can reproduce or repair a bug it has been shown, but whether it can find where a CWE lives when handed the description and a terminal.

Each of the 500 tasks pairs a repository snapshot from immediately before a real security fix with the one after it, across 290 repositories, six package ecosystems and 147 CWE categories. On the vulnerable snapshot the agent gets the CWE description and read-only terminal access and must name the files. On the patched snapshot it must say the recorded weakness is gone. Four static-analysis tools ran under the same agent interface.

The second task is where the paper’s warning lives. Systems “that identify vulnerable files effectively can still report unsupported locations on patched repositories”, so a good localiser is not a reliable clearer, and a pipeline that trusts the first to stand down after a fix will keep filing findings.

In 38.4 per cent of the 500 tasks every model scores zero, and the authors say more models and larger ones “do not remove this subset”.