LLM decompilers that pass every shipped test still behave differently on 4.9 per cent of functions

Chang Liu, Edward Raff and Kristopher Micinski built an oracle that reruns decompiled C on inputs the shipped tests never used.

Recompilability is the metric the field reports, and it rewards inventing code that compiles: one candidate in twenty that passes every shipped test still diverges.

Decompile-Diverge synthesises a driver per function, grows a fuzz corpus from the reference binary, and compares a digest of the bounded observable post-state between original and candidate. Across eight systems in nine configurations on HumanEval, ExeBench, AnghaBench and MBPP, 4.9 per cent of the 12,133 candidates that passed every shipped test diverged anyway, where the raw Ghidra and Hex-Rays front ends stayed at or below 4 per cent; 77 per cent of the divergers did so by changing an output. LLM4Decompile passed more of HumanEval than raw Ghidra, 85 against 81, while diverging on 13 to 21 per cent of functions against Ghidra’s 1 to 4.

On 300 real GitHub functions and 287 CVE-grounded ones the two metrics come apart. The refinement model with the highest build rate lifts Ghidra’s to 90 per cent, up from 75, while its matched rate falls to 62 per cent, down from 74. Up to one tenth of disclosed vulnerabilities exhibit crash absence, the crash simply gone from the recompiled code with no visible trace. Source-level analysis traces the divergence to fields, types, callees and guards the model introduced where a traditional tool would have left an undefined4 or a uVar1.

The paper’s verdict: prefer the system that edits in place. DeGPT, the one refinement system that edits Ghidra’s output rather than regenerating it, stayed at or below 5 per cent divergence everywhere, and the traditional tools’ visible unknowns are a feature the metric was punishing.