LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics

Neil Archibald; Ruben Thijssen

REVIEW 2 major objections 1 minor 18 references

LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A composite Quantitative Readability Score guides LLM agents to refine decompiled C code for better readability while preserving functional correctness.

desk verdict QRS tries to fix LLM agents gaming readability metrics in decompilation but supplies no data showing the composite actually works. read the letter →

arxiv 2606.06838 v1 pith:DJKRRQBL submitted 2026-06-05 cs.SE cs.AI

Neil Archibald , Ruben Thijssen This is my paper · ORCID

classification cs.SEcs.AI

keywords LLMagentsreverseengineeringdecompiledcodereadabilitymetricsQuantitativeScorerefinementGhidra

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic decompilers output code that is functionally correct yet often difficult for humans to read, creating a bottleneck in reverse engineering. Earlier tool-driven and structure-only approaches either left gaps or allowed agents to optimize metrics in ways that reduced readability. The paper introduces the Quantitative Readability Score framework as a composite signal that combines a structural similarity gate with three sub-metrics: Lexical Surprisal, Structural Simplicity, and Idiomatic Quality. Guided by this score, LLM agents produce targeted edits that improve readability without introducing errors or unintended simplifications. This targets one stage of the decompilation workflow while leaving binary lifting and equivalence checking out of scope.

What carries the argument

The Quantitative Readability Score (QRS) framework, a composite metric that gates refinements on structural similarity while scoring lexical surprisal, structural simplicity, and idiomatic quality to direct LLM agent changes.

What would settle it

A blind human rating study in which reverse engineers judge QRS-refined samples as no more readable than the original decompiled code or as less readable than phase-2 outputs while still scoring high on QRS.

Watch

Extended reading notes

Core claim

The Quantitative Readability Score (QRS) framework, built from a structural similarity gate plus three independent sub-metrics, lets LLM agents execute readability refinements on decompiled code that raise human legibility without sacrificing correctness or allowing metric gaming.

Load-bearing premise

The three sub-metrics plus the structural similarity gate together produce a signal that blocks unintended optimizations and aligns with actual human readability judgments in reverse engineering.

Editorial extensions

If this is right

LLM agents produce decompiled code that is both functionally equivalent and easier for humans to follow.
Reverse engineering workflows gain an automated step that reduces manual cleanup effort after decompilation.
Agents avoid the structural-similarity-only pitfall of generating equivalent but less readable variants.
The same guided-refinement stage can be inserted into existing decompiler pipelines such as Ghidra.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The QRS approach could be tested on decompiled output from languages other than C to check transferability.
Further human studies might identify additional sub-metrics that strengthen the correlation with readability.
Embedding QRS scoring inside decompilers could provide live feedback during the initial lifting stage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know is that the authors noticed their earlier structural similarity approach let LLM agents produce code that passed the metric but got less readable, and they respond with the QRS composite (structural gate plus lexical surprisal, structural simplicity, and idiomatic quality). That progression is the clearest part of the work.

They lay out the phase 1 and phase 2 problems honestly and the three sub-metrics plus gate is a reasonable attempt to block the specific failure mode they saw. The broader workflow discussion gives useful context for where this fits in reverse engineering.

The soft spot is exactly what the stress-test note flags: the abstract claims QRS-guided refinement produces targeted improvements without losing correctness, yet there are no human readability ratings, no correlation numbers, no ablation results, and no details on how the sub-metrics are calculated or weighted. Without that, the demonstration remains an assumption rather than a result. The paper stops at defining the framework.

This is for people already building LLM tools for binary analysis who might want to experiment with the metric idea. Most readers outside that narrow slice will not get much from it.

I would not bring it to reading group. I would not cite it. It does not deserve peer review until the validation experiments are added.

Referee Report

2 major / 1 minor

Summary. The paper outlines a three-phase evolution in using LLM agents to improve readability of automatically decompiled C code. Phase 1 uses Ghidra MCP for tool-driven steering but suffers from incomplete coverage; Phase 2 adds structural similarity validation but reveals unintended optimizations producing less readable yet equivalent code; the contribution is the Quantitative Readability Score (QRS) composite metric that adds a structural similarity gate to three sub-metrics (Lexical Surprisal, Structural Simplicity, Idiomatic Quality). The central claim is that QRS guidance enables targeted readability improvements without sacrificing correctness, with broader workflow context provided but declared out of scope.

Significance. If the QRS sub-metrics are shown to track human reverse-engineer comprehension and block the metric-gaming behavior identified in Phase 2, the framework could supply a practical quantitative steering signal for LLM-assisted decompilation cleanup, a recurring bottleneck in binary analysis. The phased narrative usefully documents pitfalls in metric-only optimization.

major comments (2)

[Abstract / QRS framework] Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2.
[QRS framework description] QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim.

minor comments (1)

[Abstract] The final sentence of the abstract could more explicitly list which workflow stages (binary lifting, functional equivalence verification) remain out of scope to avoid reader confusion about the paper's boundaries.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments identifying areas where the manuscript's claims and technical details require clarification. We address each point below and will revise the manuscript to present the QRS framework more accurately as a proposed approach emerging from the observed limitations in Phases 1 and 2.

read point-by-point responses

Referee: [Abstract / QRS framework] Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2.

Authors: We agree that the current wording overstates the contribution. The manuscript describes the three-phase evolution and introduces QRS as the response to the metric-gaming behavior observed in Phase 2, but does not include new experimental validation, quantitative before/after results, or human ratings. We will revise the abstract and framework sections to replace the claim of demonstration with language indicating that QRS is proposed to address the identified issues, with empirical evaluation left as future work. revision: yes
Referee: [QRS framework description] QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim.

Authors: We acknowledge that the manuscript names the sub-metrics (Lexical Surprisal, Structural Simplicity, and Idiomatic Quality) without providing their precise formulas, weighting scheme, or aggregation method into QRS. We will expand the QRS framework section in the revision to include the exact computation details for each sub-metric and the structural similarity gate, along with the composite formulation. As the current work focuses on framework definition rather than validation, we do not report correlation coefficients or human judgment studies; we will explicitly note this as a limitation and direction for future validation. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected; derivation is self-contained

full rationale

The paper defines QRS explicitly as a composite of a structural similarity gate plus three author-specified sub-metrics and then reports an empirical demonstration that agents guided by this composite produce readability gains. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the supplied text. The central claim therefore rests on the empirical outcome of applying the defined metric rather than reducing to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 0 assumptions · 1 invented entities

Based on abstract only; no explicit free parameters, axioms, or invented entities beyond the QRS itself are described. The framework is introduced as a new composite without stated derivation or external grounding.

invented entities (1)

Quantitative Readability Score (QRS)
purpose: Composite metric to guide LLM agents toward readable decompiled code
Defined in the abstract as the central new artifact combining a gate and three sub-metrics

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics." pith.science (2026). https://pith.science/paper/DJKRRQBL

@misc{pith2026260606838,
  author       = {Pith},
  title        = {Pith review of: LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJKRRQBL}},
  note         = {Machine review of arXiv:2606.06838}
}

read the original abstract

Automatic decompilers produce functionally correct but often unreadable C code. This paper addresses one stage of the reverse engineering workflow: improving the readability of decompiled code using LLM agents guided by quantitative metrics. We present a three-phase research evolution. Phase 1 (tool-driven steering via Ghidra MCP) suffered from incomplete coverage and inconsistent improvements due to lack of quantitative guidance. Phase 2 (structural similarity validation alone) revealed that agents optimize for metrics in unintended ways, producing structurally equivalent but less readable code. Our contribution is the Quantitative Readability Score (QRS) framework, a composite metric combining a structural similarity gate with three independent readability sub-metrics (Lexical Surprisal, Structural Simplicity, and Idiomatic Quality). We demonstrate that QRS-guided refinement enables LLM agents to make targeted readability improvements without sacrificing correctness. We provide a discussion of the broader reverse engineering workflow (binary lifting, decompilation cleanup, and achieving functional equivalence) as context, however, it remains out of scope.

Figures

Figures reproduced from arXiv: 2606.06838 by the authors.

**Figure 2.** QRS and Sub-metric Distributions (no Bash) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 4.** QRS and Sub-metric Distributions (with Bash) 5.3 Results and Analysis Both experiments demonstrate that: • An LLM can convert decompiler-generated C into readable source code and resolve C compilation errors, without impacting the functional equivalence of the original binary. • Short iterations grounded in a deterministic objectivebased metric enable LLMs to complete long-running and complex tasks without being … view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Binary Refinement QRS Progress We observed a significant decrease in the number of iterations required to reach the QRS and structural similarity thresholds. The experiment saw a decrease in binaries failing (18) to reach the QRS and structural similarity thresholds. …

**Figure 5.** Figure 5: Threshold and QRS Score Progress Our experiments also show that an LLM can more effectively improve source code readability and structural similarity when it can validate the prompt-based objectives using Bash. This is supported by the data from our second experiment w…

**Figure 6.** Figure 6: Mean Improvement: No Bash vs With Bash Lexical Surprisal did not see a meaningful change. This is expected, as it is a stabilising metric designed to detect hallucinations and regressions in readability. Structural Simplicity also remained relatively unchanged. This is…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

[1]

s = similarity(compile(C), B) // typically 0.5-0.7
[2]

while s < threshold do
[3]

scores = function_similarities(C, B)
[4]

targets = low_similarity_funcs(scores)
[5]

C' = agent_refine(C, scores, targets)
[6]

s' = similarity(compile(C'), B)
[7]

improved

return C 3.2 The Gamification Problem The similarity loop initially worked. Scores climbed from 0.5–0.7 to 0.85–0.95 within a few iterations. However, exam- ining the “improved” code revealed a problem: readability was degrading. Code that had been partially reverse-engineered into read- able form was regressing toward low-level complexity: // Before refi...
[8]

if binary_similarity >= 0.85 then

Show all 18 references

[9]

return 1.0 // Allow QRS score
[10]

return 0.1 // Reject: functionally incorrect
[11]

surprise

end if This binary equivalence gate penalises agents for producing source code that fails to compile or diverges (15%+) from the original binary. This is critical and prevents agents from over-optimising for readability at the cost of the original functionality. Note:The thres...
[12]

if, switch, loops)

Cyclomatic complexity (cc): the number of indepen- dent control-flow paths induced by branching constructs (e.g. if, switch, loops)
[13]

nested ifs, loops)

Nesting depth (nd): the depth of nested control struc- tures (e.g. nested ifs, loops)
[14]

Function length (fl): the total number of lines of code (LOC) in the function Mathematical Definition: SS = 1−(W cc ·n cc +W nd ·n nd +W f l ·n f l) whereW cc = 0.50,W nd = 0.30,W f l = 0.20, and: •n cc = min(cc/20,1.0) •n nd = min(nd/8,1.0) •n f l = min(fl/200,1.0) Thresholds...
[15]

Determine the structural similarity score using the ‘function-level similarity analysis using radare2’s control- flow graph (CFG) comparison functionality’ strategy
[16]

Calculate each individual QRS sub-metric (Lexical Sur- prise, Structural Simplicity, and Idiomatic Quality) and the overall QRS score
[17]

Generate improvement guidance based on the lowest scoring sub-metric(s)
[18]

preserve functional- ity

Start a refinement iteration where the agent (ClaudeAgentSDK) is issued a prompt containing: •The QRS, structural similarity score, and the values for the QRS sub-metrics (Lexical Surprise, Structural Simplicity, and Idiomatic Quality) •List of critical constraints (i.e. “pres...

2024

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.

[1] [1]

s = similarity(compile(C), B) // typically 0.5-0.7

[2] [2]

while s < threshold do

[3] [3]

scores = function_similarities(C, B)

[4] [4]

targets = low_similarity_funcs(scores)

[5] [5]

C' = agent_refine(C, scores, targets)

[6] [6]

s' = similarity(compile(C'), B)

[7] [7]

improved

return C 3.2 The Gamification Problem The similarity loop initially worked. Scores climbed from 0.5–0.7 to 0.85–0.95 within a few iterations. However, exam- ining the “improved” code revealed a problem: readability was degrading. Code that had been partially reverse-engineered into read- able form was regressing toward low-level complexity: // Before refi...

[8] [8]

if binary_similarity >= 0.85 then