{"id":"c722f3eb-04ac-4471-b1dc-a883271f3a27","arxiv_id":"2607.07738","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A provenance-tracked benchmarking pipeline with an eight-gate confidence funnel shows optimization-induced attrition and survivorship bias confound LLM function-naming evaluations.","lead":"This paper builds Reforge, a pipeline that labels which decompiled functions have reliable ground truth, and shows that compiler optimization shrinks the trustworthy set from 87% to 66%. It argues that unpaired benchmark comparisons overstate LLM performance decay via survivorship bias, and that paired analysis over stable keys fixes this.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The survivorship-bias claim rests on an unvalidated stable key; under inlining/merging, mispairing could make the null paired result a noise artifact rather than evidence.","rationale":"The reader correctly identified the stable key as the weakest assumption. My stress-test agrees and raises one concrete condition: the paper's central anti-survivorship-bias conclusion is only as strong as the pairing. Since the key is not independently validated, the null paired result could be an artifact. A `-fno-inline` replication eliminates the need for the key and directly tests within-function optimization effects on the full population, not just survivors. This is feasible and should be required before the central claim is treated as established. Hence I recommend CONDITIONAL rather than ACCEPT.","tokens_in":9959,"tokens_out":9995,"duration_ms":105386,"concrete_test":"Recompile the 15 programs at -O0 and -O3 with -fno-inline, run the Reforge pipeline and the same seven-LLM naming evaluation on all source-anchored functions, and compare per-function Token F1 deltas to the paper's paired results. If the -fno-inline cohort shows significantly negative deltas (e.g., corrected Wilcoxon p < 0.05), the original null result was an artifact of the stable key or the survivor cohort; if it reproduces near-zero deltas, the survivorship-bias claim is confirmed. As a secondary check, manually inspect 50 randomly sampled paired functions to confirm key assignments.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Sections 3.4/4.2 — that unpaired optimization-level comparisons overstate decay because of survivorship — depends entirely on the stable key (test case + declaration file/line/column + normalized name) correctly pairing the same logical function at -O0 and -O3. The paper never validates this key against independent ground truth. Under inlining/merging, a -O3 binary function can contain code from several source declarations; the address-overlap join may attach it to one source key, while the model is actually scored against a different or superposed function. If such mispairing is frequent, the reported per-function deltas (−0.006 to +0.033) and Wilcoxon p ≥ 0.209 are not evidence of absence of within-function decay — they are what noise looks like. The funnel gates reduce but do not remove this risk, since SILVER/BRONZE functions are included in evaluation. A clean way to settle the issue is to re-run the naming evaluation on the same corpus compiled with -O3 -fno-inline, where every source function survives as a separate binary function and the stable key is unnecessary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript argues that the principal bottleneck in benchmarking LLM-assisted reverse engineering is not model capability but the reliability of binary-to-source alignment under compiler optimization. It introduces Reforge, a provenance-tracked pipeline that constructs function-level ground truth from synthetic C programs compiled at four optimization levels, using a DWARF oracle and a Tree-sitter syntactic oracle, aligning them via line directives, decompiling stripped binaries with Ghidra, and scoring LLM predictions against source identifiers. Alignment uncertainty is operationalized as an eight-gate confidence funnel with GOLD/SILVER/BRONZE tiers, and function identity across optimization levels is tracked with a stable source-anchored key (test case + declaration file/line/column + normalized name). On a 15-program micro-corpus, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and the paper reports that unpaired optimization-level comparisons overstate performance decay through survivorship bias: per-function paired deltas across models range from −0.006 to +0.033, with Wilcoxon signed-rank p-values ≥ 0.209. Seven LLMs are evaluated on function naming, with modest top-1 accuracy, a small upper-bound benefit from top-K shortlists, and a semantic-similarity check bounding the lexical metric's underestimate.","tokens_in":10278,"tokens_out":5324,"duration_ms":51967,"significance":"The contribution is primarily methodological. If the findings hold, the paper provides a valuable corrective to common benchmark practice: reporting alignment yield and per-stage attrition, stratifying by confidence, and preferring paired analysis over a stable source-anchored key when comparing across optimization levels. Strengths include a deterministic ground-truth core verified by repeated hash comparison, a direct check of Ghidra decompiler determinism, an auditable attrition trail in which no function is silently discarded, and an openly available pipeline and evaluation notebooks. The paper also carefully labels the empirical LLM results as proof-of-concept, not general capability estimates. However, the central survivorship claim rests on the validity of the stable key, which is not independently validated; this needs to be addressed before the methodological conclusion can be considered fully supported.","major_comments":[{"comment":"The paired-analysis conclusion in §4.2 depends entirely on the stable key (test case + declaration file/line/column + normalized name) correctly identifying the same logical function at -O0 and -O3. The paper does not validate this key against any independent ground truth, such as manual inspection of a sample of functions or a compiler-verified mapping. Under inlining and merging, the address-overlap join can attach a decompiled binary function to one source declaration while the model is scored against a different or superposed function. If such mispairing is not rare, the reported paired deltas of −0.006 to +0.033 and Wilcoxon p ≥ 0.209 are what noise would look like, and the null result would not support the survivorship-bias explanation. A direct fix is to re-run the naming evaluation on the same corpus compiled with -O3 -fno-inline, where each source function survives as a separate","section":"§3.4"},{"comment":"The paired cohort is described only as 'functions evaluable at both the lowest and highest levels.' The paper does not report the cohort size, its distribution across GOLD/SILVER/BRONZE tiers, or whether the cohort includes weak-join and gated-out functions. This matters because the mispairing risk is highest for SILVER and BRONZE functions, whose address joins may be weak or ambiguous. The authors should report the number of paired functions, their tier composition, and the paired analysis separately for GOLD-only versus all tiers. A small stratum with weak joins could be driving the null result even if GOLD functions show a real within-function decay.","section":"§4.2"},{"comment":"The abstract states without qualification that 'unpaired optimization-level comparisons overstate performance decay through survivorship bias,' and §5 asserts this is 'expected to hold for any toolchain that emits DWARF-equivalent metadata.' The empirical support is a 15-program synthetic corpus compiled with a single compiler (GCC) for a single architecture (x86-64) and decompiled with Ghidra. The paper later lists cross-toolchain replication as future work, which is in tension with the unqualified claim. The authors should either temper the claim to the observed setting (GCC/x86-64/Ghidra) or provide at least one additional toolchain/architecture replication to justify the generalization.","section":"§5"}],"minor_comments":[{"comment":"Section numbering error: '3.5 Tier Strata Revisited' appears after §4.3 and before §4.4; it should be renumbered (e.g., §4.4) or absorbed into §3.5 with cross-references adjusted.","section":"§4.3"},{"comment":"Gate 5 ('Strong address join (≥ 0.9)') and Gate 4 ('Overlap ratio ≥ 0.95') use thresholds that are presented as fixed. The paper lists a threshold sensitivity analysis as future work, but a brief statement of how sensitive the yield numbers and paired-delta results are to these thresholds would strengthen the reader's confidence that the findings are not artifacts of the hand-chosen values.","section":"Table 1"},{"comment":"The top-K oracle metric is correctly labeled an upper bound, but the sentence 'the best candidate appears at rank two or three rather than rank one, indicating a ranking or calibration gap' should be careful: the oracle selects the best candidate after seeing ground truth, so the rank distribution of improvements is not necessarily evidence of a realizable ranking gap. A minor rewording would avoid over-interpretation.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a real referee. The core contribution is a provenance-tracked ground-truth pipeline with an eight-gate confidence funnel and three-tier stratification. That's new, and it's done carefully: deterministic ground-truth core, explicit attrition audit trail, and the authors correctly separate methodological from empirical claims. The yield drop (87.2% to 65.9%) and the transition matrices showing attrition rather than tier demotion are the strongest evidence for survivorship bias, and these don't require the paired key.\n\nThe paired analysis is the part that makes the claim actionable — unpaired comparisons overstate decay because survivors are a simpler subset — and it rests entirely on the stable key (test case, declaration file/line/column, normalized name, address-overlap join). The stress-test worry is real: under inlining/merging, the same binary function can contain code from several source declarations, and the join may attach it to one key. No validation against manual inspection or an independent mapping is reported. If mispairing is common, the near-zero paired deltas are what noise looks like, and the conclusion that individual functions don't become harder to name wouldn't be supported. I don't think this is fatal — the paper's funnel and yield analysis already demonstrate that unpaired comparisons are confounded — but it's the load-bearing assumption for the stronger claim, and the authors should test it (e.g., -O3 -fno-inline) before publication.\n\nOther soft spots are minor and acknowledged: 15 synthetic programs, one toolchain, hand-chosen thresholds, and the tiers conflating complexity with ground-truth quality. The LLM results are explicitly illustrative, which I appreciate. The top-K oracle metric is clearly labeled an upper bound, and the lower-bound re-ranker analysis is a nice touch. Also good: the decompiler determinism check for the actual corpus, and the artifact is available.\n\nSo: this deserves serious peer review. The revision should focus on validating the stable key, plus maybe a sensitivity analysis on gate thresholds. The paper is for the benchmarking community and anyone measuring LLMs on reverse-engineering tasks; it should be cited for the survivorship-bias warning and the funnel design, not for the absolute model scores.","headline":"A solid, well-scoped methodology paper — the eight-gate funnel and paired stable-key analysis are genuinely new, but the paired claim rests on a key that is asserted rather than independently validated.","tokens_in":10682,"tokens_out":2077,"would_cite":true,"duration_ms":21216,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that fair benchmarking of LLMs on reverse engineering is limited by the reliability of binary-to-source alignment, not model capability, and that unpaired optimization comparisons overstate performance decay through surviv","keywords":["reverse engineering","large language models","benchmarking","decompilation","function naming","survivorship bias","DWARF","ground truth alignment"],"falsifier":"A manual audit of a random sample of paired functions checking whether the source-anchored key maps to the same logical function across levels; if mismatches are frequent, the survivorship-bias conclusion collapses. Alternatively, a comparison on a corpus with independently verified declaration sites that shows within-function decay reappearing once pairing is corrected would falsify the claim.","tokens_in":9884,"feed_emoji":"🛠️","tokens_out":2885,"duration_ms":25563,"temperature":0.7,"pith_summary":"The paper presents Reforge, a provenance-tracked pipeline for building function-level ground truth in LLM reverse-engineering benchmarks. Its central claim is that the bottleneck is not the model but the reliability of mapping compiled binary functions back to their source declarations. High-confidence ground-truth yield falls from 87.2% to 65.9% across compiler optimization levels. Comparing aggregate means across levels compares different populations of functions; the apparent decay is mostly survivorship bias, not functions becoming harder to name. Paired analysis over a stable source-anchored key removes this confound.","feed_headline":"Survivorship bias skews LLM reverse-engineering benchmarks","feed_subtitle":"A paired, source-anchored analysis shows per-function naming accuracy barely drops as compiler optimization rises.","key_machinery":"The stable, source-anchored function key: test case, declaration file, declaration line and column, and normalized function name. It tracks a logical function across optimization levels when binary-side identifiers change for roughly 86% of functions between the two lowest levels. Paired comparison over this key turns aggregate optimization comparisons into within-function comparisons. The eight-gate confidence funnel operationalizes alignment uncertainty as explicit, stratified tiers (GOLD, SILVER, BRONZE) without silently discarding functions that fail quality checks.","core_discovery":"Reforge constructs ground truth by compiling synthetic C programs, extracting function inventories from DWARF and syntax trees, aligning them via compiler line directives, decompiling stripped binaries, and gating alignment quality through an eight-check confidence funnel. On a 15-program micro-benchmark, the funnel's high-confidence yield drops from 87.2% to 65.9% with optimization, while the population of evaluable functions shrinks from 220 to 132. When the same functions are tracked across optimization levels with a source-anchored key, per-function naming accuracy changes only marginally (−0.006 to +0.033 Token F1 across models), so the aggregate decline is attributable to which functio","pith_inferences":["The survivorship-bias critique generalizes beyond function naming to any binary-analysis benchmark where compilation attrition selects simpler functions; the same paired-key logic applies to type recovery, summarization, and vulnerability detection evaluations.","The finding that roughly one in seven lexically zero-scoring predictions is semantically apt suggests current exact-match metrics undervalue synonym recovery; a standardized semantic metric would give fairer capability comparisons across models.","The stable-key design depends on the source declaration site surviving inlining and merging; a testable threat is whether the same declaration line can host different logical functions after transformation, which would mispair the analysis.","The near-zero uplift from corpus-derived re-ranking hints that useful signals for shortlist selection must come from beyond the function body, such as call-graph context or cross-reference frequency."],"forward_implications":["Benchmarks claiming optimization-level effects should report alignment yield and per-stage attrition alongside accuracy, and use paired analysis over a source-anchored key.","Accuracy numbers reported without disclosing ground-truth reliability should be understood as describing a survivorship-filtered population, especially at higher optimization levels.","Ranked shortlists of candidate names (best of three) offer a practical boost in analyst-assistance settings, but intra-function heuristics cannot recover the oracle uplift because stripped functions retain a median of only two meaningful identifier tokens.","The methodological conclusions—attrition, shifting failure modes, and survivorship bias—are properties of the compilation-and-alignment process and are expected to hold for any toolchain emitting DWARF-equivalent metadata."],"fun_headline_variants":["LLM benchmark decline is just survivorship bias","Per-function LLM naming accuracy holds; aggregate drop is bias","Reforge: separating model skill from benchmark survivor bias","Optimization cheats LLM benchmarks via survivorship bias","Why LLM decompile scores drop: it's the benchmark, not the model"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The stable key assumes the source declaration line and column uniquely identify the same logical function across optimization levels; if inlining or merging changes the declaration-to-binary mapping, the paired analysis would pair different functions and the survivorship conclusion would be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["LLM benchmark decline is just survivorship bias","Per-function LLM naming accuracy holds; aggregate drop is bias","Reforge: separating model skill from benchmark survivor bias","Optimization cheats LLM benchmarks via survivorship bias","Why LLM decompile scores drop: it's the benchmark, not the model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1087,"prompt_tokens":742,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":486,"tokens_out":345,"duration_ms":3772,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:09:12.376814+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A manual audit of a random sample of paired functions checking whether the source-anchored key maps to the same logical function across levels; if mismatches are frequent, the survivorship-bias conclusion collapses. Alternatively, a comparison on a corpus with independently verified declaration sites that shows within-function decay reappearing once pairing is corrected would falsify the claim.","supporting_citations":[],"review_version":2}