{"id":"379088aa-7919-4a8f-bc09-5fce06db7777","arxiv_id":"2505.07360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"BinMetric is a new 1,000-question, six-task benchmark that measures LLM ability across the binary analysis lifecycle, with an empirical study of 12 models showing strong semantic understanding but weak low-level lifting and synthesis.","lead":"The authors built BinMetric, a benchmark of 1,000 questions drawn from 20 open-source C projects, to test how well large language models handle six binary analysis tasks from decompilation to assembly generation. They ran 12 LLMs on it and found that models do reasonably well at summarizing and classifying decompiled code but mostly fail at reconstructing call sites and generating working assembly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DEC/SR/CSR scores penalize unobservable stripped symbols; low scores conflate name recovery with binary analysis, and high scores would imply memorization.","rationale":"The reader's weakest assumption is that source-level ground truth from famous projects may be memorized by LLMs, because the leakage check in §4.3.1 searches only for disassembled/decompiled inputs and ignores the source-level answers. That is a real and well-supported concern. The deeper issue is that the DEC, SR, and CSR tasks are constructed so that exact identifier recovery is impossible from a stripped binary: after stripping, the input contains no symbol names, yet CodeBLEU and Rouge-L reward matching the original names. Thus the low scores are partly an artifact of the metric, not evidence of poor binary-analysis capability, and any high scores would require pretraining memorization. This makes the reader's contamination worry a symptom of a more fundamental construct-validity problem. The central empirical claim that 'LLMs can extract high-level meaning but cannot lift binaries precisely' depends on these scores; if the metric is invalid, the claim is not supported by the reported numbers. The claim is not wholly undermined, because the AIG execution-correctness result (<1%) and the AC accuracy result are less affected by identifier unobservability, and the paper is transparent about other limitations such as obfuscation and benchmark scale. Therefore the appropriate outcome remains a conditional acceptance, with the additional condition that the DEC/SR/CSR evaluation be re-validated rather than only screened for contamination; the reader's verdict does not need to change.","tokens_in":23994,"tokens_out":9103,"duration_ms":96653,"concrete_test":"Recompute CodeBLEU and Rouge-L for the existing DEC/SR/CSR model outputs using identifier-normalized ground truths (replace every function, variable, and parameter name in the original source with generic names such as v1, v2, func_N), while keeping the binary inputs unchanged. If average scores rise materially, the original metric is dominated by symbol names that are absent from stripped binaries, confirming that the low scores reflect unobservable-name recovery rather than binary analysis skill; the contamination channel would then be the only plausible source of any high scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's DEC, SR, and CSR tasks score model outputs against the original source code of 20 famous open-source projects (§4.1, §4.3.2), while the inputs are stripped binaries (§4.2.2: 'we employ the strip command to remove all symbolic information'). Function names, parameter names, and local variable names therefore do not exist in the input, yet CodeBLEU (DEC) and Rouge-L (SR/CSR) count those exact identifiers as tokens to match. A model cannot score high on these tasks unless it either guesses generic names that happen to coincide or recalls the original source from pretraining. Consequently, the low average scores (DEC 22.05, SR 18.75, CSR 5.26, Table 3) do not measure 'binary lifting' ability; they measure inability to recover unobservable names, and any success is attributable to the source-level memorization channel that §4.3.1 explicitly does not screen for: it searches the internet only for the disassembled/decompiled inputs, not the source-level answers that the models are scored against. §7.3 also does not discuss this validity threat. The conclusion that LLMs 'cannot lift binaries precisely' is therefore not established by these numbers; a model producing semantically correct C with generic identifiers would still receive a low CodeBLEU score.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BinMetric, a benchmark of 1,000 questions spanning six binary-analysis tasks (call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification, and assembly instruction generation) built from 20 real-world open-source C projects. The construction pipeline compiles the projects with DWARF information, strips symbols, disassembles/decompiles with IDA Pro, aligns binary and source via DWARF, and filters the resulting items. The authors evaluate 12 LLMs and several existing baselines, reporting per-task scores and an aggregate leaderboard. The central empirical claim is that current LLMs can extract high-level semantics from decompiled code (e.g., 53.23% accuracy on algorithm classification) but cannot yet lift binaries precisely or synthesize assembly correctly (e.g., 5.26% Rouge-L on call-site reconstruction and under 1% execution correctness on assembly instruction generation).","tokens_in":24184,"tokens_out":6037,"duration_ms":55951,"significance":"If the validity concerns raised below are resolved, BinMetric would be a useful contribution to the LLM-for-binary-analysis area: it covers a realistic task lifecycle rather than a single isolated task, uses real-world projects across multiple domains, and includes non-LLM baselines such as IDA Pro, LLM4Decompile, BinT5, and HexT5. The DWARF-based binary-source alignment is a sound technical choice, and the separation of evaluation into binary-lifting, semantic-comprehension, logical-analysis, and assembly-synthesis dimensions is a sensible way to organize the benchmark. However, the benchmark's primary value is as a measurement instrument, so the unresolved threats to the validity of the reported scores are the decisive weakness of the paper in its current form.","major_comments":[{"comment":"The data-leakage check filters against internet occurrences of the disassembled or decompiled inputs, but it does not check whether the source-level reference answers (the original C code, function signatures, and comments of the 20 flagship GitHub projects) are memorized by the evaluated LLMs. Because DEC, SR, CSR, and BCS are scored against that original source code, high scores on these tasks can be produced by pretraining memorization rather than by analysis of the stripped binary input. The threats-to-validity section (§7.3) never discusses this channel, so the claim in §6.3 that the empirical study measures LLM capability across the binary-analysis lifecycle is not yet supported.","section":"§4.3.1 and §7.3"},{"comment":"The inputs to DEC, SR, and CSR are stripped binaries with all symbolic information removed, yet the evaluation metrics (CodeBLEU and Rouge-L) reward exact token matches on function names, parameter names, and local variable names taken from the original source. Such identifiers are not present in the input, so a semantically correct reconstruction with generic identifiers would receive a low score, while a high score would require either guessing the original names or recalling them from pretraining. The reported low scores (DEC 22.05, SR 18.75, CSR 5.26 in Table 3) therefore conflate the absence of recoverable symbols with a lack of binary-lifting ability, which undermines the interpretation that LLMs 'cannot lift binaries precisely.'","section":"§4.1, §4.2.2, and §5.3"},{"comment":"The overall leaderboard in Figure 3(b) is computed by aggregating task-level metrics that have different distributions and difficulty (CodeBLEU, BLEU, METEOR, Rouge-L, Accuracy, and syntax/execution correctness), but the paper does not state the exact normalization or weighting used and does not justify equal weighting. In addition, Table 3 reports a single run for each model without variance or confidence intervals, so the reported ranking (GPT-4 first; CodeLlama-34B as the open-source winner) may not be stable; the Cohen's d analysis in Figure 4 does not remedy the absence of repeated-run variance for the headline numbers.","section":"§6.4, Figure 3, and Table 3"},{"comment":"The ground truth for the BCS task is generated by ChatGPT, which is the same model family evaluated on that task. Even though the summaries were manually reviewed for correctness, stylistic and content biases from the generator can inflate BCS scores of ChatGPT-family models relative to models whose outputs are compared against ChatGPT-style references; the paper does not quantify or control for this circularity.","section":"§4.3.2"}],"minor_comments":[{"comment":"The sentence stating that 'DeepSeek-7B and GPT-4 recording the highest and lowest scores of 25.99% and 16.63%, respectively' contradicts Table 3, where GPT-4 achieves 25.99 and DeepSeek-7B achieves 16.63 on the DEC CodeBLEU column.","section":"§6.3"},{"comment":"There are typographical errors: 'Assmebly Code' should be 'Assembly Code' in the prompt template, and 'METHOR' should be 'METEOR' in the semantic-comprehension evaluator description.","section":"Table 2 and §5.3"},{"comment":"The text uses 'CR' when referring to the CSR task, and Figure 3 labels 'Mixtral 7B' where the model is likely Mistral-7B; please standardize the naming.","section":"§6.4 and Figure 3"},{"comment":"The 'double threshold filtering' step is mentioned but the actual lower and upper code-length thresholds are never reported; these values should be given so that the filtering is reproducible.","section":"§4.3.1"},{"comment":"The table's row grouping (baseline methods vs. open-source vs. closed-source LLMs) is confusing because the 'Type' column is not explicitly populated for every row; a cleaner separation would improve readability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper does not state whether the benchmark data, prompts, and evaluation code will be released, which is important for a benchmark paper. Also, the §6.3 text/table discrepancy and the BCS label-generation circularity should both be addressed carefully in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BinMetric is a serious attempt at a gap that really exists: a six-task, 1,000-item benchmark for LLM binary analysis, built from 20 real C projects and swept across 12 models with appropriate baselines (IDA Pro, LLM4Decompile, BinT5, HexT5). The construction pipeline is documented in unusual detail—DWARF-based alignment, stripping, filtering, manual annotation effort, one-shot prompt design. If the artifacts ship, this could easily become the default evaluation harness for this subfield.\n\nThe main problem is in the scoring of three tasks. DEC, SR, and CSR are evaluated against the original source code of well-known projects, while the inputs are stripped binaries. Function names, parameter names, and local variable names are simply absent from the input, yet CodeBLEU and Rouge-L count those exact identifiers as tokens to match. A model producing semantically correct C with generic identifiers would still score near zero; a model that scores high would almost have to be recalling the source from pretraining. The leakage check only searches for the disassembled/decompiled inputs, not the source-level answers, so this channel is unexamined. The result is that the low scores on those tasks conflate name recovery with binary analysis, and the conclusion that LLMs 'cannot lift binaries precisely' is not actually established by these numbers.\n\nThe paper has other soft spots: BCS ground truth was generated by ChatGPT and human-reviewed, a circular-ish channel for the very model family under test; the leaderboard aggregates metrics with different scales; single-run scores have no variance; and no data or code link is provided. The 'first comprehensive benchmark' claim should also be qualified against Jin et al. [39], who already did binary code summarization.\n\nCredit where due: the multi-task package, the baselines, and the level of transparency about the pipeline are real strengths. The issues here are fixable with identifier-agnostic metrics, a source-level contamination check, and artifact release. I would send this to serious peer review, with major-revision conditions. It is worth a reading-group slot to debate the metric design.","headline":"A genuinely useful multi-task benchmark for LLM binary analysis, but the source-matching tasks have a serious unexamined name-recovery/memorization confound that should block acceptance until fixed.","tokens_in":24800,"tokens_out":2996,"would_cite":false,"duration_ms":29131,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BinMetric, a 1,000-question benchmark across six tasks, claims current LLMs can extract high-level meaning from binary code but cannot reliably reconstruct call sites or synthesize executable assembly.","keywords":["binary analysis","large language models","reverse engineering","benchmark","decompilation","assembly instruction generation","call-site reconstruction","binary code summarization"],"falsifier":"Re-run the evaluation on question items built from functions written after the training cutoffs of all evaluated models, or search the models' training corpora for the original source code of the 20 projects; if scores drop sharply on held-out items, or if model outputs for signature recovery and decompilation closely match the memorized originals, then the reported results include a memorization component.","tokens_in":23755,"feed_emoji":"🧩","tokens_out":4580,"duration_ms":40997,"temperature":0.7,"pith_summary":"BinMetric is a benchmark claiming to be the first comprehensive, multi-task evaluation of large language models on binary analysis, with 1,000 questions from 20 open-source C projects across six tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification, and assembly instruction generation. The paper's empirical study of 12 LLMs reports that current models can grasp high-level meaning from decompiled code—reaching 53.23% average accuracy on algorithm classification, with GPT-4 at 83.75%—but fail at precise binary lifting and assembly synthesis, scoring on average 5.26% Rouge-L on call-site reconstruction and under 1% execution correctness on generated assembly. If these results are right, they map a clear boundary: LLMs are usable for coarse semantic triage of binaries but not yet for exact reconstruction or code synthesis at the assembly level. The benchmark also provides an automated pipeline with four evaluators and a leaderboard to track progress.","feed_headline":"LLMs grasp binary code but fail to rebuild it","feed_subtitle":"A 1,000-question benchmark finds GPT-4 leads, yet assembly synthesis and call-site recovery remain unsolved.","key_machinery":"The benchmark itself is the central mechanism: a six-task question set built by compiling 20 flagship open-source C projects with DWARF debugging info, stripping symbols, disassembling and decompiling with IDA Pro, aligning binaries to source via DWARF, then filtering by length and an internet-based leakage check. Four evaluators convert raw model output into scores: a binary-lifting evaluator (Rouge-L for call-site reconstruction, CodeBLEU for decompilation), a semantic-comprehension evaluator (BLEU-1, METEOR, Rouge-L for signature recovery and summarization), a logical-analysis evaluator (accuracy for algorithm classification), and an assembly-synthesis evaluator (syntax correctness by compilation, execution correctness by test cases, Rouge-L). The six tasks are organized as stages of a reverse-engineering workflow—binary code lifting, semantic comprehension, logical analysis, assembly synthesis—so the benchmark measures a whole task lifecycle, not one isolated skill.","core_discovery":"The central discovery is that LLM binary-analysis ability splits sharply by task granularity. Across the 12 models, the average scores are high for algorithm classification (53.23% accuracy) and moderate for binary code summarization (Rouge-L 23.31%), but collapse for call-site reconstruction (Rouge-L 5.26%; only GPT-4, at 9.61%, beats IDA Pro's 8.52%) and for assembly instruction generation (syntax correctness 22.25% on average, execution correctness 0.75% on average). GPT-4 leads overall, while CodeLlama-34B is the strongest open-source model; code-specialized and larger models tend to do better, one-shot prompts help, and longer code inputs hurt. The paper reads these results as evidence that LLMs can extract and summarise high-level semantics from decompiled binaries but cannot yet perform reliable binary lifting or assembly synthesis.","pith_inferences":["The sharp task split suggests a testable hypothesis the paper does not pursue: if the source-level ground truth (the original code of Redis, SQLite, OpenSSL, etc.) is present in model training corpora, the high scores on signature recovery and summarization may be partly memorization, while the near-zero assembly-execution scores are more likely genuine capability measures.","A natural extension would be a held-out version of BinMetric built only from functions written after each model's training cutoff, or from obfuscated binaries; the paper notes obfuscation as future work, and this would clarify how much of the reported performance reflects general reasoning rather than familiar code patterns.","The very low execution correctness on assembly generation suggests that improving this task may require dedicated assembly-level training or retrieval-augmented generation rather than simply scaling general-purpose models, an inference consistent with but beyond the paper's stated conclusions."],"forward_implications":["General LLMs cannot yet serve as reliable decompilers for precise call-site reconstruction: the average Rouge-L is 5.26%, below the IDA Pro baseline of 8.52% for every model except GPT-4.","Assembly synthesis is effectively unsolved: average execution correctness is below 1%, and even GPT-4 reaches only 11% syntactic correctness.","High-level semantic tasks are closer to usable: algorithm classification averages 53.23% accuracy, with GPT-4 at 83.75%, and code-specific models outperform general ones by about 139.6% on this task.","Model size and code-specific pretraining broadly help, one-shot prompts improve overall scores by 16.65% on average, and longer code inputs reduce performance by about 6%.","CodeLlama-34B is competitive with closed-source models and is proposed as a practical open-source baseline for future binary-analysis research.","The benchmark's design also yields a workflow template: compiling, stripping, aligning, filtering, and evaluating across a six-task reverse-engineering lifecycle can be reused for future, larger binary-analysis benchmarks."],"supporting_citations":[{"why":"Supplies the LLM4Decompile decompilation baseline whose CodeBLEU scores BinMetric compares against in the DEC task.","marker":"[81]"},{"why":"Provides the IDA Pro disassembly and decompilation tool used to build the dataset and the baseline scores for CSR, DEC, and SR tasks.","marker":"[29]"},{"why":"Supplies the BinT5 baseline reproduced for the binary code summarization task.","marker":"[3]"},{"why":"Supplies the HexT5 baseline reproduced for the binary code summarization task.","marker":"[93]"},{"why":"Defines the Rouge-L metric used to measure textual consistency in CSR, SR, BCS, and AIG.","marker":"[52]"},{"why":"Defines the CodeBLEU metric used to evaluate decompilation output in the DEC task.","marker":"[75]"},{"why":"Provides the C-Algorithms project from which the 80 pseudo-code snippets for the algorithm classification task are sampled.","marker":"[10]"},{"why":"Cites HumanEval as a precedent for benchmark sample sizes, supporting the paper's argument that 1,000 items is a moderate and acceptable scale.","marker":"[12]"},{"why":"Introduces the CodeLlama models, including CodeLlama-34B, the open-source model the paper recommends as a strong baseline.","marker":"[76]"},{"why":"Introduces GPT-4, the closed-source model that leads the overall BinMetric leaderboard in the paper's study.","marker":"[2]"}],"fun_headline_variants":["Benchmark shows LLMs ace binary summary, flub assembly","LLM binary smarts: summaries easy, assembly near zero","GPT-4 leads binary benchmark but can't write working assembly","BinMetric: 1,000 questions expose LLM binary blind spots","Assembly synthesis stumps all 12 LLMs in new benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scores are only meaningful as measures of analysis skill if the models have not memorized the ground-truth source code; the 20 projects are extremely well-known and almost certainly in training corpora, but the paper's leakage check searches only for the disassembled or decompiled inputs, not for the original source answers the models are scored against.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark shows LLMs ace binary summary, flub assembly","LLM binary smarts: summaries easy, assembly near zero","GPT-4 leads binary benchmark but can't write working assembly","BinMetric: 1,000 questions expose LLM binary blind spots","Assembly synthesis stumps all 12 LLMs in new benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2437,"prompt_tokens":963,"completion_tokens":1474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":1386}},"tokens_in":579,"tokens_out":1474,"duration_ms":11189,"temperature":1.0,"reasoning_tokens":1386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:18:34.316967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on question items built from functions written after the training cutoffs of all evaluated models, or search the models' training corpora for the original source code of the 20 projects; if scores drop sharply on held-out items, or if model outputs for signature recovery and decompilation closely match the memorized originals, then the reported results include a memorization component.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HexT5 baseline reproduced for the binary code summarization task."}],"review_version":1}