Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BinMetric, a 1,000-question benchmark across six tasks, claims current LLMs can extract high-level meaning from binary code but cannot reliably reconstruct call sites or synthesize executable assembly.

desk verdict A genuinely useful multi-task benchmark for LLM binary analysis, but the source-matching tasks have a serious unexamined name-recovery/memorization confound that should block acceptance until fixed. read the letter →

arxiv 2505.07360 v1 pith:IJANPCTC submitted 2025-05-12 cs.SE

Xiuwei Shang , Guoqiang Chen , Shaoyin Cheng , Benlong Wu , Li Hu , Gangyang Li , Weiming Zhang , Nenghai Yu This is my paper · ORCID
classification cs.SE
keywords binaryanalysislargelanguagemodelsreverseengineeringbenchmarkdecompilationassemblyinstructiongenerationcall-sitereconstructioncodesummarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BinMetric is a benchmark claiming to be the first comprehensive, multi-task evaluation of large language models on binary analysis, with 1,000 questions from 20 open-source C projects across six tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification, and assembly instruction generation. The paper's empirical study of 12 LLMs reports that current models can grasp high-level meaning from decompiled code—reaching 53.23% average accuracy on algorithm classification, with GPT-4 at 83.75%—but fail at precise binary lifting and assembly synthesis, scoring on average 5.26% Rouge-L on call-site reconstruction and under 1% execution correctness on generated assembly. If these results are right, they map a clear boundary: LLMs are usable for coarse semantic triage of binaries but not yet for exact reconstruction or code synthesis at the assembly level. The benchmark also provides an automated pipeline with four evaluators and a leaderboard to track progress.

What carries the argument

The benchmark itself is the central mechanism: a six-task question set built by compiling 20 flagship open-source C projects with DWARF debugging info, stripping symbols, disassembling and decompiling with IDA Pro, aligning binaries to source via DWARF, then filtering by length and an internet-based leakage check. Four evaluators convert raw model output into scores: a binary-lifting evaluator (Rouge-L for call-site reconstruction, CodeBLEU for decompilation), a semantic-comprehension evaluator (BLEU-1, METEOR, Rouge-L for signature recovery and summarization), a logical-analysis evaluator (accuracy for algorithm classification), and an assembly-synthesis evaluator (syntax correctness by compilation, execution correctness by test cases, Rouge-L). The six tasks are organized as stages of a reverse-engineering workflow—binary code lifting, semantic comprehension, logical analysis, assembly synthesis—so the benchmark measures a whole task lifecycle, not one isolated skill.

What would settle it

Re-run the evaluation on question items built from functions written after the training cutoffs of all evaluated models, or search the models' training corpora for the original source code of the 20 projects; if scores drop sharply on held-out items, or if model outputs for signature recovery and decompilation closely match the memorized originals, then the reported results include a memorization component.

Watch

Extended reading notes

Core claim

The central discovery is that LLM binary-analysis ability splits sharply by task granularity. Across the 12 models, the average scores are high for algorithm classification (53.23% accuracy) and moderate for binary code summarization (Rouge-L 23.31%), but collapse for call-site reconstruction (Rouge-L 5.26%; only GPT-4, at 9.61%, beats IDA Pro's 8.52%) and for assembly instruction generation (syntax correctness 22.25% on average, execution correctness 0.75% on average). GPT-4 leads overall, while CodeLlama-34B is the strongest open-source model; code-specialized and larger models tend to do better, one-shot prompts help, and longer code inputs hurt. The paper reads these results as evidence that LLMs can extract and summarise high-level semantics from decompiled binaries but cannot yet perform reliable binary lifting or assembly synthesis.

Load-bearing premise

The scores are only meaningful as measures of analysis skill if the models have not memorized the ground-truth source code; the 20 projects are extremely well-known and almost certainly in training corpora, but the paper's leakage check searches only for the disassembled or decompiled inputs, not for the original source answers the models are scored against.

Editorial extensions

If this is right

  • General LLMs cannot yet serve as reliable decompilers for precise call-site reconstruction: the average Rouge-L is 5.26%, below the IDA Pro baseline of 8.52% for every model except GPT-4.
  • Assembly synthesis is effectively unsolved: average execution correctness is below 1%, and even GPT-4 reaches only 11% syntactic correctness.
  • High-level semantic tasks are closer to usable: algorithm classification averages 53.23% accuracy, with GPT-4 at 83.75%, and code-specific models outperform general ones by about 139.6% on this task.
  • Model size and code-specific pretraining broadly help, one-shot prompts improve overall scores by 16.65% on average, and longer code inputs reduce performance by about 6%.
  • CodeLlama-34B is competitive with closed-source models and is proposed as a practical open-source baseline for future binary-analysis research.
  • The benchmark's design also yields a workflow template: compiling, stripping, aligning, filtering, and evaluating across a six-task reverse-engineering lifecycle can be reused for future, larger binary-analysis benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sharp task split suggests a testable hypothesis the paper does not pursue: if the source-level ground truth (the original code of Redis, SQLite, OpenSSL, etc.) is present in model training corpora, the high scores on signature recovery and summarization may be partly memorization, while the near-zero assembly-execution scores are more likely genuine capability measures.
  • A natural extension would be a held-out version of BinMetric built only from functions written after each model's training cutoff, or from obfuscated binaries; the paper notes obfuscation as future work, and this would clarify how much of the reported performance reflects general reasoning rather than familiar code patterns.
  • The very low execution correctness on assembly generation suggests that improving this task may require dedicated assembly-level training or retrieval-augmented generation rather than simply scaling general-purpose models, an inference consistent with but beyond the paper's stated conclusions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces BinMetric, a benchmark of 1,000 questions spanning six binary-analysis tasks (call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification, and assembly instruction generation) built from 20 real-world open-source C projects. The construction pipeline compiles the projects with DWARF information, strips symbols, disassembles/decompiles with IDA Pro, aligns binary and source via DWARF, and filters the resulting items. The authors evaluate 12 LLMs and several existing baselines, reporting per-task scores and an aggregate leaderboard. The central empirical claim is that current LLMs can extract high-level semantics from decompiled code (e.g., 53.23% accuracy on algorithm classification) but cannot yet lift binaries precisely or synthesize assembly correctly (e.g., 5.26% Rouge-L on call-site reconstruction and under 1% execution correctness on assembly instruction generation).

Significance. If the validity concerns raised below are resolved, BinMetric would be a useful contribution to the LLM-for-binary-analysis area: it covers a realistic task lifecycle rather than a single isolated task, uses real-world projects across multiple domains, and includes non-LLM baselines such as IDA Pro, LLM4Decompile, BinT5, and HexT5. The DWARF-based binary-source alignment is a sound technical choice, and the separation of evaluation into binary-lifting, semantic-comprehension, logical-analysis, and assembly-synthesis dimensions is a sensible way to organize the benchmark. However, the benchmark's primary value is as a measurement instrument, so the unresolved threats to the validity of the reported scores are the decisive weakness of the paper in its current form.

major comments (4)
  1. [§4.3.1 and §7.3] The data-leakage check filters against internet occurrences of the disassembled or decompiled inputs, but it does not check whether the source-level reference answers (the original C code, function signatures, and comments of the 20 flagship GitHub projects) are memorized by the evaluated LLMs. Because DEC, SR, CSR, and BCS are scored against that original source code, high scores on these tasks can be produced by pretraining memorization rather than by analysis of the stripped binary input. The threats-to-validity section (§7.3) never discusses this channel, so the claim in §6.3 that the empirical study measures LLM capability across the binary-analysis lifecycle is not yet supported.
  2. [§4.1, §4.2.2, and §5.3] The inputs to DEC, SR, and CSR are stripped binaries with all symbolic information removed, yet the evaluation metrics (CodeBLEU and Rouge-L) reward exact token matches on function names, parameter names, and local variable names taken from the original source. Such identifiers are not present in the input, so a semantically correct reconstruction with generic identifiers would receive a low score, while a high score would require either guessing the original names or recalling them from pretraining. The reported low scores (DEC 22.05, SR 18.75, CSR 5.26 in Table 3) therefore conflate the absence of recoverable symbols with a lack of binary-lifting ability, which undermines the interpretation that LLMs 'cannot lift binaries precisely.'
  3. [§6.4, Figure 3, and Table 3] The overall leaderboard in Figure 3(b) is computed by aggregating task-level metrics that have different distributions and difficulty (CodeBLEU, BLEU, METEOR, Rouge-L, Accuracy, and syntax/execution correctness), but the paper does not state the exact normalization or weighting used and does not justify equal weighting. In addition, Table 3 reports a single run for each model without variance or confidence intervals, so the reported ranking (GPT-4 first; CodeLlama-34B as the open-source winner) may not be stable; the Cohen's d analysis in Figure 4 does not remedy the absence of repeated-run variance for the headline numbers.
  4. [§4.3.2] The ground truth for the BCS task is generated by ChatGPT, which is the same model family evaluated on that task. Even though the summaries were manually reviewed for correctness, stylistic and content biases from the generator can inflate BCS scores of ChatGPT-family models relative to models whose outputs are compared against ChatGPT-style references; the paper does not quantify or control for this circularity.
minor comments (5)
  1. [§6.3] The sentence stating that 'DeepSeek-7B and GPT-4 recording the highest and lowest scores of 25.99% and 16.63%, respectively' contradicts Table 3, where GPT-4 achieves 25.99 and DeepSeek-7B achieves 16.63 on the DEC CodeBLEU column.
  2. [Table 2 and §5.3] There are typographical errors: 'Assmebly Code' should be 'Assembly Code' in the prompt template, and 'METHOR' should be 'METEOR' in the semantic-comprehension evaluator description.
  3. [§6.4 and Figure 3] The text uses 'CR' when referring to the CSR task, and Figure 3 labels 'Mixtral 7B' where the model is likely Mistral-7B; please standardize the naming.
  4. [§4.3.1] The 'double threshold filtering' step is mentioned but the actual lower and upper code-length thresholds are never reported; these values should be given so that the filtering is reproducible.
  5. [Table 3] The table's row grouping (baseline methods vs. open-source vs. closed-source LLMs) is confusing because the 'Type' column is not explicitly populated for every row; a cleaner separation would improve readability.

Circularity Check

1 steps flagged · score 3.0 of 10

BCS ground truth is ChatGPT-generated while GPT-family models are scored against it; the other five tasks are externally anchored, so circularity is partial.

  1. self definitional [§4.3.2 (Data Construction) and §6.3, Table 3 (Semantic Comprehension results)]
    "we leverage ChatGPT to generate summaries of the source code, outlining the function’s purpose and functionality. We select a total of 250 pseudo code-summary pairs, and each generated summary undergoes manual review for correctness, ensuring high-quality ground truth of the BCS task."

    For BCS, the reference summaries against which GPT-3.5 and GPT-4 are scored were generated by ChatGPT, the same model family under evaluation. BLEU/METEOR/Rouge-L then measure alignment with ChatGPT’s annotation style, not with independent human-written summaries (which the paper rejected as unreliable). For the GPT models this makes the BCS component a self-consistency check: the yardstick is produced by the same family as the subjects. Manual review filters errors but the reference text remains ChatGPT output, so any shared stylistic prior is counted as success. Since the overall “GPT-4 leads” result averages over BCS, one of the six tasks is partially circular; DEC/SR/CSR/AC/AIG stay externally anchored.

full rationale

BinMetric is not a derivation chain; it is a benchmark with an empirical study. No fitted parameter is renamed as a prediction, and no “uniqueness theorem” or load-bearing self-citation is invoked. The DEC, SR, and CSR ground truths are the original source code of 20 public projects (an external reference, albeit one that raises memorization concerns); AC labels are manually annotated; AIG correctness is checked by compiling and executing against test cases. Thus the central claims rest mostly on independent measurements. The one genuine self-referential element is BCS: its labels were produced by ChatGPT and then GPT-3.5/GPT-4 were evaluated against them. I score this as partial circularity (3/10), not higher, because it affects one of six tasks and the top-level conclusions are also supported by non-circular tasks. Two non-circularity threats are noted for completeness but not counted as circular: (1) §4.3.1’s leakage check searches only for disassembled/decompiled inputs, not the source-level answers used as ground truth for DEC/SR/CSR, so memorization of famous projects is not ruled out; (2) low DEC/SR/CSR scores may reflect the unobservability of stripped symbol names rather than an inability to lift binary semantics. These are validity concerns, not reductions of the result to its inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark claims no fitted parameters; its free parameters are design choices (length thresholds, one-shot example selection, metric weighting). The load-bearing premises are DWARF alignment, IDA pseudo-code as the analysis substrate, ChatGPT-generated BCS labels, and absence from training corpora; the last is the most fragile because ground truth comes from the most widely copied C projects in existence.

free parameters (1)
  • code length filtering thresholds (double threshold) = not specified in paper
    §4.3.1 removes snippets 'too short' or exceeding the LLM context window; the numeric thresholds are hand-chosen and unreported, and they shape the difficulty distribution of the dataset.
assumptions (4)
  • domain assumption DWARF debugging information provides a correct and complete source-to-binary function alignment after stripping.
    Invoked in §4.2.2 to match assembly and decompiled pseudo-code to ground-truth source; if alignment is incomplete, the sampled question items inherit mislabeled ground truth.
  • domain assumption IDA Pro's Hex-Rays decompiler output is a faithful enough proxy for 'the binary' that LLM performance on it reflects binary analysis ability.
    All DEC, SR, BCS, and AC inputs are IDA-produced pseudo-code rather than raw bytes (§4.2.2), so the benchmark measures analysis of decompiler output, not of the binary itself; errors or idiosyncrasies of IDA propagate into every task.
  • domain assumption ChatGPT-generated summaries, after manual review, are valid ground truth for binary code summarization.
    §4.3.2 replaces human-written comments with ChatGPT-generated summaries because fewer than 20% of functions have usable comments; this imports a model-generated label into a benchmark whose subjects include ChatGPT and GPT-4.
  • domain assumption The benchmark inputs and ground truth are absent from LLM training corpora.
    §4.3.1 claims this is 'greatly prevented' by compiling stripped binaries the authors made, verified only by Google exact-match searches on the disassembled or decompiled code; the ground-truth source of flagship projects (Redis, SQLite, OpenSSL, FFmpeg, curl) is very likely in training data, so the assumption is fragile and unchecked for the ground truth side.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/IJANPCTC

@misc{pith2026250507360,
  author       = {Pith},
  title        = {Pith review of: BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJANPCTC}},
  note         = {Machine review of arXiv:2505.07360}
}
read the original abstract

Binary analysis remains pivotal in software security, offering insights into compiled programs without source code access. As large language models (LLMs) continue to excel in diverse language understanding and generation tasks, their potential in decoding complex binary data structures becomes evident. However, the lack of standardized benchmarks in this domain limits the assessment and comparison of LLM's capabilities in binary analysis and hinders the progress of research and practical applications. To bridge this gap, we introduce BinMetric, a comprehensive benchmark designed specifically to evaluate the performance of large language models on binary analysis tasks. BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks, including decompilation, code summarization, assembly instruction generation, etc., which reflect actual reverse engineering scenarios. Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field. The findings indicate that while LLMs show strong potential, challenges still exist, particularly in the areas of precise binary lifting and assembly synthesis. In summary, BinMetric makes a significant step forward in measuring the binary analysis capabilities of LLMs, establishing a new benchmark leaderboard, and our study provides valuable insights for the future development of these LLMs in software security.

Figures

Figures reproduced from arXiv: 2505.07360 by the authors.

Figure 1
Figure 1. Binary analysis process in case story in §3.2. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview framework of BinMetric benchmark. 4 BinMetric Benchmark This section provides an overview of the binary analysis tasks included in BinMetric (detailed in §4.1) and outlines the dataset collection, preprocessing, and construction in §4.2 and §4.3. The overview framework of BinMetric is shown in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison between LLMs. Conversely, we find that the two small-sized general models performed the worst, with Llama2-7B and DeepSeek-7B achieving scores of 15.00% and 20.00%, respectively. In this task, the code-specific models show an average 139.6% outperformance than the general ones. Meanwhile, increasing the model size also brings a notable improvement. Results on Assembly Synthesis. Generating ass… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results of practical significance (Cohen’s d). [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Impact of prompt word and code length on performance [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 13 citations worldwide. Full citation record

  1. REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Unpaired LLM binary-function naming benchmarks overstate optimization-induced decay via survivorship bias; Reforge's eight-gate funnel and paired source-anchored keys make alignment yield and attrition explicit.

  2. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.

Reference graph

Works this paper leans on

110 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [39]

    Xin Jin, Jonathan Larson, Weiwei Yang, and Zhiqiang Lin. 2023. Binary code summarization: Benchmarking chatgpt/gpt-4 and other large language models.arXiv preprint arXiv:2312.09601(2023)

    arXiv 2023

  2. [1]

    7z. 2024. https://github.com/kornelski/7z

    2024

  3. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  4. [3]

    Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen

  5. [4]

    Anthropic. 2023. Claude. https://claude.ai

    2023

  6. [5]

    Jordi Armengol-Estapé, Jackson Woodruff, Chris Cummins, and Michael FP O’Boyle. 2024. SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly. In2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, 67–80

    2024

  7. [6]

    aubio. 2024. https://github.com/aubio/aubio

    2024

  8. [7]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732 (2021). , Vol. 1, No. 1, Article . Publication date: September 2025. 20 Shang et al

    arXiv 2021

Show all 110 references
  1. [8]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al . 2024. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954(2024)

    2024 arXiv

  2. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  3. [10]

    C-Algorithms. 2024. https://github.com/TheAlgorithms/C

    2024

  4. [11]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering in large language models: a comprehensive review.arXiv preprint arXiv:2310.14735(2023)

    2023 arXiv

  5. [12]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    2021 arXiv

  6. [13]

    Qibin Chen, Jeremy Lacomis, Edward J Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. 2022. Augmenting decompiler output with learned variable names and types. In31st USENIX Security Symposium (USENIX Security 22). 4327–4343

    2022

  7. [14]

    Curl. 2024. https://github.com/curl/curl

    2024

  8. [15]

    John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature Communications15, 1 (2024), 1418

    2024

  9. [16]

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers.arXiv preprint arXiv:2212.10559 (2022)

    2022 arXiv

  10. [17]

    Yaniv David, Uri Alon, and Eran Yahav. 2020. Neural reverse engineering of stripped binaries using augmented control flow graphs.Proceedings of the ACM on Programming Languages4, OOPSLA (2020), 1–28

    2020

  11. [18]

    DeepSpeed. 2024. https://www.deepspeed.ai/

    2024

  12. [19]

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. InProceedings of the 32nd ACM SIGSOFT international symposium on software testing and ...

    2023

  13. [20]

    Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang. 2023. Towards understanding the capability of large language models on code clone detection: a survey.arXiv preprint arXiv:2308.01191(2023)

    2023 arXiv

  14. [21]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861(2023)

    2023 arXiv

  15. [22]

    EvalPlus. 2024. https://evalplus.github.io/leaderboard.html

    2024

  16. [23]

    FFmpeg. 2024. https://github.com/FFmpeg/FFmpeg

    2024

  17. [24]

    2011.Systems and Software Engineering: Systems and Software Quality Requirements and Evaluation (SQuaRE): System and Software Quality Models

    International Organization for Standardization. 2011.Systems and Software Engineering: Systems and Software Quality Requirements and Evaluation (SQuaRE): System and Software Quality Models. ISO

    2011

  18. [25]

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How Far Have We Gone in Vulnerability Detection Using Large Language Models.arXiv preprint arXiv:2311.12420(2023)

    2023 arXiv

  19. [26]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  20. [27]

    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, et al. 2024. CodeEditorBench: Evaluating Code Editing Capability of Large Language Models.arXiv preprint arXiv:2404.03543(2024)

    2024 arXiv

  21. [28]

    D Heimann. 2014. Ieee standard 730-2014 software quality assurance processes.IEEE Computer Society, New York, NY, USA, IEEE Std730 (2014), 2014

    2014

  22. [29]

    Hex-RaysSA. 2024. "IDA Pro". https://www.hex-rays.com/products/ida

    2024

  23. [30]

    Iman Hosseini and Brendan Dolan-Gavitt. 2022. Beyond the C: Retargetable decompilation using neural machine translation.arXiv preprint arXiv:2212.08950(2022)

    2022 arXiv

  24. [31]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review.arXiv preprint arXiv:2308.10620(2024)

    2024 arXiv

  25. [32]

    Peiwei Hu, Ruigang Liang, and Kai Chen. 2024. DeGPT: Optimizing Decompiler Output with LLM. InProceedings 2024 Network and Distributed System Security Symposium (2024). https://api. semanticscholar. org/CorpusID, Vol. 267622140

    2024

  26. [33]

    HuggingFace. 2024. https://huggingface.co/ , Vol. 1, No. 1, Article . Publication date: September 2025. BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models 21

    2024

  27. [34]

    ImageMagick. 2024. https://github.com/ImageMagick/ImageMagick

    2024

  28. [35]

    UNIX International

    I. UNIX International. 2010. Dwarf debugging information format version 4. https://dwarfstd.org/doc/DWARF4.pdf

    2010

  29. [36]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts.arXiv preprint arXiv:2401.04088(2024)

    2024 arXiv

  30. [37]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1430–1442

    2023

  31. [38]

    Nan Jiang, Chengxiao Wang, Kevin Liu, Xiangzhe Xu, Lin Tan, and Xiangyu Zhang. 2023. Nova+: Generative Language Models for Binaries.arXiv preprint arXiv:2311.13721(2023)

    2023

  32. [40]

    Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. 2022. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1631–1645

    2022

  33. [41]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 international symposium on software testing and analysis. 437–440

    2014

  34. [42]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2312–2323

    2023

  35. [43]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xin Zhou. 2023. Better zero-shot reasoning with role-play prompting.arXiv preprint arXiv:2308.07702(2023)

    2023 arXiv

  36. [44]

    Denkowski

    Alon Lavie and Michael J. Denkowski. 2009. The Meteor Metric for Automatic Evaluation of Machine Translation. Machine Translation23, 2–3 (sep 2009), 105–115. https://doi.org/10.1007/s10590-009-9059-4

    2009 doi

  37. [45]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931

    2023

  38. [46]

    Mira Leung and Gail Murphy. 2023. On Automated Assistants for Software Development: The Role of LLMs. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1737–1741

    2023

  39. [47]

    Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Zhi Jin, Hao Zhu, Huanyu Liu, Kaibo Liu, Lecheng Wang, Zheng Fang, et al

  40. [48]

    Libexpat. 2024. https://github.com/libexpat/libexpat

    2024

  41. [49]

    libhv. 2024. https://github.com/ithewei/libhv

    2024

  42. [50]

    libsodium. 2024. https://github.com/jedisct1/libsodium

    2024

  43. [51]

    Libvips. 2024. https://github.com/libvips/libvips

    2024

  44. [52]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013

    2004

  45. [53]

    Llama2.c. 2024. https://github.com/karpathy/llama2.c

    2024

  46. [54]

    LLM-Perf. 2024. https://huggingface.co/spaces/optimum/llm-perf-leaderboard

    2024

  47. [55]

    Clang Static Analyzer

    LLVM. 2024. "Clang Static Analyzer". https://clang-analyzer.llvm.org/

    2024

  48. [56]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568(2023)

    2023 arXiv

  49. [57]

    Maletic and Michael L

    Jonathan I. Maletic and Michael L. Collard. 2015. Exploration, Analysis, and Manipulation of Source Code Using srcML. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. 951–952. https: //doi.org/10.1109/ICSE.2015.302

    2015 doi

  50. [58]

    Masscan. 2024. https://github.com/robertdavidgraham/masscan

    2024

  51. [59]

    miniaubio. 2024. https://github.com/mackron/miniaudio

    2024

  52. [60]

    Mongoose. 2024. https://github.com/cesanta/mongoose

    2024

  53. [61]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  54. [62]

    NationalSecurityAgency. 2024. "Ghidra". https://github.com/NationalSecurityAgency/ghidra

    2024

  55. [63]

    Thinh Nguyen Hung, Hai Nguyen Phuc, Khoa Tran Dinh, Nhan Le Tran Thanh, Nghia To Trong, Khoa Ngo Khanh, Duy Phan The, and Hau Pham Van. 2023. Binary Representation Embedding and Deep Learning For Binary Code Similarity Detection in Software Security Domain. InProceedings of th...

    2023

  56. [64]

    National Institute of Standards and Technology (NIST). 2023. Trustworthy and Responsible AI. https://www.nist. gov/trustworthy-and-responsible-ai

    2023

  57. [65]

    OpenSSL. 2024. https://github.com/openssl/openssl

    2024

  58. [66]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems35 ...

    2022

  59. [67]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia...

    2002

  60. [68]

    James Patrick-Evans, Lorenzo Cavallaro, and Johannes Kinder. 2020. Probabilistic naming of functions in stripped binaries. InProceedings of the 36th Annual Computer Security Applications Conference. 373–385

    2020

  61. [69]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768

    2022

  62. [70]

    Kexin Pei, Jonas Guan, Matthew Broughton, Zhongtian Chen, Songchen Yao, David Williams-King, Vikas Ummadisetty, Junfeng Yang, Baishakhi Ray, and Suman Jana. 2021. Stateformer: Fine-grained type recovery from binaries using generative state modeling. InProceedings of the 29th A...

    2021

  63. [71]

    Pintools. 2024. https://www.intel.com/content/www/us/en/developer/articles/tool/pin-a-dynamic-binary- instrumentation-tool.html

    2024

  64. [72]

    PyTorch. 2024. https://pytorch.org/

    2024

  65. [73]

    Radare2. 2024. https://github.com/radareorg/radare2

    2024

  66. [74]

    Redis. 2024. https://github.com/redis/redis

    2024

  67. [75]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  68. [76]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  69. [77]

    Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. Arb: Advanced reasoning benchmark for large language models.arXiv preprint arXiv:2307.13692(2023)

    2023 arXiv

  70. [78]

    SQLite. 2024. https://github.com/sqlite/sqlite

    2024

  71. [79]

    Statista. 2024. https://www.statista.com/statistics/1101442/iot-number-of-connected-devices-worldwide/

    2024

  72. [80]

    Iain Sutherland, George E Kalb, Andrew Blyth, and Gaius Mulley. 2006. An empirical examination of the reverse engineering process for binary files.Computers & Security25, 3 (2006), 221–228

    2006

  73. [81]

    Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. 2024. LLM4Decompile: Decompiling Binary Code with Large Language Models.arXiv preprint arXiv:2403.05286(2024)

    2024 arXiv

  74. [82]

    Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation: A Survey.arXiv preprint arXiv:2402.13446(2024)

    2024 arXiv

  75. [83]

    Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, and Nenghai Yu. 2024. GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models.arXiv preprint arXiv:2408.12494(2024)

    2024 arXiv

  76. [84]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  77. [85]

    Transformers. 2024. https://huggingface.co/

    2024

  78. [86]

    Ultrajson. 2024. https://github.com/ultrajson/ultrajson

    2024

  79. [87]

    Valgrind. 2024. https://valgrind.org/

    2024

  80. [88]

    Binary Ninja

    Vector35. 2024. "Binary Ninja". https://binary.ninja/

    2024

  81. [89]

    Hao Wang, Zeyu Gao, Chao Zhang, Zihan Sha, Mingyang Sun, Yuchen Zhou, Wenyu Zhu, Wenju Sun, Han Qiu, and Xi Xiao. 2024. CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision.arXiv preprint arXiv:2402.16928(2024)

    2024 arXiv

  82. [90]

    Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. InProceedings of the 31st ACM Joint European Software , Vol. 1, No. 1, Article . Publication date: September 2...

    2023

  83. [91]

    Whisper.cpp. 2024. https://github.com/ggerganov/whisper.cpp

    2024

  84. [92]

    Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah

  85. [93]

    Jiaqi Xiong, Guoqiang Chen, Kejiang Chen, Han Gao, Shaoyin Cheng, and Weiming Zhang. 2023. HexT5: Unified Pre-Training for Stripped Binary Code Information Inference. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 774–786

    2023

  86. [94]

    Zhengzi Xu, Bihuan Chen, Mahinthan Chandramohan, Yang Liu, and Fu Song. 2017. Spain: security patch analysis for binaries towards understanding the pain and pills. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 462–472

    2017

  87. [95]

    InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

    How effective are neural networks for fixing security vulnerabilities. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1282–1294

  88. [96]

    Shouguo Yang, Long Cheng, Yicheng Zeng, Zhe Lang, Hongsong Zhu, and Zhiqiang Shi. 2021. Asteria: Deep learning- based AST-encoding for cross-platform binary code similarity detection. In2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN...

    2021

  89. [97]

    Shouguo Yang, Chaopeng Dong, Yang Xiao, Yiran Cheng, Zhiqiang Shi, Zhi Li, and Limin Sun. 2023. Asteria-Pro: Enhancing Deep Learning-based Binary Code Similarity Detection by Incorporating Domain Knowledge.ACM Transactions on Software Engineering and Methodology33, 1 (2023), 1–40

    2023

  90. [98]

    Dapeng Yan, Zhipeng Gao, and Zhiming Liu. 2023. A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1887–1898

    2023

  91. [99]

    Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang

  92. [100]

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation.arXiv preprint arXiv:2305.04207(2023)

    2023 arXiv

  93. [101]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Soft...

    2024

  94. [102]

    Yifan Zhang, Chen Huang, Yueke Zhang, Kevin Cao, Scott Thomas Andersen, Huajie Shao, Kevin Leach, and Yu Huang

  95. [103]

    KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models.arXiv preprint arXiv:2402.15043(2024)

    2024 arXiv

  96. [104]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, and et.al. 2023. A Survey of Large Language Models. arXiv:2303.18223

    2023 arXiv

  97. [105]

    Zhengran Zeng, Yidong Wang, Rui Xie, Wei Ye, and Shikun Zhang. 2024. CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios.arXiv preprint arXiv:2403.19287(2024)

    2024 arXiv

  98. [108]

    Zhuo Zhang, Wei You, Guanhong Tao, Yousra Aafer, Xuwei Liu, and Xiangyu Zhang. 2021. Stochfuzz: Sound and cost-effective fuzzing of stripped binaries by incremental and stochastic rewriting. In2021 IEEE Symposium on Security and Privacy (SP). IEEE, 659–676

    2021

  99. [110]

    zstd. 2024. https://github.com/facebook/zstd , Vol. 1, No. 1, Article . Publication date: September 2025

    2024

  100. [2022]

    Pre-Training Representations of Binary Code Using Contrastive Learning.arXiv preprint arXiv:2210.05102 (2022)

    2022

  101. [2023]

    In2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

    Extending source code pre-trained language models to summarise decompiled binarie. In2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 260–271

  102. [2024]

    DevEval: Evaluating Code Generation in Practical Software Projects.arXiv preprint arXiv:2401.06401(2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.