Pith. sign in

REVIEW 5 major objections 7 minor 94 references

ReCopilot: Reverse Engineering Copilot in Binary Analysis

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7-billion-parameter reverse-engineering model, trained on a large corpus of stripped binaries and augmented with data-flow context, outperforms existing tools and general LLMs by 13% on average across binary-analysis tasks.

desk verdict Solid systems work with a real risk that the SOTA margin is inflated by train/test overlap; needs artifact release and dedup evidence before the 13% claim is taken seriously. read the letter →

arxiv 2505.16366 v1 pith:7DRKA2LH submitted 2025-05-22 cs.CR

Guoqiang Chen , Huiqi Sun , Daguang Liu , Zhiqi Wang , Qiang Wang , Bin Yin , Lu Liu , Lingyun Ying This is my paper · ORCID
classification cs.CR
keywords binaryanalysisreverseengineeringdecompiledpseudocodefunctionnamerecoveryvariabletypeinferencechain-of-thoughtdirectpreferenceoptimizationstaticprogram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small, domain-trained language model can take over the most tedious parts of binary reverse engineering: recovering function names, variable names and types, structure layouts, and readable pseudo code from stripped binaries. It reports that a 7-billion-parameter model trained in three stages—continued pretraining, supervised fine-tuning, and direct preference optimization—beats both existing binary-analysis tools and much larger general-purpose LLMs by an average of 13% on its own benchmark. The largest single win is variable name recovery, where it leads the previous best tool by 19.74 percentage points. The implication the authors emphasize is that domain-specific training plus static-analysis context, rather than sheer model size, is what moves the needle in this niche.

What carries the argument

The load-bearing mechanism is the prompt built from static analysis: the target decompiled function plus up to ten context functions selected by an informativeness score, the call chains linking callers and callees, and a data-flow trace that propagates a traced variable through assignment, callee, and caller sites while recording alias relationships. This trace is computed by a lightweight recursive traversal of the abstract syntax tree of the pseudo code, using rules for definitions, assignments, expressions, and function calls; it is what lets a small model see how a variable is used across function boundaries. On the training side, a generator-discriminator loop produces chain-of-thought supervised fine-tuning examples so the model learns to reason before answering, and direct preference optimization aligns it toward format-valid JSON outputs.

What would settle it

A blind comparison in which experienced analysts receive ReCopilot's output and the best baseline's output for the same stripped functions, side by side, and have to pick which names, types, and summaries they trust; if the analysts do not choose ReCopilot's predictions at a meaningful rate, the claimed 13% advantage does not translate into analyst value.

Watch

Extended reading notes

Core claim

On the paper's own terms, ReCopilot's central discovery is that a 7B model, when trained on aligned stripped-binary/source-code pairs and prompted with call chains and variable alias flows, recovers symbolic information from decompiled pseudo code at a level comparable to or better than a much larger general reasoning model. Across six evaluated tasks—function name recovery, variable name recovery, variable type recovery, struct recovery, decompilation, and summarization—ReCopilot reports state-of-the-art scores, for example 50.59 for function name recovery and 43.50 for variable name recovery on its benchmark, with an average 13% advantage over the second-best method. The authors present this as evidence that an expert-model approach with context enhancement is a viable path to automating binary analysis.

Load-bearing premise

The load-bearing premise is that the benchmark's automated scores—partial-credit name matching, coarse type clusters, CodeBLEU, and an LLM acting as judge—are valid proxies for how much ReCopilot helps a human reverse engineer.

Editorial extensions

If this is right

  • If the benchmark results hold, a laptop-deployable 7B model can serve as a practical assistive tool for reverse engineers, reducing dependence on huge cloud LLMs for each decompiled function.
  • The data-flow tracing context is transferable: adding it to a general LLM improved structure-layout recovery by 4.7 points, so the technique can be bolted onto models that have not been domain-trained.
  • Direct preference optimization's main measurable benefit here is format reliability, lifting the success ratio from 0.82 to 0.92, at a small cost in structure-recovery and decompilation scores.
  • The paper's findings indicate that the remaining bottleneck is training data and reasoning stability rather than representation: the super-long chain-of-thought variant did not improve results, and the authors attribute that to an 11K-example dataset.
  • Absolute scores still leave room before the tool can be trusted for high-assurance downstream tasks, since function-name and type recovery remain below 60 percent on the paper's own metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 13% headline depends on metric choices—partial-credit name matching, coarse clustering of basic types, CodeBLEU for decompilation, and LLM-as-a-judge for summarization—so a human-utility study could shrink or erase the margin; the paper does not run one.
  • The alias-trace data-flow context is cheap to compute (about 0.018 seconds per function at one level), so a natural extension is to combine it with a decompiler's own type propagation or with formal dataflow analysis to produce verified traces beyond the paper's heuristic rules.
  • A testable extension of the same recipe would target Go, Rust, or Windows binaries; the authors claim no serious obstacle, but the corpora and sanitizers would need to be rebuilt.
  • The Super-CoT result is underpowered by the authors' own account; scaling that dataset by an order of magnitude, or switching to reinforcement learning as the paper suggests, is the direct check on whether long chains of thought help in this domain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents ReCopilot, a 7B-parameter LLM specialized for binary-analysis tasks on decompiled pseudo-code. The authors collect a raw dataset of 101M stripped binary functions, build a 36B-token continued-pretraining corpus, a 1.8B-token SFT mixture, and a 2.4K-pair DPO set, and train from Qwen2.5-Coder-7B. They augment prompts with call-chain and data-flow context and use a generator-discriminator pipeline to synthesize long chain-of-thought data (including 'Super-CoT'). They introduce a six-task benchmark (function name, variable name, variable type, struct, summarization, decompilation) and report state-of-the-art results, claiming a 13% average improvement over the second-best method, while also showing degradation on general benchmarks and a failure of Super-CoT to improve performance.

Significance. If the 13% claim survives scrutiny, this is a practically valuable result: a laptop-deployable 7B model that beats 671B general-purpose LLMs on decompiled-code analysis would reduce the cost of reverse-engineering assistance and support offline security work. The paper's strengths are its large and reproducibly described data-collection pipelines, the clean three-stage training recipe, the inclusion of general-capability and ablation evaluations, honest reporting of the Super-CoT limitation, and a public demo. However, the headline claim is currently not backed by the reported tables, and several evaluation-protocol gaps (leakage control, single runs, unvalidated judge) make the result fragile.

major comments (5)
  1. [Abstract; §5.2, Tables 4-5] The claimed '13%' average improvement is not recoverable from the reported numbers. Recomputing the per-task gaps in Table 4 against the strongest visible baseline gives roughly 16.04 (func name), 19.74 (var name), 5.06 (var type vs DeepSeek-V3 in Table 5), 13.20 (struct), 3.72 (decompilation), and 0.06 (summarization), which averages to about 9.6 percentage points; against DeepSeek-R1 in Table 5 the average gap is about 5.0 points, with ReCopilot actually behind on decompilation (-0.54) and summarization (-1.11). The paper must state exactly which baselines are included in the average and recompute the headline, or the '13%' claim should be withdrawn.
  2. [§4 (Test Dataset); §3.3.1] The benchmark's validity rests on the 1,038-function test set being unseen and non-duplicated. The test set is drawn from open-source projects in crypto, network, multimedia, etc., which are the same distribution as the 101M-function training corpus from Archlinux/Ubuntu/Debian/CompileAgent. Section 4 only mentions a 'sanitizer' without describing any test-set-to-training-set deduplication (e.g., MinHash or exact-match on decompiled pseudo-code), even though MinHash dedup is described within the raw corpus. Without such a check, near-duplicates could explain part or all of the SOTA margin via memorization. Please add and report a test-set contamination analysis.
  3. [§5.1, Tables 4-5] The paper acknowledges that 'LLM-based methods have inherent randomness' but reports a single run with at most three format-error retries. No repeated runs, confidence intervals, or significance tests are given. Because the average margins against some baselines are small (e.g., 0.06 points on summarization in Table 4), the SOTA claim is not statistically grounded. Report variance over multiple seeds/runs and/or per-item bootstrap intervals.
  4. [§4 (Binary Code Summarization)] The summarization evaluator is an LLM-as-a-judge, but the paper gives no judge model, prompt, or human-agreement validation for this specific task. Since summarization scores enter the averaged SOTA and the reported margin there is negligible, an unvalidated judge could reverse the ranking. Provide human-correlation data or at least a judge ablation.
  5. [§4 (Variable Type Recovery)] Variable type recovery is evaluated only on hand-clustered basic types (e.g., __int64 == unsigned __int64) with qualifiers ignored, so the scores do not measure the full type-recovery problem advertised in the abstract. The coarse clustering likely inflates accuracy and makes the comparison to prior fine-grained type-recovery work non-comparable. Please report fine-grained metrics or at least clearly scope the claim.
minor comments (7)
  1. [§3.4.1, Eq. (2)] The informative score has an empirical constant β=25 and a default top-k=10; no sensitivity analysis or ablation is provided, so the reader cannot tell how sensitive results are to these choices.
  2. [Table 4] Several baseline cells are blank because the tools do not support the task; please mark these as 'not supported' rather than leaving them empty, and clarify how the average is computed across a non-uniform task set.
  3. [§4 (Test Dataset)] The phrase 'private environments' is ambiguous because the test binaries are compiled from public open-source repositories; specify what 'private' means (e.g., private compilation flags/scripts) and how this addresses exposure.
  4. [Tables 5 and 7] DeepSeek-R1 appears with different prompts (general vs general_wo_guide) in different tables; the paper should make explicit which configuration is used in each comparison and why the best configuration varies by table.
  5. [§3.3.3] The claim that the generator-discriminator pipeline has a 'success rate of more than 90%' is not accompanied by the number of attempts or the success rate for the Super-CoT loop, which is later said to be costly; please report these numbers.
  6. [References] References [85], [86], and [92] are incomplete (URL-only or mislabeled); in particular [92] is titled 'TRL' but points to Cursor. Please fix these entries.
  7. [Figure 6] There is an inconsistency between the caption/text referring to 'sub_1903@a1' and the example showing 'sub_1909'; correct the mismatch.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity by construction: ReCopilot's reported predictions are not defined in terms of its benchmark outputs, and its self-citations are background only; the main residual risk is unverified test/train separation, which is an evaluation-independence concern rather than circular reasoning.

full rationale

The paper's central claims are empirical rather than derived from equations whose outputs equal their inputs. Training data pairs decompiled pseudo-code with source-level ground truth recovered through DWARF debug information (Section 3.3.1), while the benchmark independently compiles 16 open-source projects and scores function-name and variable recovery against source-level ground truth (Section 4); no parameter is fitted to the benchmark and then reported as a prediction. The CoT supervision is produced by a generator-discriminator loop that deliberately omits direct ground-truth citations (Section 3.3.3), and the DPO pairs come from failed retry generations, so the supervision signal is not the evaluation target renamed. Self-citations [19], [25], and [31] support only background or related-work claims; the premise that general LLMs underperform on stripped binaries is independently re-established by the paper's own DeepSeek-V3/R1 evaluations in Table 5. The one genuine validity risk is leakage: Section 3.3.1 collects 101M functions from open-source package repositories, and Section 4 selects test functions from open-source projects in overlapping domains, with only an undescribed sanitizer asserted to 'prevent data leakage from our training dataset.' That is an unverified independence assumption about benchmark cleanliness, not a circular reduction of the model's output to its training input by construction, so it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard domain assumptions about decompiler fidelity, debug-info-based ground truth, and the validity of automatic metrics. No new physical or mathematical entities are postulated. The free parameters are prompt-building and training-configuration choices that are not swept or fitted with error bars, so their influence on the reported 13% margin is unknown.

free parameters (5)
  • beta for informative score S(f) = 25
    Empirically set in Eq. 2 to balance string density; directly controls which context functions are selected, and thus prompt content, for all ReCopilot evaluations.
  • top-k context functions = 10 (default)
    Configurable prompt-building parameter in Section 3.4.1; affects context length and model input. No sensitivity analysis is reported.
  • data-flow trace depth = 1 hop for callees and callers (default)
    Evaluation default in Section 5.1; deeper traces would change the prompts and, potentially, the measured performance.
  • maximum output length = 16K tokens (eval)
    Evaluation setting in Section 5.1; a longer budget could change how often long reasoning or JSON predictions succeed.
  • pretraining mixture ratio = 60:25:15 binary/code/text
    Chosen by reference to prior work [55,56], not derived from task performance; the paper itself reports general-benchmark degradation from CPT.
assumptions (5)
  • domain assumption Decompiled pseudo-code from IDA Pro preserves enough semantic information for recovering source-level symbols and summaries.
    The entire dataset, prompts, and benchmark are built on pseudo-code; any information loss at decompilation limits all tasks. See Section 2 and Figure 1.
  • domain assumption DWARF debug information accurately links stripped binary functions to their source-level counterparts for ground-truth extraction.
    Ground truth for CPT and SFT is obtained by using debug info file paths and line numbers in Section 3.3.1; errors here directly corrupt labels.
  • domain assumption LLM-as-a-judge scores for summarization align with human preferences.
    The summarization evaluator uses a binary LLM judgment on four dimensions, relying on prior studies in Section 4 rather than a human validation on this benchmark.
  • domain assumption The private test set of 1,038 functions from 16 projects is leakage-free and representative of real binary-analysis workloads.
    The authors state that sanitizers and private environments make leakage unlikely in Section 4, but no deduplication or overlap analysis against baseline training corpora is provided.
  • domain assumption ROUGE, clustered-type exact match, CodeBLEU, and struct-boundary F1 are valid proxies for the practical value of recovered symbols.
    These metrics measure token or memory overlap rather than analyst usefulness; the paper does not validate them against human reverse-engineering judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReCopilot: Reverse Engineering Copilot in Binary Analysis." pith.science (2026). https://pith.science/paper/7DRKA2LH

@misc{pith2026250516366,
  author       = {Pith},
  title        = {Pith review of: ReCopilot: Reverse Engineering Copilot in Binary Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7DRKA2LH}},
  note         = {Machine review of arXiv:2505.16366}
}
read the original abstract

Binary analysis plays a pivotal role in security domains such as malware detection and vulnerability discovery, yet it remains labor-intensive and heavily reliant on expert knowledge. General-purpose large language models (LLMs) perform well in programming analysis on source code, while binaryspecific LLMs are underexplored. In this work, we present ReCopilot, an expert LLM designed for binary analysis tasks. ReCopilot integrates binary code knowledge through a meticulously constructed dataset, encompassing continue pretraining (CPT), supervised fine-tuning (SFT), and direct preference optimization (DPO) stages. It leverages variable data flow and call graph to enhance context awareness and employs test-time scaling to improve reasoning capabilities. Evaluations on a comprehensive binary analysis benchmark demonstrate that ReCopilot achieves state-of-the-art performance in tasks such as function name recovery and variable type inference on the decompiled pseudo code, outperforming both existing tools and LLMs by 13%. Our findings highlight the effectiveness of domain-specific training and context enhancement, while also revealing challenges in building super long chain-of-thought. ReCopilot represents a significant step toward automating binary analysis with interpretable and scalable AI assistance in this domain.

Figures

Figures reproduced from arXiv: 2505.16366 by the authors.

Figure 1
Figure 1. An example of decompiled pseudo code and the corresponding source code from a real-world [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of the ReCopilot model building. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example for demonstrating the pretraining data format and inner-shuffling. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The input-output template designed for binary analysis tasks in ReCopilot. We carefully defined a general input-output template for the binary analysis tasks, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: An overview of the generator-discriminator framework for building supervised fine-tuning (SFT) data of binary [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An example of prompt with context enhancement by ReCopilot. The variable aliases yield from data flow [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Inference rules used in data flow analysis for ReCopilot. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Our goal was to create an automatic and extensible evaluation pipeline for the LLM-based reverse engineering [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 40 canonical work pages

  1. [1]

    "ida pro"

    Hex-RaysSA. "ida pro". https://www.hex-rays.com/products/ida, 2025. 16 ReCopilot - August 10, 2025

    2025

  2. [2]

    "ghidra"

    NationalSecurityAgency. "ghidra". https://github.com/NationalSecurityAgency/ghidra, 2025

    2025

  3. [3]

    On automated assistants for software development: The role of llms

    Mira Leung and Gail Murphy. On automated assistants for software development: The role of llms. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 1737–1741. IEEE, 2023

    2023

  4. [4]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE, 2022

    2022

  5. [5]

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y . Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Ga...

    arXiv 2024

  6. [6]

    Impact of code language models on automated program repair

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 1430–1442. IEEE, 2023

    2023

  7. [7]

    How far have we gone in vulnerability detection using large language models

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420, 2023

    arXiv 2023

  8. [8]

    Large language models are few-shot testers: Exploring llm-based general bug reproduction

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 2312–2323. IEEE, 2023

    2023

Show all 94 references
  1. [9]

    Competitive programming with large reasoning models, 2025

    OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contrib...

    2025 arXiv

  2. [10]

    LLM4Decompile: Decompiling binary code with large language models

    Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. LLM4Decompile: Decompiling binary code with large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 3473–3...

    2024 doi

  3. [11]

    Resym: Harnessing llms to recover variable and data structure symbols from stripped binaries

    Danning Xie, Zhuo Zhang, Nan Jiang, Xiangzhe Xu, Lin Tan, and Xiangyu Zhang. Resym: Harnessing llms to recover variable and data structure symbols from stripped binaries. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS) , 2024

    2024

  4. [12]

    len or index or count, anything but v1

    Kuntal Kumar Pal, Ati Priya Bajaj, Pratyay Banerjee, Audrey Dutcher, Mutsumi Nakamura, Zion Leonahenahe Basque, Himanshu Gupta, Saurabh Arjun Sawant, Ujjwala Anantheswaran, Yan Shoshitaishvili, Adam Doupé, Chitta Baral, and Ruoyu Wang. "len or index or count, anything but v1":...

    2024

  5. [13]

    URL: https://github.com/kokke/tiny-AES-c

    kokke, 2025. URL: https://github.com/kokke/tiny-AES-c

    2025

  6. [14]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL: https://openrev...

    2023

  7. [15]

    Evaluating language models for efficient code generation

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling , 2024. URL: https:// openreview.net/forum?id=IBCBMeAhmC

    2024

  8. [16]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  9. [17]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL: https://arxiv.org/abs/ 2310.06770, arXiv:2310.06770

    2024 arXiv

  10. [18]

    Binary code summarization: Benchmarking chatgpt/gpt- 4 and other large language models

    Xin Jin, Jonathan Larson, Weiwei Yang, and Zhiqiang Lin. Binary code summarization: Benchmarking chatgpt/gpt- 4 and other large language models. arXiv preprint arXiv:2312.09601, 2023. 17 ReCopilot - August 10, 2025

    2023 arXiv

  11. [19]

    How far have we gone in stripped binary code understanding using large language models

    Xiuwei Shang, Shaoyin Cheng, Guoqiang Chen, Yanming Zhang, Li Hu, Xiao Yu, Gangyang Li, Weiming Zhang, and Nenghai Yu. How far have we gone in stripped binary code understanding using large language models. arXiv e-prints, pages arXiv–2404, 2024

    2024

  12. [20]

    Enhancing reverse engineering: Investigating and benchmarking large language models for vulnerability analysis in decompiled binaries, 2024

    Dylan Manuel, Nafis Tanveer Islam, Joseph Khoury, Ana Nunez, Elias Bou-Harb, and Peyman Najafirad. Enhancing reverse engineering: Investigating and benchmarking large language models for vulnerability analysis in decompiled binaries, 2024. URL: https://arxiv.org/abs/2411.04981...

    2024 arXiv

  13. [21]

    Debin: Predicting debug information in stripped binaries

    Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin Vechev. Debin: Predicting debug information in stripped binaries. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS ’18, page 1667–1680, 2018. doi:10.1145/3243734.3243866

    2018

  14. [22]

    Neural reverse engineering of stripped binaries using augmented control flow graphs

    Yaniv David, Uri Alon, and Eran Yahav. Neural reverse engineering of stripped binaries using augmented control flow graphs. Proc. ACM Program. Lang., 4(OOPSLA), November 2020. doi:10.1145/3428293

    2020 doi

  15. [23]

    A lightweight framework for function name reassignment based on large-scale stripped binaries

    Han Gao, Shaoyin Cheng, Yinxing Xue, and Weiming Zhang. A lightweight framework for function name reassignment based on large-scale stripped binaries. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 607–619, 2021

    2021

  16. [24]

    Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings

    Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , pages 1631–1645, 2022

    2022

  17. [25]

    Investigating neural-based function name reassignment from the perspective of binary code representation

    Guoqiang Chen, Han Gao, Jie Zhang, Yanru He, Shaoyin Cheng, and Weiming Zhang. Investigating neural-based function name reassignment from the perspective of binary code representation. In2023 20th Annual International Conference on Privacy, Security and Trust (PST) , pages 1–1...

    2023

  18. [26]

    Xfl: Naming functions in binaries with extreme multi-label learning

    James Patrick-Evans, Moritz Dannehl, and Johannes Kinder. Xfl: Naming functions in binaries with extreme multi-label learning. In 2023 IEEE Symposium on Security and Privacy (SP) , pages 2375–2390. IEEE, 2023

    2023

  19. [27]

    Enhancing function name prediction using votes-based name tokenization and multi-task learning

    Xiaoling Zhang, Zhengzi Xu, Shouguo Yang, Zhi Li, Zhiqiang Shi, and Limin Sun. Enhancing function name prediction using votes-based name tokenization and multi-task learning. Proc. ACM Softw. Eng., 1(FSE), July

  20. [28]

    In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA, August 2022

    Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security 22), Boston, MA, August 2022. USENIX Association. URL: https://www.usenix.org/ conference/usenixsecurity22/presentation/chen-qibin

    2022

  21. [29]

    Direct: A transformer-based model for decompiled identifier renaming

    Vikram Nitin, Anthony Saieva, Baishakhi Ray, and Gail Kaiser. Direct: A transformer-based model for decompiled identifier renaming. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 48–57, 2021

    2021

  22. [30]

    Lacomis, P

    J. Lacomis, P. Yin, E. Schwartz, M. Allamanis, C. Le Goues, G. Neubig, and B. Vasilescu. Dire: A neural approach to decompiled identifier naming. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 628–639, Nov 2019. doi:10.1109/ASE.2019.00064

    2019

  23. [31]

    Hext5: Unified pre-training for stripped binary code information inference

    Jiaqi Xiong, Guoqiang Chen, Kejiang Chen, Han Gao, Shaoyin Cheng, and Weiming Zhang. Hext5: Unified pre-training for stripped binary code information inference. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 774–786. IEEE, 2023

    2023

  24. [32]

    Stateformer: Fine-grained type recovery from binaries using generative state modeling

    Kexin Pei, Jonas Guan, Matthew Broughton, Zhongtian Chen, Songchen Yao, David Williams-King, Vikas Ummadisetty, Junfeng Yang, Baishakhi Ray, and Suman Jana. Stateformer: Fine-grained type recovery from binaries using generative state modeling. In Proceedings of the 29th ACM Jo...

    2021

  25. [33]

    Typesqueezer: When static recovery of function signatures for binary executables meets dynamic analysis

    Ziyi Lin, Jinku Li, Bowen Li, Haoyu Ma, Debin Gao, and Jianfeng Ma. Typesqueezer: When static recovery of function signatures for binary executables meets dynamic analysis. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , CCS ’23, page...

    2023

  26. [34]

    TYGR: Type Inference on Stripped Binaries using Graph Neural Networks

    Chang Zhu, Ziyang Li, Anton Xue, Ati Priya Bajaj, Wil Gibbs, Yibo Liu, Rajeev Alur, Tiffany Bao, Hanjun Dai, Adam Doupé, et al. TYGR: Type Inference on Stripped Binaries using Graph Neural Networks. In 33rd USENIX Security Symposium (USENIX Security 24) , pages 4283–4300, 2024

    2024

  27. [35]

    Typefsl: Type prediction from binaries via inter-procedural data-flow analysis and few-shot learning

    Zirui Song, Yutong Zhou, Shuaike Dong, Ke Zhang, and Kehuan Zhang. Typefsl: Type prediction from binaries via inter-procedural data-flow analysis and few-shot learning. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1269–1...

    2024

  28. [36]

    Extending source code pre-trained language models to summarise decompiled binarie

    Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie van Deursen. Extending source code pre-trained language models to summarise decompiled binarie. In 2023 IEEE 18 ReCopilot - August 10, 2025 International Conference on Software Analys...

    2023

  29. [37]

    Abhinav Jauhri

    Aaron Grattafiori, Abhimanyu Dubey, and etc. Abhinav Jauhri. The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783, arXiv:2407.21783

    2024 arXiv

  30. [38]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and etc. Deepseek-v3 technical report, 2025. URL: https: //arxiv.org/abs/2412.19437, arXiv:2412.19437

    2025 arXiv

  31. [39]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  32. [40]

    Openai o1 system card, 2024

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, and Ahmed El-Kishky etc. Openai o1 system card, 2024. URL: https://arxiv.org/abs/2412.16720, arXiv:2412.16720

    2024 arXiv

  33. [41]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, and etc. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL: https://arxiv.org/abs/2501.12948, arXiv:2501.12948

    2025 arXiv

  34. [42]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL: https://arxiv.org/ abs/2305.18290, arXiv:2305.18290

    2024 arXiv

  35. [43]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  36. [44]

    Arch linux packages, 2025

    archlinux.org. Arch linux packages, 2025. URL: https://archlinux.org/packages/

    2025

  37. [45]

    Ubuntu packages, 2025

    ubuntu.com. Ubuntu packages, 2025. URL: https://packages.ubuntu.com/

    2025

  38. [46]

    Debian packages, 2025

    debian.org. Debian packages, 2025. URL: https://www.debian.org/distrib/packages

    2025

  39. [47]

    Compileagent: Automated real-world repo-level compilation with tool-integrated LLM-based agent system

    Anonymous. Compileagent: Automated real-world repo-level compilation with tool-integrated LLM-based agent system. In Submitted to ACL Rolling Review - December 2024 , 2025. under review. URL: https: //openreview.net/forum?id=1kic2XYZiR

    2024

  40. [48]

    tree-sitter

    tree sitter. "tree-sitter". https://github.com/tree-sitter/tree-sitter , 2025

    2025

  41. [49]

    A.Z. Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171) , pages 21–29, 1997. doi:10.1109/SEQUEN.1997.666900

    1997

  42. [50]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  43. [51]

    Maurice Weber, Daniel Y . Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Red...

    2024

  44. [52]

    Wikimedia downloads, 2025

    Wikimedia Foundation. Wikimedia downloads, 2025. URL: https://dumps.wikimedia.org

    2025

  45. [53]

    stackoverflow-posts, 2025

    mikex86. stackoverflow-posts, 2025. URL: https://huggingface.co/datasets/mikex86/ stackoverflow-posts

    2025

  46. [54]

    security-paper-datasets, 2025

    clouditera. security-paper-datasets, 2025. URL: https://huggingface.co/datasets/clouditera/ security-paper-datasets. 19 ReCopilot - August 10, 2025

    2025

  47. [55]

    CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models

    Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, and Fei Tan. CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Method...

    2024 doi

  48. [56]

    Qwen2.5- coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 arXiv

  49. [57]

    Scaling synthetic data creation with 1,000,000,000 personas, 2024

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas, 2024. URL: https://arxiv.org/abs/2406.20094, arXiv:2406.20094

    2024 arXiv

  50. [58]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2025 arXiv

  51. [59]

    Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code

    Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code. In The Thirteenth International Conference on Learning Representations , 202...

    2025

  52. [60]

    Cross-task generalization abilities of large language models

    Qinyuan Ye. Cross-task generalization abilities of large language models. In Yang (Trista) Cao, Isabel Papadim- itriou, Anaelia Ovalle, Marcos Zampieri, Francis Ferraro, and Swabha Swayamdipta, editors, Proceedings of the 2024 Conference of the North American Chapter of the As...

    2024

  53. [61]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL: https: //huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  54. [62]

    URL: https://aclanthology.org/2024.naacl-srw.27/, doi:10.18653/v1/2024.naacl-srw.27

    Association for Computational Linguistics. URL: https://aclanthology.org/2024.naacl-srw.27/, doi:10.18653/v1/2024.naacl-srw.27

    2024 doi

  55. [63]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024

    2024 arXiv

  56. [64]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. UR...

    2024

  57. [65]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  58. [66]

    Open-Source-O1. Open-O1. https://huggingface.co/datasets/O1-OPEN/OpenO1-SFT, 2024

    2024

  59. [67]

    semgrep. Semgrep. https://github.com/semgrep/semgrep, 2025

    2025

  60. [68]

    Joern: The Bug Hunter’s Workbench, January 2024

    joern.io. Joern: The Bug Hunter’s Workbench, January 2024. URL: https://github.com/joernio/joern

    2024

  61. [69]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  62. [70]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74–81, Barcelona, Spain, 2004. Association for Computational Linguistics. URL: https://aclanthology.org/W04-1013

    2004

  63. [71]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.com/open-compass/opencompass, 2023

    2023

  64. [72]

    A survey on llm-as-a-judge, 2025

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL: https://arxiv.org/abs/2411.15594, arXiv:...

    2025 arXiv

  65. [73]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  66. [74]

    Codebleu: a method for automatic evaluation of code synthesis

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambro- sio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020. 20 ReCopilot - August 10, 2025

    2009 arXiv

  67. [75]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL: https: //arxiv.org/abs/2311.12022, arXiv:2311.12022

    2023 arXiv

  68. [76]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL: https://arxiv.org/abs/2009.03300, arXiv:2009.03300

    2021 arXiv

  69. [77]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL: https://arxiv.org/abs/2311. 07911, arXiv:2311.07911

    2023 arXiv

  70. [78]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL: https://arxiv.org/abs/2108.07732, arXiv:2108.07732

    2021 arXiv

  71. [79]

    TRL: Transformer Reinforcement Learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforcement Learning. URL: https://github.com/huggingface/trl

  72. [80]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural...

    2020

  73. [81]

    URL: https://www.deepspeed.ai/

    DeepSpeed, 2025. URL: https://www.deepspeed.ai/

    2025

  74. [82]

    LlamaFactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ...

    2024 doi

  75. [83]

    Wpechatgpt

    WPeace-HcH. Wpechatgpt. https://github.com/WPeace-HcH/WPeChatGPT, 2023

    2023

  76. [84]

    JusticeRage. Gepetto. https://github.com/JusticeRage/Gepetto, 2022

    2022

  77. [85]

    https://mlm01.com, 2024

    Mlm. https://mlm01.com, 2024

    2024

  78. [86]

    atredispartners. aidapal. https://github.com/atredispartners/aidapal, 2024

    2024

  79. [87]

    binary ninja

    Vector35. "binary ninja". https://binary.ninja/, 2025

    2025

  80. [88]

    reverser_ai

    mrphrazer. reverser_ai. https://github.com/mrphrazer/reverser_ai, 2024

    2024

  81. [89]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL: https://arxiv.org/abs/2402.03300, arXiv:2402.03300

    2024 arXiv

  82. [90]

    URL: https://evalplus.github.io/leaderboard.html

    EvalPlus, 2024. URL: https://evalplus.github.io/leaderboard.html

    2024

  83. [91]

    Introducing deep research

    OpenAI. Introducing deep research. https://openai.com/index/introducing-deep-research/ , 2025

    2025

  84. [92]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL: https: //arxiv.org/abs/2001.08361, arXiv:2001.08361

    2020 arXiv

  85. [94]

    TRL: Transformer Reinforcement Learning, 2025

    Anysphere Inc. TRL: Transformer Reinforcement Learning, 2025. URL: https://www.cursor.com. 21

    2025

  86. [2024]

    doi:10.1109/SP54263.2024.00152

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.