REVIEW 4 cited by
Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of large language models (LLMs) for binary code comprehension. To this end, we present BinSum, a comprehensive benchmark and dataset of over 557K binary functions and introduce a novel method for prompt synthesis and optimization. To more accurately gauge LLM performance, we also propose a new semantic similarity metric that surpasses traditional exact-match approaches. Our extensive evaluation of prominent LLMs, including ChatGPT, GPT-4, Llama 2, and Code Llama, reveals 10 pivotal insights. This evaluation generates 4 billion inference tokens, incurred a total expense of 11,418 US dollars and 873 NVIDIA A100 GPU hours. Our findings highlight both the transformative potential of LLMs in this field and the challenges yet to be overcome.
Forward citations
Cited by 4 Pith papers
-
JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation
On a new execution-verifiable benchmark, LLMs simplify obfuscated JavaScript far better than rule-based tools but preserve syntax and runtime behavior much less reliably.
-
Adapting Online Customer Reviews for Blind Users: A Case Study of Restaurant Reviews
QuickCue, an LLM-powered browser extension that reorganizes restaurant reviews into aspect-sentiment summaries, significantly improved usability and reduced workload for blind screen reader users in a 10-user study.
-
ReCopilot: Reverse Engineering Copilot in Binary Analysis
ReCopilot is a 7B binary-analysis LLM that reports 13% higher average scores than existing tools on a new six-task benchmark, but the benchmark and artifacts are not released.
-
Idioms: Neural Decompilation With Joint Code and Type Definition Prediction
Finetuned LLMs that jointly generate decompiled C code and user-defined type definitions outperform prior neural decompilers on ExeBench and the new Realtype dataset.
Discussion (0). Continue with ORCID to comment.