REVIEW 11 cited by
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Deep Learning (DL) library bugs affect downstream DL applications, emphasizing the need for reliable systems. Generating valid input programs for fuzzing DL libraries is challenging due to the need for satisfying both language syntax/semantics and constraints for constructing valid computational graphs. Recently, the TitanFuzz work demonstrates that modern Large Language Models (LLMs) can be directly leveraged to implicitly learn all the constraints to generate valid DL programs for fuzzing. However, LLMs tend to generate ordinary programs following similar patterns seen in their massive training corpora, while fuzzing favors unusual inputs that cover edge cases or are unlikely to be manually produced. To fill this gap, this paper proposes FuzzGPT, the first technique to prime LLMs to synthesize unusual programs for fuzzing. FuzzGPT is built on the well-known hypothesis that historical bug-triggering programs may include rare/valuable code ingredients important for bug finding. Traditional techniques leveraging such historical information require intensive human efforts to design dedicated generators and ensure the validity of generated programs. FuzzGPT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (including fine-tuning and in-context learning), while being generalizable and applicable to challenging domains. While FuzzGPT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: Codex and CodeGen. Moreover, FuzzGPT also shows the potential of directly leveraging the instruct-following capability of the recent ChatGPT for effective fuzzing. Evaluation on two popular DL libraries (PyTorch and TensorFlow) shows that FuzzGPT can substantially outperform TitanFuzz, detecting 76 bugs, with 49 already confirmed as previously unknown bugs, including 11 high-priority bugs or security vulnerabilities.
Forward citations
Cited by 11 Pith papers
-
GapForge: Directed Compiler Fuzzing via Coverage-Gap Analysis
GapForge targets uncovered compiler code regions by having an LLM infer what program structures and compiler flags reach them, improving GCC/LLVM line coverage and finding 12 real failures.
-
When Fuzzing Meets Understanding: LLM-Driven Semantic Test Generation for RTL Verification
LLM-guided dual-stage hardware fuzzing raises condition coverage by 5.8 points and bug detection by 21.1 points over the strongest baseline on three open-source CPUs.
-
SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models
SAGE generates gray-box game regression tests via LLM-guided RL, compresses them with Pareto optimization, and prioritizes them via LLM-analyzed update logs, beating automated baselines on unique-bug detection at lowe...
-
Benchmarking LLMs for Unit Test Generation from Real-World Functions
A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.
-
LLMCup: Ranking-Enhanced Comment Updating with LLMs
LLMCup combines multi-prompt LLM comment generation with a learned ranker, achieving 0.383 exact-match accuracy on the CUP test set versus 0.257 for HebCup and 0.177 for CUP.
-
Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis
An empirical study of 38,742 issue reports and 19 interviews produces a 20-theme, 75-sub-theme taxonomy of LLM-centric framework challenges and five recommendations.
-
The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries
API misuse is the dominant root cause of bugs in HuggingFace Transformers and vLLM, and most bugs are missed by existing tests because of missing drivers, missing cases, and weak oracles.
-
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
XAMT matches functionally equivalent APIs across five deep learning frameworks and uses variance-guided differential testing to detect 17 bugs, 12 confirmed, that intra-framework tests miss.
-
MultiFuzz: A Dense Retrieval-based Multi-Agent System for Network Protocol Fuzzing
MultiFuzz combines retrieval-augmented generation and multiple LLM agents within the ChatAFL protocol fuzzer, reporting marginal and statistically unsupported gains in branch coverage and state exploration for RTSP.
-
Pixels to Play: A Foundation Model for 3D Gameplay
Pixels2Play-0.1 is a decoder-only transformer trained via behavior cloning and inverse-dynamics-imputed actions to play 3D games from pixels, with only qualitative results reported.
-
Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.
Discussion (0). Continue with ORCID to comment.