Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

All You Need Is A Fuzzing Brain: An LLM-Powered System for Automated Vulnerability Detection and Patching

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Cyber Reasoning System using parallel LLM-driven fuzzing can find 28 real-world vulnerabilities—six zero-days—and patch 14 during AIxCC finals.

desk verdict A credible, useful AIxCC systems report with real open-source artifacts and a new leaderboard, but the headline 28/14/6 counts need better sourcing before they can be taken at face value. read the letter →

arxiv 2509.07225 v1 pith:IWSXNJ2Y submitted 2025-09-08 cs.CR

Ze Sheng , Qingxiao Xu , Jianwei Huang , Matthew Woodcock , Heqing Huang , Alastair F. Donaldson , Guofei Gu , Jeff Huang This is my paper · ORCID
classification cs.CR
keywords LLM-poweredfuzzingcyberreasoningsystemvulnerabilitydetectionautomaticpatchingproofsofstaticanalysiszero-daydiscoveryAIxCC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FuzzingBrain is a Cyber Reasoning System that combines traditional fuzzing with many parallel LLM-driven strategies to find and fix security vulnerabilities in real open-source C and Java projects. The paper's central claim is that this hybrid, massively parallel design let the team autonomously discover 28 vulnerabilities—including six previously unknown zero-days—and generate 14 accepted patches during the final round of the AIxCC competition, placing fourth among seven finalists. The LLM-based strategies generated nearly all of the crash-triggering inputs, while the traditional fuzzer contributed only one or two. The paper also details a silent patch-validation failure in Section 8.5 that may mean some patches were submitted without verification, and it introduces a public leaderboard that runs the same system with a single LLM per trial to compare models on vulnerability detection and patching.

What carries the argument

The workhorse is the iterative LLM dialogue loop. Each strategy runs as an isolated process that sends code context—commit diffs, crash logs, reachable functions, or coverage data—to an LLM, executes the model's generated Python to produce binary inputs or patches, and turns the execution results into the next user message. A static-analysis layer computed from the target project supplies function metadata, reachability, and call paths, narrowing which functions the LLM is asked to analyze. The submission layer uses canonical signatures plus LLM consensus to deduplicate POVs, and caps patch submissions per vulnerability to preserve the competition's accuracy multiplier.

What would settle it

Re-run FuzzingBrain's submitted patch set from the final round against the union of all teams' POVs and the functionality tests, with the Section 8.5 validation bug fixed; if the number of patches that actually eliminate the POVs falls below 14, the central patching claim is overstated.

Watch

Extended reading notes

Core claim

The paper asserts that a Cyber Reasoning System whose core loop is iterative dialogue with LLMs—each attempt producing a candidate input or patch, executing it against a sanitizer-instrumented harness, and feeding compiler, coverage, and crash output back into the next prompt—can discover and patch real vulnerabilities under competition constraints. This is supported by the system's ten POV-generation strategies and thirteen patching strategies, unified by a static-analysis service that prunes the search space to reachable functions and call paths, and a submission service that deduplicates, validates, and bundles findings. The headline evidence is the final-round tally: 28 discovered vulner

Load-bearing premise

The headline numbers assume the competition evaluator's POV and patch validations are accurate, but Section 8.5 reports that a missing parameter made the system's own validation subprocesses crash silently, so many patches were submitted without being checked against known POVs.

Editorial extensions

If this is right

  • LLM-based strategies can be the main engine of a Cyber Reasoning System: almost every POV in the described deployment came from LLM-driven inputs rather than from the traditional fuzzer.
  • The feedback loop of generate, execute, and re-prompt with coverage and sanitizer output is a transferable recipe for automated vulnerability detection and patching in both C/C++ and Java.
  • Because the complete system is released as open source, other teams can reuse the architecture for further competition or benchmark work instead of rebuilding infrastructure.
  • The public leaderboard provides a reproducible single-model, one-hour evaluation on AIxCC-derived challenges, enabling direct comparison of LLMs on real-world detection and patching tasks.
  • The Section 8.5 validation failure shows that patch pipelines need explicit status reporting and fail-safe checks; silently crashing validation can flood a submission service with unverified patches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LLM-generated inputs really produced nearly all POVs, the economic balance of fuzzing research shifts toward prompt and context design rather than mutator engineering; this is not a controlled comparison, since the deployment capped API credits and skipped some sanitizers.
  • The fact that the system runs 23 strategies over five frontier models suggests an ensemble effect: the same underlying model in different analysis contexts may matter more than any single model, a hypothesis the leaderboard could test by ablating strategies one at a time.
  • A clean re-run with the Section 8.5 bug fixed, on the same challenges, would give a direct estimate of how much the silent validation failure inflated the reported 14-patch success count.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes FuzzingBrain, an LLM-powered Cyber Reasoning System developed for DARPA's AIxCC competition. The authors report that their system placed fourth among seven finalists, autonomously discovered 28 vulnerabilities (including six previously unknown zero-days) in real-world C and Java projects, and successfully patched 14 of them. The paper gives a detailed account of the system architecture, its ten LLM-based POV-generation strategies, thirteen patching strategies, static-analysis pipeline, SARIF handling, performance optimizations, and lessons learned. It also introduces a public leaderboard, based on AIxCC exhibition challenges, for comparing LLMs on vulnerability detection and patching. The full system is open-sourced.

Significance. If the headline claims hold, this is a valuable data point for LLM-driven vulnerability discovery and repair: a deployed, large-scale CRS achieving competitive results on real-world code, with a substantial open-source release and a reproducible leaderboard. The paper's engineering detail is a genuine strength, especially the description of iterative LLM feedback loops, corpus sharing between LLM-based fuzzing and libFuzzer, and static-analysis integration. However, the empirical claims are not adequately evidenced within the manuscript. The paper's own Sections 8.4 and 8.5 document a silent patch-validation failure, loss of all logs, and a multi-day period with no submissions. Since the paper does not cite the official AIxCC scoreboard or provide a per-vulnerability breakdown, the central counts of 28, 6, and 14 cannot be independently verified from the manuscript. This makes the headline claims load-bearing but currently under-supported.

major comments (4)
  1. [Abstract; §8.5; §8.4] The central claim of 'successfully patched 14' vulnerabilities is not adequately substantiated. Section 8.5 states that a missing Python parameter caused all LLM-patching subprocesses to crash silently, so 'many patches were submitted without verification against known POVs,' and Section 8.4 states that all logs were lost after VM recycling. The paper does not cite the official AIxCC scoreboard and does not provide a per-vulnerability table. Please clarify whether the 14 patches were officially accepted by the competition evaluator, and add the scoreboard reference and a per-vulnerability breakdown (project, CVE/commit, POV status, patch acceptance status).
  2. [Abstract; §8.4] The claim of 'six previously unknown zero-days' is asserted without a list of the affected projects, CVE/commit identifiers, or organizer confirmation. Given the total log loss reported in §8.4, the zero-day status cannot be independently verified from the paper. Please provide an appendix table listing each zero-day (project, version, harness, sanitizer, POV trigger) and state explicitly how the 'previously unknown' status was established (e.g., organizer confirmation, CVE assignment, or absence from public databases).
  3. [§8.5] The statement that the missing-parameter bug caused 'a significant drop in accuracy' and that 'many patches were submitted without verification against known POVs' is in direct tension with the clean 14-patch success figure. The paper needs quantitative context: how many patch submissions were made total, how many were internally verified before submission, how many were accepted by the competition, and, of the 14 reported successful patches, how many were submitted before versus after the bug was introduced. Without this, a reader cannot tell whether the 14 figure reflects official validation or self-reported submissions that may not have been checked against all competing POVs.
  4. [§8.4] The paper reports that the system 'submitted nothing after Day 4' of the final round, with the root cause unknown because logs were lost. This is a material limitation for a paper whose primary evidence is competition performance. The manuscript should explicitly acknowledge that this silent failure may have reduced the final counts and that the system's reliability during the later phase of the competition is undocumented. This does not invalidate the architecture, but it is essential context for interpreting the reported results.
minor comments (5)
  1. [§1.1] Typo: 'AddressSanitizier' should be 'AddressSanitizer'.
  2. [§1.3] The scoring formula would benefit from a short worked example, and the quantities 'acc' and 'inacc' should be defined immediately before the formula.
  3. [§6.1] The claim that the bitcode generation pipeline succeeded for 'over 95% of source files' lacks a denominator. Specify the number of source files/projects tested so the success rate is meaningful.
  4. [References] Add a citation to the official AIxCC final scoreboard or results page; this is needed to corroborate the central competition counts.
  5. [§9] The leaderboard description would be clearer if it specified how patch validity is determined on the leaderboard (e.g., using the same OSS-Fuzz POV set as AIxCC) and whether static analysis results are shared across models or recomputed per model. This affects comparability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: competition results are externally evaluated; leaderboard is benchmark scoring, not a self-referential prediction.

full rationale

This paper is a systems and competition report rather than a derivation of predicted quantities from fitted inputs. The headline numbers (28 discovered vulnerabilities, 6 zero-days, 14 successfully patched, 4th place) are reported outcomes of the externally administered AIxCC evaluation, not consequences of a definitional identity or of parameters fitted to the same data. The scoring formulas in Section 1.3 are the competition's official rubric, used as an evaluation metric, not as a generative or predictive model. Section 9's FuzzingBrain Leaderboard reuses the AIxCC rubric to score LLMs on a fixed benchmark; this is a benchmark evaluation methodology, not a circular derivation, since the reported scores are measured outputs and the rubric does not define vulnerability existence or patch validity. There are no self-citations: the references are external tools (OSS-Fuzz, LLVM, CodeQL, SVF, Bear, Honggfuzz, AFL++) and the AIxCC program documents. The paper's own limitations sections do not indicate circularity: Section 8.4 reports loss of logs preventing postmortem analysis, and Section 8.5 reports a silent patch-validation pipeline failure causing many patches to be submitted without verification against known POVs. These are evidentiary and reproducibility concerns that weaken confidence in the reported patch count, but they do not make the paper's reasoning circular. No load-bearing step reduces, by the paper's own equations or by self-citation, to its own inputs. The honest finding is no significant circularity, score 0.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The ledger captures the engineering parameters chosen by hand and the domain assumptions the system relies on. No free parameters are fit to data in a statistical sense, but the system's reported effectiveness depends on these hand-tuned thresholds. No new physical or conceptual entities are postulated; the system is a composition of existing tools and LLM APIs.

free parameters (9)
  • call_path_depth_cap = 50 for C/C++, 10 for Java
    Set heuristically based on empirical observations from exhibition rounds (Section 2.4). Central to reachability-based targeting.
  • max_call_paths = 20
    Capped at 20 to avoid path explosion (Section 2.4).
  • function_source_cap_lines = 2000
    Modified-function context injection cap (Section 3.2).
  • xpatch_top_k = 5
    Default top-k functions for XPatch (Section 4.7).
  • xpatch_score_threshold = 7
    Only functions scored >=7 retained (Section 4.7).
  • corpus_age_cleanup_minutes = 10
    Files older than 10 minutes deleted from shared corpus (Section 2.2).
  • llm_fuzzing_time_budget = 60 minutes, reduced to 45 if POVs found
    Budget to control API spend (Section 7).
  • libfuzzer_time_budget = half of competition time
    CPU preservation policy (Section 7).
  • patch_submission_cap = 5 POV-based, 3 XPatch
    Tradeoff between success probability and accuracy multiplier (Section 7).
assumptions (6)
  • domain assumption AIxCC competition evaluation correctly validates POVs and patches
    The 28/14 claims rely on the external evaluator's counts; the authors themselves note many patches were submitted without internal verification (Section 8.5).
  • domain assumption OSS-Fuzz harnesses and sanitizer configurations are correct and complete for the target projects
    POV validity depends on harness correctness; the paper uses OSS-Fuzz utilities (Section 2.1).
  • domain assumption LLM-generated Python scripts to create x.bin are executed in a safe and reliable environment
    The base POV strategy extracts Python code from LLM responses and executes it (Section 3.1).
  • domain assumption Static analysis tools (SVF, CodeQL, Bear, LLVM) provide sound reachability and call-path results
    Reachability-based function ranking and call path prompts depend on these (Sections 3.3, 6). Note they trim bitcode, set timeouts, and return empty results on failure.
  • domain assumption LLM outputs are parseable and follow format instructions (JSON, Python code)
    All strategies rely on structured LLM outputs (Sections 3, 4.7).
  • domain assumption Coverage feedback from llvm-profdata/JaCoCo is accurate
    Coverage-guided feedback is used to refine POV attempts (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of All You Need Is A Fuzzing Brain: An LLM-Powered System for Automated Vulnerability Detection and Patching." pith.science (2026). https://pith.science/paper/IWSXNJ2Y

@misc{pith2026250907225,
  author       = {Pith},
  title        = {Pith review of: All You Need Is A Fuzzing Brain: An LLM-Powered System for Automated Vulnerability Detection and Patching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWSXNJ2Y}},
  note         = {Machine review of arXiv:2509.07225}
}
read the original abstract

Our team, All You Need Is A Fuzzing Brain, was one of seven finalists in DARPA's Artificial Intelligence Cyber Challenge (AIxCC), placing fourth in the final round. During the competition, we developed a Cyber Reasoning System (CRS) that autonomously discovered 28 security vulnerabilities - including six previously unknown zero-days - in real-world open-source C and Java projects, and successfully patched 14 of them. The complete CRS is open source at https://github.com/o2lab/afc-crs-all-you-need-is-a-fuzzing-brain. This paper provides a detailed technical description of our CRS, with an emphasis on its LLM-powered components and strategies. Building on AIxCC, we further introduce a public leaderboard for benchmarking state-of-the-art LLMs on vulnerability detection and patching tasks, derived from the AIxCC dataset. The leaderboard is available at https://o2lab.github.io/FuzzingBrain-Leaderboard/.

Figures

Figures reproduced from arXiv: 2509.07225 by the authors.

Figure 1
Figure 1. Overview of FuzzingBrain Architecture. contains metadata describing the challenge, e.g., a delta-scan or a full-scan of a target project), the CRS Web Service first builds the target project and its fuzzers using OSS-Fuzz utilities. Each fuzzer is an executable binary generated from a fuzzer harness instrumented with a sanitizer (AddressSanitizer, MemorySanitizer, or UndefinedBehaviorSanitizer for C/C++1 , and Jazze… view at source ↗
Figure 2
Figure 2. Task Distribution & Strategy Running [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Basic POV Generation Strategy Modified-Function Context Injection. Beyond the commit diff, we identify all modified files and functions and append the full source of each modified function to the prompt to provide precise context. To stay within model context limits and emphasize salient code, we cap the injected source at 2,000 lines per function. Call-Path–Based Analysis. This module queries the Static Analysis Se… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Basic Patch Generation Strategy (4) Function Rewrite: Replace the original function with the LLM-generated content. (5) Diff Creation: Generate a .diff file using Git differential tools. (6) Validation: Ensure compilation, execute POV tests, and run functionality tests…
Figure 5
Figure 5. Figure 5: SARIF Report-based Strategy command associated with each source file. We then process each compile command entry to produce bitcode for each source file, maintaining separate bitcode sets for fuzzer harnesses and project source files. For each fuzzer, all bitcode files…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PBFuzz: Agentic Directed Fuzzing for PoV Generation

    cs.CR 2025-12 conditional novelty 6.0 of 10

    An agentic fuzzing system lets LLM agents infer vulnerability constraints, encode them as parameter generators, and solve them with property-based testing; it triggered 57 Magma CVEs, 17 missed by other fuzzers.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    [n. d.]. The AFL++ fuzzing framework | AFLplusplus. https://aflplus.plus/. Ac- cessed: 2025-09-08

    work page 2025

  2. [2]

    [n. d.]. AI Cyber Challenge (AIxCC). https://aicyberchallenge.com/. Accessed: 2025-09-08

    work page 2025

  3. [3]

    [n. d.]. Bear: Build EAR. https://github.com/rizsotto/Bear. Accessed: 2025-09-08

    work page 2025

  4. [4]

    [n. d.]. CodeQL. https://codeql.github.com/. Accessed: 2025-09-08

    work page 2025

  5. [5]

    [n. d.]. Honggfuzz. https://github.com/google/honggfuzz. Accessed: 2025-09-08

    work page 2025

  6. [6]

    [n. d.]. LLVM Compiler Infrastructure. https://llvm.org/. Accessed: 2025-09-08

    work page 2025

  7. [7]

    [n. d.]. OSS-Fuzz. https://google.github.io/oss-fuzz/. Accessed: 2025-09-08

    work page 2025

  8. [8]

    [n. d.]. SVF: Static Value-Flow Analysis. https://svf-tools.github.io/SVF/. Ac- cessed: 2025-09-08

    work page 2025

Show all 10 references
  1. [9]

    AIxCC Final Competition Procedures and Scoring Guide

    2025. AIxCC Final Competition Procedures and Scoring Guide. https:// aicyberchallenge.com/final-competition-procedures-and-scoring-guide/. Ac- cessed: 2025-09-08

    2025

  2. [10]

    Fanning and Laurence J

    Michael C. Fanning and Laurence J. Golding. 2023.Static Analysis Re- sults Interchange Format (SARIF) Version 2.1.0 Errata 01. Technical Report. OASIS. https://docs.oasis-open.org/sarif/sarif/v2.1.0/errata01/os/sarif-v2.1.0- errata01-os.html OASIS Approved Errata

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.