{"id":"4bcc9260-9535-48a2-a83f-f9b98239bc58","arxiv_id":"2601.17471","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An orchestrated ensemble of LLM agents with patch-based deduplication patched 31 of 43 confirmed vulnerabilities in the AIxCC final and 84 of 92 in an internal benchmark.","lead":"PatchIsland uses a team of diverse AI agents to automatically patch security bugs discovered by continuous fuzzing. In DARPA's AIxCC final it produced the most accepted patches of any team — 31 of 43 confirmed vulnerabilities — including a pdfbox fix matching the maintainer's own.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3, the sole basis for the headline 84/92 and the 29.3% Buttercup margin, is internally inconsistent: the PatchIsland column sums to 57, Buttercup sums to 84, and per-row percentages do not match fractions, so the internal result is unverifiable as printed.","rationale":"The Reader's CONDITIONAL verdict is appropriate. My stress-test found a more concrete weak point than the one named as the weakest assumption: the internal benchmark table contradicts itself before any question of semantic correctness. This is load-bearing because the abstract and baseline comparisons are built on that number. I credit the official 31/43 AIxCC result and the pdfbox case as independent evidence of real capability, which is why REJECT would be too strong. But the paper's internal quantitative claims cannot be accepted as written. If the authors supply the per-PoV matrix and the table repairs to 84/92, the condition would be satisfied; otherwise the internal claim should be withdrawn or relabeled. I mark agreement as partial because the Reader's formal weakest_assumption focused on plausible-patch fragility, while the most immediate blocker is the internal arithmetic inconsistency, though the Reader's rationale did note the same table problems.","tokens_in":23415,"tokens_out":10941,"duration_ms":100171,"concrete_test":"Obtain from the authors the raw per-PoV success vector for each system used in Table 3 (one row per 92 vulnerability IDs, with columns RoboDuck/Buttercup/PatchIsland and a plausible-patch pass/fail), then recompute the per-project and total sums. If the corrected PatchIsland total is 84 and baseline totals are 16 and 57, the printed table is a formatting error and the claim stands; if instead PatchIsland totals 57 or the distribution differs, revise the headline and the 29.3% margin. Publishing this matrix (or the code/logs) would settle the discrepancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 is the only quantitative support for the abstract's 'repaired 84 of 92' and the §6.1 'outperforms Buttercup by 29.3%' claim, but its arithmetic cannot be reproduced. Summing the PatchIsland column across the 17 project rows gives 7+4+3+0+2+2+1+2+5+0+0+6+4+3+7+10+1 = 57, not the printed 84. The Buttercup column sums to 84, and the RoboDuck column to 24, while the totals row prints 16/57/84 and the text says RoboDuck/Buttercup patched 57/16. Many percentages are attached to the wrong fractions (e.g., 10/11 shown as 63.6%, 5/7 as 57.1%, 5/5 as 40.0%). Table 4/8's best-of-N column also reports 87/92, which the text labels an upper bound, so it is unclear which number corresponds to the deployed FP2 system. Because no code, logs, or per-PoV outcome matrix are released, the reader cannot tell whether this is a typesetting/column-swap error or a real miscount. The official 31/43 AIxCC result is externally validated and remains credible, but the paper's headline internal comparison and the baseline-margin claim rest on this unverified table. The paper's own §7 caveat about plausible-patch correctness is a separate, second-order concern; the first-order problem is that the raw plausible-patch counts themselves do not add up.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PatchIsland is a continuous vulnerability repair (CVR) system that orchestrates multiple LLM agents in a fuzzing pipeline. It introduces an ensemble-of-agents design, two-phase patch-based deduplication, and an FP2 (FCFS, Preference-based, Provider-aware) orchestration layer. The paper evaluates the system internally on 92 AIxCC vulnerabilities, claiming 84/92 repaired and a 29.3% improvement over the Buttercup baseline, and reports results from the official AIxCC final competition, where PatchIsland generated 31 verified patches out of 43 vulnerabilities (72.1%). The authors also present a postmortem analysis of the competition, including a 0-day pdfbox patch identical to the maintainer's fix.","tokens_in":23787,"tokens_out":6538,"duration_ms":64225,"significance":"If the results hold, PatchIsland is a meaningful step toward practical, fully autonomous vulnerability repair in continuous fuzzing settings. The official AIxCC final result (31/43, manually validated by organizers) is a strong external validation, and the reported pdfbox 0-day patch matching the upstream fix demonstrates real-world impact. The paper's system-design contributions — ensemble orchestration, two-phase deduplication, and a reusable agent framework — are relevant to the security/systems community. However, the internal evaluation table contains serious arithmetic inconsistencies, and the absence of released artifacts prevents independent verification of the headline internal numbers.","major_comments":[{"comment":"Table 3, the sole support for the abstract's '84 of 92' and the 'outperforms Buttercup by 29.3%' claim, is internally inconsistent. Summing the PatchIsland column across the 17 rows gives 57, not 84; the Buttercup column sums to 84; the RoboDuck column sums to 24, yet the text states RoboDuck and Buttercup patched 57 and 16. Many percentages are attached to the wrong fractions (e.g., Buttercup 10/11 shown as 63.6% instead of 90.9%; freerdp 5/5 shown as 40.0% instead of 100%). This appears to be a column/percentage swap, but as printed the table is not auditable. The authors must provide a corrected table and a per-PoV outcome matrix (or release logs) to support the headline internal result.","section":"§6.1, Table 3"},{"comment":"The manuscript conflates 'plausible patch' and 'correct patch' in its headline claims. The internal evaluation (Table 3, §6.1) counts a vulnerability as 'repaired' if the patch compiles, suppresses the PoV, and passes functional tests (§3.1), i.e., plausibility. The postmortem (§7) shows that for the 33 distinct intended vulnerabilities, only 27 were correct after manual review, and for unintended vulnerabilities 10 of 14 plausible patches were incorrect. The abstract and introduction should explicitly state that 84/92 is a plausible-patch count, not a correctness count. Moreover, §7's claim that 27/33 (actually 81.8%, not 82.8%) is 'consistent with our evaluation results' compares a correctness metric on a different set to a plausibility metric on the internal set; this needs clarification.","section":"§6.1 vs §7"},{"comment":"The relationship between the deployed FP2 system and the reported numbers is ambiguous. Table 8's rightmost 'PatchIsland' column reports 87/92 as a best-of-N upper bound, while §6.2 calls this number an upper bound. The text does not state explicitly whether the deployed FP2 orchestration achieved 84/92 (Table 3) or 87/92 (best-of-N). Because the headline contribution is the actual system rather than an oracle ensemble, the paper must clearly distinguish the best-of-N reference from the realized FP2 result and report the realized result with the corresponding per-vulnerability outcomes.","section":"§6.2, Table 8"},{"comment":"No code, logs, or per-PoV outcome matrices are released. Given the inconsistencies in Table 3 and the importance of the internal comparison to the paper's claims, I consider the absence of these artifacts a load-bearing issue. The authors should release at least the per-vulnerability outcome table for all systems (as in Table 8 but extended to baselines), the exact agent configurations, and the FP2 scheduling decisions for each task, so that the internal evaluation can be verified independently.","section":"Reproducibility"}],"minor_comments":[{"comment":"'success rate of 82.8% (27 / 33)' should be 81.8% (27/33 = 0.8181...).","section":"§7"},{"comment":"Typo in the first sentence: 'presetned' should be 'presented'.","section":"§10"},{"comment":"The 'Number of Vulnerabilities' column in Table 6 is ambiguous: it appears to be the number of vulnerabilities for which each team generated a PoV, not the total number of official confirmed vulnerabilities. Please define this column and explain why denominators differ across teams.","section":"§6.1, Table 6"},{"comment":"The discussion of full vs delta mode is clear generally, but Table 2 lists several projects with zero vulnerabilities (e.g., Dicoogle, dcm4che). It would help to state explicitly that the selected 92 vulnerabilities are only those with an intended PoV in the released dataset, and that some projects had no such vulnerabilities.","section":"§2.2"},{"comment":"The correctness of the patch-side deduplication loop (Lines 9–12) is not immediately obvious, particularly the condition 'if not ResolvedByPatch(patch, crash)' before merging. A brief explanation or example would improve readability.","section":"§4.2, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The internal Table 3 inconsistency is the main reason for major revision. I believe it is a fixable presentation/data-release issue rather than a fundamental flaw, because the official AIxCC 31/43 result is externally validated. However, the manuscript cannot be accepted in its current form with an internally inconsistent table as the basis for the headline internal claim. I would condition acceptance on a corrected table, a per-PoV outcome matrix, and a clear statement of plausible-vs-correct metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, PatchIsland's official AIxCC final result — 31 accepted patches out of 43 manually validated vulnerabilities — is externally verified and gives the paper real weight. Second, the internal benchmark table that supports the abstract's \"repaired 84 of 92\" and the claimed 29.3% edge over Buttercup is arithmetically broken. As printed, the columns do not sum to the totals: the PatchIsland column sums to 57, the Buttercup column to 84, and the RoboDuck column to 24, while the totals row prints 16/57/84 and the text says RoboDuck and Buttercup patched 57 and 16. Several percentages are attached to the wrong fractions (e.g., 10/11 shown as 63.6%). This is the only quantitative support for the headline internal comparison, and it cannot be reproduced from the paper.\n\nWhat the paper does well: it is an honest integration of known pieces — ensemble LLM agents, patch-based deduplication, crash-and-patch orchestration, and a build-caching framework (Crete) — into a continuous vulnerability repair pipeline. The system ran fully autonomously in AIxCC and produced the most accepted patches of any team. The postmortem is candid in a way that earns credit: it quantifies plausible-but-incorrect patches (10 of 14 for unintended vulnerabilities), documents a single point of failure in systemd, and openly states that the pdfbox zero-day fix may not have originated from their system even though it matches the maintainer's patch. That level of self-scrutiny is not common.\n\nThe soft spots are proportionate to the evidence. The Table 3 problem is load-bearing because the paper's internal margin and the \"84 of 92\" claim rest on it. No code, logs, or per-PoV outcome matrix are released, so the reader cannot tell whether this is a column-swap typesetting error or a genuine miscount. The paper also tunes k=5 on the same benchmark used for evaluation, and the ensemble's headline benefit is presented as a best-of-N upper bound (87/92) rather than the deployed FP2 system's actual result. The paper's own §7 caveat about plausible-patch correctness is a separate, second-order concern; the first-order problem is that the raw counts themselves do not add up.\n\nWho should read it: anyone working on LLM-based automated repair or continuous fuzzing pipelines. The official AIxCC result and the architectural lessons are valuable even if the internal numbers are untrustworthy as printed. I would cite this for the system design and the competition outcome, not for the internal benchmark comparisons.\n\nFor peer review: this deserves a serious referee, but the verdict should be major revision. The tables need to be corrected and reconciled with the text, the artifacts (or at least the per-PoV matrix) should be released, and the deployed-versus-upper-bound distinction needs to be clear. Do not desk-reject it; the external validation is too strong for that.","headline":"PatchIsland has a genuinely strong externally validated AIxCC result, but the internal 84/92 benchmark table does not add up, so the paper needs major revision before it can be trusted.","tokens_in":24354,"tokens_out":5183,"would_cite":true,"duration_ms":48366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A robustly orchestrated ensemble of LLM agents can continuously repair fuzzing-discovered vulnerabilities, achieving a 72.1% success rate in a live competition.","keywords":["automated vulnerability repair","continuous fuzzing","LLM agents","ensemble orchestration","crash deduplication","patch validation","zero-day patch","AIxCC"],"falsifier":"Manually review all 84 internally 'repaired' patches from the 92-vulnerability benchmark against the ground-truth patches and classify each as semantically correct or symptom-suppressing; if more than a handful are incorrect (as the paper's own postmortem found for 10 of 14 unintended-vulnerability patches), the core claim of a 91.3% repair rate collapses to a much lower correct-repair rate. For the competition result, re-run the same manual review on the 31 accepted patches and check each against the actual root cause.","tokens_in":23287,"feed_emoji":"🛠️","tokens_out":4855,"duration_ms":51185,"temperature":0.7,"pith_summary":"The paper argues that automated vulnerability repair must move from static single-run benchmarks into continuous fuzzing pipelines, where crashes arrive at scale and systems must keep running for days. To do this it introduces PatchIsland, a coordinator-worker system that runs a diverse ensemble of LLM-based repair agents, deduplicates both crashes and patch candidates, and schedules agents to balance speed, cost, and provider rate limits. In an in-house evaluation on a 92-vulnerability benchmark drawn from the official competition dataset, PatchIsland produced plausible patches for 84 vulnerabilities, and in the fully autonomous competition final it produced 31 accepted patches for 43 confirmed vulnerabilities (72.1%). The authors further report that one of its patches for a real zero-day bug in the pdfbox library exactly matched the patch eventually merged by the project maintainer.","feed_headline":"Multi-agent LLM system patches 31 of 43 real fuzzing bugs","feed_subtitle":"A fully autonomous repair pipeline beat prior systems in a live competition and matched a maintainer's zero-day fix.","key_machinery":"The carrying mechanism is the ensemble of LLM agents: multiple independent patch generators with different workflows and models run in parallel, so a single agent's failure does not stall repair. Two supporting mechanisms make the ensemble practical: two-phase deduplication, which re-runs new crashes against existing patches to identify duplicates and merges new patches that subsume older ones; and FP2 orchestration, which runs high-preference agents first, keeps agents from the same LLM provider off different workers to avoid throttling, and takes the first plausible patch (FCFS). The Crete framework standardizes build, test, and fault-localization infrastructure across agents and caches ex","core_discovery":"The central claim is that continuous vulnerability repair is feasible when multiple LLM agents are orchestrated into a robust, always-on system. PatchIsland embodies this: an ensemble of diverse agents explores different repair strategies and covers for each other's failures; a two-phase patch-based deduplication groups crashes by root cause and merges partial patches; and an FCFS/preference/provider-aware scheduler keeps the system efficient across providers. The paper's evidence is the 91.3% plausible-patch rate on 92 internal benchmark vulnerabilities and the 72.1% officially validated repair rate (31 of 43) in the competition final, including a patch identical to the maintainer's fix for","pith_inferences":["The 84/92 internal result should be read as a plausible-patch rate; if the correctness rate observed on unintended vulnerabilities (4/14) applied more broadly, the true repair rate would be substantially lower.","The design suggests that AVR evaluation needs to incorporate semantic validation, either via human review or via richer test oracles, before claiming real-world repair capability.","The same ensemble-plus-deduplication pattern may transfer to other high-volume, noise-prone automation tasks, such as continuous issue triage or dependency-failure repair.","Cost and resource usage remain significant (the paper concedes higher cost than baselines), so practical deployment would require budget controls or cheaper fallback agents."],"forward_implications":["Plausible-patch repair becomes a drop-in stage between continuous fuzzing and the bug tracker.","Ensemble diversity plus simple scheduling makes the system resilient to agent-level failures and provider rate limits.","Patch-based deduplication can reduce redundant crash-triage and patch-submission effort in existing fuzzing pipelines.","The gap between plausible and semantically correct patches (10 of 14 on unintended vulnerabilities) underscores that functional tests and manual review remain essential."],"fun_headline_variants":["LLM agent orchestration repairs 72% of live fuzzing bugs","PatchIsland: autonomous multi-agent patching for continuous fuzzing","Orchestrated LLM agents fix 31 of 43 vulnerabilities in competition","Multi-agent repair system hits 72.1% success on real fuzzing bugs","PatchIsland: ensemble LLMs keep patching after fuzzing finds bugs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system's measured repair success rests on the assumption that a patch which compiles, prevents the proof-of-vulnerability from reproducing, and passes the included functional tests is a genuine repair; for vulnerabilities with sparse functional tests this assumption demonstrably fails, so the headline rates may overstate correct repairs.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent orchestration repairs 72% of live fuzzing bugs","PatchIsland: autonomous multi-agent patching for continuous fuzzing","Orchestrated LLM agents fix 31 of 43 vulnerabilities in competition","Multi-agent repair system hits 72.1% success on real fuzzing bugs","PatchIsland: ensemble LLMs keep patching after fuzzing finds bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2215,"prompt_tokens":773,"completion_tokens":1442,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1339}},"tokens_in":517,"tokens_out":1442,"duration_ms":10509,"temperature":1.0,"reasoning_tokens":1339,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:16:49.877446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually review all 84 internally 'repaired' patches from the 92-vulnerability benchmark against the ground-truth patches and classify each as semantically correct or symptom-suppressing; if more than a handful are incorrect (as the paper's own postmortem found for 10 of 14 unintended-vulnerability patches), the core claim of a 91.3% repair rate collapses to a much lower correct-repair rate. For the competition result, re-run the same manual review on the 31 accepted patches and check each against the actual root cause.","supporting_citations":[],"review_version":1}