{"id":"5bddf43c-5567-457e-93fe-ceabe2d5b489","arxiv_id":"2608.13450","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-generated dynamic analysis harnesses for Autoware rarely compile or link against the real build, no weakness was confirmed, and build integration, not fuzzing, is the binding constraint.","lead":"This study tested whether large language models can build executable test harnesses that confirm software weaknesses in Autoware, an open-source self-driving stack. Almost all generated harnesses failed to compile or link against the real build, and none confirmed a weakness, showing that build integration is the current bottleneck.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim over-generalizes from one AV stack and two open-weight models; Section 9 itself warns against direct generalization.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing premise: Autoware and two open-weight models represent full AV stacks and current LLM practice. My stress-test agrees. The paper's own limitation (Section 9) says the rates 'should not be generalized directly' and acknowledges a link-aware repair process could perform better, which is in tension with the abstract's unqualified generalization. No internal inconsistency undermines the observed pipeline outcomes; the attrition numbers and stub-convergence analysis are credible and independently supported by the case studies. The concern is external validity, not internal validity. Because the reader already conditioned the verdict on this issue, I recommend no change (UNCHANGED) rather than a new verdict. A single-stack, two-model study cannot, without further evidence, establish that build integration is the primary barrier for all full AV stacks; the Apollo replication is the concrete test that would settle it.","tokens_in":14407,"tokens_out":4609,"duration_ms":43843,"concrete_test":"Replicate the pipeline on Apollo (Baidu, Apache-2.0) using the same two models, decoding parameters, repair budget (≤3 rounds), and 600s fuzzing budget per target, and compare: (a) the share of first-shot compile failures classified as dependency wiring (missing includes + invalid source paths) and (b) the fraction of harnesses that link and reach the fuzzer. If (a) is not a clear majority or (b) exceeds 50% in Apollo, the headline claim does not generalize beyond Autoware. As a secondary probe, re-run the Autoware condition with a link-aware repair step that re-imports real symbols instead of stubs; if linked-and-fuzzed rises substantially, the bottleneck is repair design rather than an intrinsic property of AV stacks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that build integration is the primary barrier to reliable LLM-assisted dynamic analysis of full autonomous-vehicle software stacks rests on an unstated representativeness assumption. The evidence is generated from exactly one stack (Autoware, ROS 2/colcon, cloned 10 Feb 2026) and two local open-weight models (codestral:22b, gpt-oss:20b) served via Ollama. The causal inference 'build integration, not candidate generation or fuzzing' is therefore only established for this configuration. Section 9 states: 'Because the study evaluates only Autoware and two open-weight models, the reported rates should not be generalized directly,' and adds that 'a link-aware repair process should perform better.' Yet the abstract and conclusions extend the claim to 'full autonomous-vehicle software stacks.' If Apollo (which uses a different build system, CyberRT/Bazel) or a link-aware repair loop or a stronger frontier model changes the bottleneck — e.g., dependency-wiring drops below a majority of failures or the linked-and-fuzzed rate rises above 50% — the primary-barrier claim weakens substantially. The internal data (2,259/2,960 first-shot failures; 652/2,960 linked; 0 confirmations) is consistent with the claim for this setup, but cannot establish generality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an end-to-end feasibility study of LLM-assisted dynamic analysis on Autoware, an open-source autonomous-driving stack. The authors first perform a compiler-driven static analysis over 185 packages, recovering 1,375 decision rules, 2,274 validation checks, and 482 input-to-safety-output flows, and derive a weakness taxonomy. From this they select 740 high-priority sites and, for each, generate four artifacts with two local open-weight LLMs (codestral:22b and gpt-oss:20b), a no-static-context ablation, and a naive baseline, yielding 3,700 artifact sets. The artifacts are compiled against the real Autoware build under ASan/UBSan, repaired in a compiler-in-the-loop loop for up to three rounds, and fuzzed when they link. The central observation is an attrition curve: 2,259 of 2,960 LLM-generated harnesses fail first-shot compilation, with 1,817 of those failures classified as dependency-wiring errors; after repair, only 652 harnesses link and reach the fuzzer; no candidate weakness is dynamically confirmed, and all 37 reported crashes originate in stubbed code. The paper concludes that build integration, not candidate generation or fuzzing, is the primary barrier to LLM-assisted dynamic confirmation for full AV software stacks.","tokens_in":14722,"tokens_out":6731,"duration_ms":68528,"significance":"If the results hold, this is a useful and honest negative result: it is among the first end-to-end studies of LLM-assisted dynamic confirmation on a complete AV stack, and it provides a concrete build-integration failure taxonomy that distinguishes object compilation, linking, target reachability, and genuine dynamic confirmation. The paper's strengths include a reproducible pipeline with released prompts, artifacts, static-analysis outputs, and logs; a controlled comparison across model, static context, and a baseline; and explicit separation of 'not exercised' from 'disconfirmed,' which prevents build failures from being misread as evidence against the static candidates. The stub-convergence observation, showing that compiler-in-the-loop repair can increase compileability while moving harnesses away from real target code, is an important practical insight for the fuzz-driver-generation community. The main weakness is scope: the strong headline claim is derived from a single AV stack and two open-weight models, and the paper's own limitations section cautions against direct generalization.","major_comments":[{"comment":"The abstract and conclusions state that build integration is the primary barrier to reliable LLM-assisted dynamic analysis of 'full autonomous-vehicle software stacks,' but the evidence comes from exactly one stack (Autoware, ROS 2/colcon) and two open-weight models served through Ollama. Section 9 itself says 'the reported rates should not be generalized directly' and that a link-aware repair process 'should perform better.' This is a load-bearing overgeneralization of the central claim. Please either restrict the headline claim to Autoware-class ROS 2 stacks under open-weight models, or add evidence on a second stack or a link-aware repair condition that would test the generality of the build-integration bottleneck.","section":"Abstract and Section 9 (Conclusions)"},{"comment":"Table 8 reports that static context raised the gpt-oss model's first-shot compileability from 13.3% to 56.7%, which contradicts Table 7, where gpt-oss without context compiled 179/740 (24.2%) and gpt-oss with context compiled 473/740 (63.9%). These numbers cannot both be correct; please reconcile the case-study summary with the main results table. As printed, the inconsistency undermines confidence in the reported effect of static context.","section":"Table 8, case study 3 vs Table 7"},{"comment":"The claim that build integration, rather than fuzzing, is the primary barrier is inferred from attrition (only 652 of 2,960 harnesses linked and reached the fuzzer) rather than from a comparison that varies fuzzing conditions. The paper also reports that the realized fuzzing time was 60 seconds per target, not the configured 600 seconds. Please state explicitly that the fuzzing conclusion is an inference from attrition and not a controlled demonstration that fuzzing would have been ineffective had more harnesses linked; otherwise the 'not fuzzing' part of the central claim is stronger than the experimental design supports.","section":"Section 7, Fuzzing attribution"},{"comment":"The abstract and conclusions say that all 37 crashes originated in stubbed code rather than Autoware, but the case-study section describes only one crash (Table 8, case 4). Since the claim that no crash reached real Autoware code is important for the interpretation of the negative result, please provide a full inventory or a summary table of all 37 crashes with their originating stack frames and classification.","section":"Abstract and Section 8/Conclusions, crash count"}],"minor_comments":[{"comment":"The abstract calls the 740 sites a 'sample,' while Section 6 and Table 5 describe them as a deterministic 'census' of all P1 and P2 sites. Please align the terminology.","section":"Abstract and Section 6 (Target Selection)"},{"comment":"The term 'compiler-precise' is used repeatedly; consider defining it explicitly as 'analysis driven by the actual compile_commands.json flags, include paths, and preprocessor definitions,' and noting that this does not imply soundness or completeness.","section":"Section 5 (Analysis Method and Scale)"},{"comment":"Case study 2 refers to 'all 30 targets' for the code-specialized model without defining which 30 targets these are and how they were selected; please specify this subset or its sampling basis.","section":"Table 8, case study 2"},{"comment":"The realized fuzzing time of 60 seconds per target appears only in the limitations section; it would be clearer to include both the configured budget and the realized budget in Table 6.","section":"Table 6 (Experimental Configuration)"},{"comment":"The bar chart does not clearly distinguish the five conditions in grayscale; please add a legend or use distinguishable patterns so the paper is readable in print.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"No concerns about author conduct or data fabrication are raised by my reading. The paper is a solid empirical contribution, but the abstract overstates the generality of the central claim in a way that Section 9 itself disavows, and there is a concrete numerical inconsistency between Table 7 and Table 8. I would support publication after the authors narrow the scope of the headline claim, reconcile the numbers, and provide the crash inventory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what I make of 2608.13450. The headline result: when you actually try to run LLM-generated fuzz harnesses against a real large AV stack, almost all of them die before they execute. The paper's data is unusually specific about where: 1,817 of 2,259 first-shot compile failures are dependency-wiring (missing includes, bad source paths), not program logic. After compiler-in-the-loop repair, the reasoning model gets 100% object compileability, but most of that is stubbing; only 652 of 2,960 harnesses link and reach the fuzzer, and all 37 crashes happen in stubs. That is a clear, reproducible negative result, and it is genuinely new: no prior work I know of runs this end-to-end on a full AV stack with real-build compilation.\n\nWhat I like: the authors draw a sharp line between compileability, linking, and target reachability, and they don't pretend a crash is a finding. The stub-convergence observation is the most useful thing in the paper—compiler feedback alone pushes models toward self-contained fakes, not toward the real target. The static-context ablation (reasoning model first-shot compile 63.9% vs 24.2%) also shows the static stage earns its keep. Numbers are consistent throughout.\n\nSoft spots, in proportion. The biggest is scope. The evidence is one stack (Autoware on ROS2/colcon), one commit date, and two open-weight models. The abstract says 'full autonomous-vehicle software stacks,' and that overstates it. The authors themselves write in Section 9 that the rates should not be generalized directly, so the abstract and conclusions are just written too wide. If Apollo (Bazel/CyberRT) or a link-aware repair loop changes the bottleneck, the primary-barrier claim weakens. I don't think that makes the paper wrong about Autoware or about large ROS2-style codebases—the failure mode is plausible there—but the scope claim needs softening.\n\nTwo smaller things. 'Compiler-precise static analysis' is misleading: the compiler precision is about condition-site recovery; the decision-rule and flow classification is heuristic, as the limitations section admits. And the promised artifact release has no URL or commit hash, which makes independent checking harder. Both are fixable. The fuzzing-budget mismatch (configured 600s, realized 60s) is acknowledged and doesn't affect the main finding since so few harnesses executed.\n\nOverall: this is a solid empirical paper with an honest negative result. It deserves a serious referee. I'd send it out; the main revision is to align the abstract and conclusions with the single-stack evidence, and to make the artifacts actually findable.","headline":"A careful, honest empirical account of where LLM-generated fuzz harnesses fail on a full AV stack—the bottleneck is build integration, not input generation—but the scope claims reach beyond the single stack and two models studied.","tokens_in":15214,"tokens_out":3216,"would_cite":true,"duration_ms":31654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM-assisted dynamic weakness confirmation in a full autonomous-vehicle software stack fails primarily at build integration—wiring harnesses into the native build—rather than at candidate generation or fuzzing.","keywords":["LLM-assisted fuzzing","build integration","Autoware","autonomous vehicle software","static analysis","dynamic weakness confirmation","compiler-in-the-loop repair","ROS 2"],"falsifier":"Apply the same pipeline to a second full autonomous-driving stack (for example, Apollo) or with a link-aware repair process that optimizes for preserving target linkage rather than mere object compilation, and observe whether the fraction of generated harnesses that link and exercise real target code rises from 22 percent to a clear majority and whether any static candidate is dynamically confirmed; if either happens, the claim that build integration is the primary barrier would be contradicted.","tokens_in":14245,"feed_emoji":"🚗","tokens_out":9115,"duration_ms":77446,"temperature":0.7,"pith_summary":"The paper investigates whether large language models can turn static-analysis findings in a full autonomous-driving codebase into executable harnesses that dynamically confirm whether a weakness is real. It finds that the answer is currently no, and the reason is not weak candidate generation or immature fuzzing: across 740 reachable sites in Autoware, 80 percent of the 2,259 first-shot compile failures were dependency-wiring errors (missing headers and invalid source paths), and after compiler-guided repair only 652 of 2,960 harnesses linked and reached the fuzzer, with all 37 observed crashes arising in generated stub code rather than in Autoware. The paper's contribution is a build-integration failure taxonomy that separates object compileability from linking and target reachability, locating the binding constraint one stage before the fuzzer. A sympathetic reading is that LLM-assisted dynamic analysis is not yet a trustworthy assurance stage, but the failure taxonomy points exactly to where tooling should improve.","feed_headline":"LLM AV checks fail at build integration, not at fuzzing","feed_subtitle":"Of 2,960 LLM-generated harnesses for Autoware, only 652 linked; none confirmed a weakness.","key_machinery":"The mechanism that carries the argument is the five-phase pipeline itself, which stages LLM-generated harnesses through compilation against the real Autoware build, compiler-in-the-loop repair (up to three rounds of feeding Clang diagnostics back to the model), linking, fuzzing, and triage, while recording why each artifact drops out. Its output is the build-integration failure taxonomy that classifies first-shot failures into missing includes, invalid source paths, API mismatches, signature mismatches, syntax errors, and other, and the observed 'stub convergence' behavior in which repair satisfies the compiler by removing real dependencies and reimplementing interfaces locally. This taxonomy is what shifts the bottleneck from compilation to linking and target reachability, and it is the paper's main analytical result.","core_discovery":"The central claim is that faithful integration with the native build, not candidate selection or fuzz-input generation, is the primary barrier to LLM-assisted dynamic confirmation of software weaknesses in a production-scale autonomous-vehicle stack. The evidence is an end-to-end run on Autoware: 1,436 missing-include and 381 invalid-source-path errors make up 1,817 of the 2,259 first-shot compile failures, repair raises object compileability to 100 percent for the reasoning model only at the cost of stub convergence, and 2,308 of 2,960 harnesses never reach the fuzzer. All 37 crashes observed during fuzzing originate in model-generated stub code, not in Autoware, and no candidate is dynamically confirmed within budget. The authors therefore state that under a full-link, real-target-execution criterion the effective success rate approaches zero across all conditions.","pith_inferences":["A testable extension would be to give the LLM the package-level build graph rather than per-function source windows, generating harnesses against named build targets; if missing-include and invalid-path failures drop substantially, the paper's diagnosis is confirmed as the actionable bottleneck.","The stub-convergence failure mode suggests a new evaluation metric for LLM harness generation: target-preserving linkage, the fraction of generated artifacts that bind to the intended functions at link time and exercise them at run time, which is more meaningful than compileability.","Porting the pipeline to a smaller, dependency-light library would likely show much higher linking and confirmation rates, which would support the interpretation that the barrier scales with the size and middleware complexity of the integration graph rather than with the LLM's coding ability."],"forward_implications":["Static-analysis-derived inventories of safety-relevant sites in stacks like Autoware are plentiful, but without verified executable harnesses they remain candidates; the paper shows the missing infrastructure is harness construction, not candidate discovery.","Object compileability should not be used as a success metric for LLM-generated harnesses in dependency-rich codebases, because repair routinely achieves it by stubbing the intended implementation.","Providing static-analysis context materially improves first-shot artifact quality (first-shot compileability rose from 24.2 percent to 63.9 percent for the reasoning model), so the static stage pays off even when dynamic confirmation fails.","LLM-generated harnesses may still be useful as drafts that human engineers complete and validate, but not as an unattended assurance stage in continuous integration.","Model choice matters: the general reasoning model compiled 473 of 740 harnesses first-shot with static context versus 46 for the code-specialized model, indicating that wiring ability, not code specialization, drives early-stage success."],"supporting_citations":[{"why":"Establishes that LLMs can act as zero-shot fuzzers finding real bugs in deep-learning libraries; the premise of LLM-generated dynamic artifacts that this study scales from libraries to a full AV stack.","marker":"Deng et al. 2023"},{"why":"Studies how effective LLM-based fuzz driver generation is for libraries; supplies the expectation that generated drivers should compile that the paper tests and fails to meet at build-integration scale.","marker":"Zhang et al. 2024"},{"why":"Fuzz4All claims universal LLM fuzzing across many programs; a key prior baseline the paper contrasts with repository-scale ROS 2 harness construction.","marker":"Xia et al. 2024"},{"why":"OSS-Fuzz-Gen shows LLM-aided fuzz target generation in continuous fuzzing; provides a practice the paper evaluates on Autoware and finds blocked by dependency wiring.","marker":"Google 2024"},{"why":"Defines ROS 2's architecture—generated interfaces, DDS-based pub/sub, package boundaries—the concrete source of the missing-include and invalid-path failures.","marker":"Macenski et al. 2022"},{"why":"libFuzzer is the mature coverage-guided fuzzer used as the execution stage, supporting the claim that fuzzing itself is not the bottleneck.","marker":"LLVM Project 2024a"},{"why":"Articulates the gap between static findings and dynamic confirmation, motivating why executable harnesses must be built at all.","marker":"Aggarwal and Jalote 2006"}],"fun_headline_variants":["LLM AV harnesses fail at build, not fuzz","Build integration blocks LLM AV testing","No Autoware weakness confirmed by LLMs","LLM dynamic analysis stumbles on build"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generalizing step assumes that Autoware's ROS 2 build graph and the two open-weight models tested (codestral:22b and gpt-oss:20b) are representative of full autonomous-vehicle stacks and current LLM practice, so that the build-integration bottleneck is a property of the class of systems, not of this particular stack and model pair.","fun_headline_variants_meta":{"raw":{"variants":["LLM AV harnesses fail at build, not fuzz","Build integration blocks LLM AV testing","No Autoware weakness confirmed by LLMs","LLM dynamic analysis stumbles on build"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":2084,"prompt_tokens":1035,"completion_tokens":1049,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":990}},"tokens_in":651,"tokens_out":1049,"duration_ms":7621,"temperature":1.0,"reasoning_tokens":990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:42:20.892898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same pipeline to a second full autonomous-driving stack (for example, Apollo) or with a link-aware repair process that optimizes for preserving target linkage rather than mere object compilation, and observe whether the fraction of generated harnesses that link and exercise real target code rises from 22 percent to a clear majority and whether any static candidate is dynamically confirmed; if either happens, the claim that build integration is the primary barrier would be contradicted.","supporting_citations":[{"cited_title":"https://github.com/google/oss-fuzz-gen, 2024, accessed","cited_arxiv_id":null,"evidence_quote":"OSS-Fuzz-Gen shows LLM-aided fuzz target generation in continuous fuzzing; provides a practice the paper evaluates on Autoware and finds blocked by dependency wiring."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Articulates the gap between static findings and dynamic confirmation, motivating why executable harnesses must be built at all."}],"review_version":1}