{"id":"f439a231-8de8-4a1f-b33e-87857436ec19","arxiv_id":"2501.04312","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-driven fuzzer extracts transferable edge cases from API source-code checks and covers more TensorFlow/PyTorch APIs while finding 37 bugs.","lead":"DFUZZ is a new fuzzing tool that uses large language models to read the internal checks in deep learning libraries and turn them into test cases for many API functions. In tests on TensorFlow and PyTorch, it covered more APIs than earlier LLM-based fuzzers and found 37 bugs, 8 of which developers have already fixed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bug oracle may overcount TensorFlow 'abort signals' as defects, leaving the transferability claim validated only through the very bug counts it is meant to prove.","rationale":"I read the paper as an empirical claim: a white-box LLM fuzzer with context-free edge-case transfer outperforms black-box LLM fuzzers on coverage and bug discovery. The coverage comparison (Table III) is plausible, and the developer-confirmed fixes are real supporting evidence. The reader's weakest_assumption correctly identifies the transferability heuristic, but I think the more acute risk is the oracle used to count bugs: if TensorFlow abort signals are mostly intentional CHECK failures, then the 27 TensorFlow bugs, which constitute the main cross-framework transfer result, are not defects under a stricter bug criterion. Since Table VI places 24 of 27 TensorFlow bugs in the abort category and the paper does not report manual triage of these against the library's error-handling contract, the central bug-finding claim is underdetermined. A matched-control transfer experiment plus manual triage would settle this. This concern does not invalidate the coverage claim or the 8 fixed bugs, and it is consistent with a conditional rather than unconditional verdict, so I recommend no change to the reader's verdict.","tokens_in":20534,"tokens_out":4132,"duration_ms":45418,"concrete_test":"Run a matched-control transfer experiment on the released artifact: (1) For each of the 132 extracted edge cases, apply it to all APIs whose etype pattern matches and record execution outcomes; (2) repeat with the same prompt/mutation pipeline but with random input perturbations of the same types as a control; (3) manually triage every TensorFlow abort signal by checking whether the code path is an intentional CHECK/LOG(FATAL) on invalid arguments versus an unguarded memory error, and compare with TensorFlow's documented input-validation policy. If the matched-edge-case crash rate is not significantly above the random control, or if most TensorFlow aborts are intentional CHECK failures, then the RQ2 bug-finding and transferability advantage must be revised downward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is not simply that LLMs can extract checks accurately (the pilot study supports that), but that edge-case transfer across etype-matched APIs yields genuine, previously missed defects rather than expected failures on invalid inputs. Two gaps combine. First, Observation II (Sec. II-B) is operationalized with only seven coarse type categories (Sec. III-A) and is validated only indirectly: Table V reports bugs, but there is no control measuring how often applying a matched edge case to an arbitrary API produces a 'bug' under the paper's oracle, nor a comparison against random mutations with matched input distributions. Second, Table VI reports 24 of 27 TensorFlow findings as 'abort signals.' TensorFlow's CHECK and LOG(FATAL) macros abort the process by design on invalid arguments; DocTer's 'DL APIs should not crash' doctrine is a design preference, not a universal bug criterion. Unless these 24 aborts were manually triaged against the library's intended error-handling contract, the 37-bug headline overstates the advantage, and the across-framework transfer evidence (PyTorch edge cases finding 27 TensorFlow bugs) is not established. The 8 fixed bugs are strong positive evidence, but they are not disaggregated by type or framework; if the fixed set is dominated by PyTorch's runtime/inconsistent-output bugs, the TensorFlow abort-signal claim remains untested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DFUZZ, an LLM-driven fuzzing framework for deep learning libraries. DFUZZ uses GPT-3.5 to extract 'edge cases' (error-triggering conditions) from TORCH_CHECK statements in PyTorch's ATen source code, abstracts them into a context-free form indexed by coarse 'etype patterns' (seven basic types plus compound combinations), and then transfers these edge cases to test other APIs in PyTorch and TensorFlow. A second LLM component generates initial test programs with a debug loop, and a third stage mutates those programs to satisfy the transferred edge cases. Evaluation on PyTorch v1.12 and TensorFlow v2.10 reports higher API coverage than TitanFuzz and FuzzGPT with fewer LLM invocations (Table III), and on latest versions (PyTorch v2.2.1, TensorFlow v2.15) reports 37 bugs, of which 8 are developer-fixed and 19 are replicated by developers (Table V). The authors attribute the TensorFlow bug discoveries to cross-framework transferability of edge cases extracted from PyTorch.","tokens_in":20789,"tokens_out":3077,"duration_ms":31659,"significance":"If the results hold, DFUZZ is a meaningful advance: it achieves higher API coverage than prior LLM-based fuzzers while using only a fraction of the LLM calls, and it reports genuine developer-confirmed fixes (8 fixed bugs), which is strong external evidence that at least some findings are real defects. The paper also gives a concrete mechanism for cross-framework transfer (etype abstraction), and the coverage comparison is performed in the same Docker environment as TitanFuzz, making that part of the evaluation credible. The pilot study on LLM-based check comprehension (Table II) is a useful data point. However, the bug-finding and transferability claims rest on two load-bearing assumptions that are not yet rigorously validated: that abort signals in TensorFlow constitute bugs under a defensible oracle, and that the seven-type etype abstraction transfers edge cases without a high false-positive rate. The lack of a control for false transfers and the absence of a per-bug disaggregation by type/status leave the central 'transferable edge cases find real bugs' claim under-supported, though not refuted.","major_comments":[{"comment":"The bug oracle is not calibrated for TensorFlow abort signals. Table VI reports that 24 of the 27 TensorFlow findings are 'abort signals,' and TensorFlow's CHECK/LOG(FATAL) macros abort the process by design on invalid arguments. The paper cites DocTer's doctrine that 'DL API functions should not crash' as justification, but that is a design preference, not a universal correctness contract. Without a per-case triage showing that these 24 aborts occur on inputs the API is documented or intended to accept (or that maintainers acknowledged them as bugs), the 37-bug headline may substantially overstate the number of real defects. The paper should disaggregate the 8 fixed and 19 replicated bugs by bug type and framework, and explain, for the TensorFlow abort signals specifically, how they were validated against the library's intended error-handling behavior.","section":"Sec. V-B, Table VI"},{"comment":"Observation II ('if APIs have the same input parameter types, they shall likely share edge cases') is implemented through only seven coarse type categories plus 13 compound etype patterns, yet the evaluation validates this heuristic only indirectly through the bugs found. There is no control measuring how often applying a matched edge case to an arbitrary API produces a 'bug' under the paper's oracle, nor a comparison against random mutations with similar input distributions. Without such a control, the observed bug counts could partly reflect aggressive testing with unusual inputs rather than the transferability of semantically meaningful edge cases. The authors should add an ablation or control experiment, e.g., applying the same set of edge cases to APIs with mismatched etype patterns, or comparing bug yield per generated test against random type-matched mutations, to demonstrate that the transferred edge cases are specifically what drives the advantage.","section":"Sec. II-B and Sec. III-A"},{"comment":"The claim that DFUZZ finds bugs missed by TitanFuzz, FuzzGPT, and IvySyn is inferred from the bugs' existence in older versions tested by those tools, rather than from a controlled side-by-side run. This inference assumes that those tools, given their schedules and mutation budgets, would have triggered the bugs if they existed in the code they tested. The paper should either rerun the baselines on the same versions (at least on the specific bug-triggering inputs) or explicitly discuss the limits of the version-existence argument. As it stands, the statement that 'none of these fuzzers discovered them' is a claim about the baselines' efficacy that is not directly tested in this paper.","section":"Sec. V-B, RQ2"}],"minor_comments":[{"comment":"The type list is given as '{Tensor, Int, Bool, Str, Float, Scalar, List}' but the text later refers to 'Str' and 'str' inconsistently; the paper should standardize the type names and state explicitly whether Python-level types (e.g., 'str' vs 'Str') are intended.","section":"Sec. III-A, prompt in Fig. 5"},{"comment":"There is a typo: 'ruturn' should be 'return'.","section":"Alg. 1, line 11"},{"comment":"The table reports 198 successful and 138 failed debugging attempts for PyTorch, but the text says 336 APIs require debugging; please reconcile these numbers and clarify whether 'attempts' refers to APIs or to individual debug iterations.","section":"Table IV"},{"comment":"The evaluation of llama2 models replays prompts obtained from bugs discovered by ChatGPT-3.5; this measures the smaller models' ability to reproduce known bug-triggering programs, not their ability to discover new bugs in an end-to-end run. The claim that DFUZZ 'can effectively employ' these LLMs should be softened accordingly.","section":"Sec. V-C, RQ3"},{"comment":"The statement that edge cases extracted from PyTorch are 'more effective' for TensorFlow than for PyTorch is interesting, but the comparison is confounded by different API sets, fix histories, and testing efforts; please avoid causal language without additional evidence.","section":"Sec. V-B, 'Transferability' paragraph"},{"comment":"The phrase 'white-box view' is used throughout, but the extraction is performed on source code of PyTorch only; clarify that TensorFlow is tested using edge cases transferred from PyTorch rather than its own source checks.","section":"Sec. VII and Sec. II-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for a software engineering or security venue, and the artifact plus the 8 developer-fixed bugs give it real substance. The main risk is not the novelty of the idea but the validity of the bug oracle and the transferability evidence; both are fixable with additional experiments and a more careful disaggregation of results. I would not reject on the current evidence, but the current manuscript overstates the bug-finding and transferability conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real advance in LLM-based fuzzing for DL libraries, and the 8 developer-fixed bugs are genuine evidence. But the 37-bug headline needs a grain of salt: 24 of 27 TensorFlow findings are abort signals, and TensorFlow's CHECK macros abort on invalid input by design. Without a clear triage showing which aborts violate the library's intended error contract, the advantage over prior fuzzers is overstated.\n\nThe new thing here is the transfer mechanism: DFUZZ uses an LLM to read inline checks (TORCH_CHECK), synthesizes context-free edge cases, abstracts them by input-type pattern, and reuses them across APIs and even across frameworks (PyTorch edge cases applied to TensorFlow). That is genuinely different from TitanFuzz's black-box mutation, FuzzGPT's historical bug-code retrieval, or IvySyn's manual CVE-derived mutators. The pilot study (95%+ accuracy on extracting checks) is a nice sanity check, and the coverage comparison is run in TitanFuzz's own Docker environment, so the head-to-head numbers are credible. LLM usage is also much lower.\n\nThe weak spots are real but addressable. The bug oracle is the big one. The paper quotes DocTer's 'DL APIs should not crash' doctrine, but that is a design preference, not a universal contract. If a TensorFlow op aborts on a clearly invalid input (e.g., wrong dtype), that may be exactly what the developer intended. The paper does not disaggregate the 8 fixed bugs by type or framework, so we cannot tell whether the TensorFlow findings are mostly fixed aborts or mostly repurposed as bugs. The transferability claim is validated only indirectly through the bug counts; there is no control measuring how often a matched edge case produces a false 'bug' on an arbitrary API, nor a comparison against random mutations with matched input distributions. And the experiments are single-run, without variance.\n\nNone of this kills the paper. The mechanism is plausible, the artifact is available, and the fixed bugs prove the approach can find real defects. But the evaluation should be tightened: report bug triage details (which aborts were accepted by developers), disaggregate the fixed/replicated bugs by framework and type, and add a false-transfer control.\n\nThis deserves a serious referee. I would send it to review.","headline":"A real advance in LLM-based fuzzing, with a bug count that needs triage before the headline is trusted.","tokens_in":21343,"tokens_out":2788,"would_cite":true,"duration_ms":25336,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that defensive checks in one DL library API, when read by an LLM and abstracted to type-level edge cases, can be transferred to similar APIs and produce both higher coverage and real bug discoveries.","keywords":["deep learning library fuzzing","API fuzzing","large language models","edge case extraction","type-based transfer","fuzzing coverage","PyTorch","TensorFlow"],"falsifier":"Run DFUZZ on a fixed set of matched (edge case, API) pairs and count how often a transferred edge case actually triggers a crash, abort, segfault, runtime error, or CPU/GPU inconsistency; if this transfer success rate is no higher than applying the same edge cases to APIs with mismatched type signatures, the central transferability assumption collapses.","tokens_in":20346,"feed_emoji":"🐛","tokens_out":9154,"duration_ms":83243,"temperature":0.7,"pith_summary":"DFUZZ's central claim is that the input checks buried in DL library source code are reusable test inputs: an edge case that triggers a TORCH_CHECK in one API can be lifted to an abstract, type-based form and fed to other APIs with the same parameter types. Because LLMs can read those checks and name the failing inputs, DFUZZ automates what previously required human-written mutators or historical bug corpora. The paper reports that this mechanism covers more APIs than prior LLM-based fuzzers on TensorFlow and PyTorch while using a fraction of the LLM calls, and uncovers 37 bugs in the latest versions, with 8 fixed and 19 acknowledged by developers. A fuzzer that treats fixes as exploits is the practical payoff: the defenders' own guards reveal how APIs fail.","feed_headline":"LLM fuzzer turns API fixes into exploits, finds 37 bugs","feed_subtitle":"DFUZZ reuses edge cases from one API's checks to test similar APIs, beating prior fuzzers on coverage and bug counts.","key_machinery":"The machinery is the etype-pattern abstraction: each edge case is paired with a set of parameter types (e.g., {Tensor} or {Int, Tensor}) so that it can be detached from the specific API it came from and matched to any API with the same or a superset of those parameter types. Around this abstraction DFUZZ wraps a three-step pipeline: a Code Extractor that pulls check statements and function interfaces out of the library source; an Analyzer that asks an LLM to read each check and report the checked variables, their types, and the edge case; and a Standardizer that strips variable names to produce context-free edge cases clustered by etype pattern. A debug loop then synthesizes an initial program invoking the target API, and a mutation step selects matched edge cases, prioritizing those that affect early parameters, to generate bug-triggering programs. The abstraction carries the argument because it is what makes knowledge transfer across APIs and across frameworks possible.","core_discovery":"The paper's central discovery is that edge cases are not API-specific but type-specific. Reading TORCH_CHECK statements in PyTorch's ATen source, an LLM can articulate what input would violate the check (e.g., 'a complex tensor'), and that description remains valid after variable names are stripped, because the only thing that matters is the parameter's type. DFUZZ clusters these context-free edge cases under 'etype patterns' built from seven primitive types, then matches any API whose parameter types contain that pattern. The empirical payoff is the transfer itself: edge cases extracted exclusively from PyTorch produced 27 of the 37 reported bugs when applied to TensorFlow, and most of those bugs already existed in versions that prior fuzzers had tested. The paper interprets this as evidence that white-box reasoning about checks, rather than black-box mutation, is what unlocks comprehensive DL library fuzzing.","pith_inferences":["Editorial inference: the seven-type abstraction probably both over-approximates (same type label does not guarantee the same edge case applies) and under-approximates (tensor dtype, shape, layout, and sparsity are erased); a finer type lattice would likely cut false transfers and find more precise bugs.","Editorial inference: the same 'fix is my exploit' mechanism could power differential testing across backends or frameworks, since each check is a candidate input to compare implementations against each other.","Editorial inference: the paper's coverage numbers justify a cheaper validation study: measure the per-edge-case transfer hit rate directly, rather than only reporting aggregate bugs, to quantify how much of the gain comes from transfer versus from LLM program generation.","Editorial inference: if the heuristic holds beyond DL libraries, any software with prevalent inline parameter checks, such as the financial software the paper mentions, could be fuzzed with the same pipeline."],"forward_implications":["A library's own check statements become a reusable fuzz corpus, so the cost of fuzzing a new DL library drops to the cost of extracting its checks and matching them to etype patterns.","Coverage need not be proportional to hand-written mutators or web-scraped bug codes; it scales with the number and diversity of input checks in the framework.","Cross-framework transfer means a bug pattern discovered in one ecosystem can be tested immediately in another, as demonstrated by PyTorch-harvested edge cases finding TensorFlow bugs.","Because most reported bugs are crashes on invalid inputs, DFUZZ's edge-case mutations align with the field's expectation that DL APIs should reject bad inputs gracefully rather than crash."],"supporting_citations":[{"why":"Provides the scheduling-based LLM fuzzing baseline whose API coverage and LLM invocation counts are compared with DFUZZ's.","marker":"[15]"},{"why":"Provides the historical-bug-code LLM fuzzing baseline for coverage and bug-discovery comparison.","marker":"[16]"},{"why":"Provides the manually-crafted-mutator baseline whose detected bugs are compared with DFUZZ's.","marker":"[14]"},{"why":"Provides the accepted criterion that DL API functions should not crash on invalid inputs, used to classify DFUZZ's bugs as critical.","marker":"[64]"},{"why":"The LLM whose accuracy in extracting edge cases was measured in the pilot study and used throughout the main experiments.","marker":"[1]"},{"why":"Provides open-source LLMs used to check whether edge-case mutation works beyond the commercial LLM.","marker":"[3]"}],"fun_headline_variants":["LLM fuzzer exploits API edge cases, digs up 37 bugs","DFUZZ: LLM uses API fixes to find 37 new DL library bugs","Edge cases from one API find bugs in another: LLM fuzzer nets 37","LLM-driven fuzzer reuses API checks to unearth 37 bugs","Fuzzing DL libraries: LLM reasons about API errors, finds 37 bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the premise that APIs with the same input parameter types share the same edge cases, so an edge case harvested from one API remains a meaningful test input for another API with the same type signature.","fun_headline_variants_meta":{"raw":{"variants":["LLM fuzzer exploits API edge cases, digs up 37 bugs","DFUZZ: LLM uses API fixes to find 37 new DL library bugs","Edge cases from one API find bugs in another: LLM fuzzer nets 37","LLM-driven fuzzer reuses API checks to unearth 37 bugs","Fuzzing DL libraries: LLM reasons about API errors, finds 37 bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1357,"prompt_tokens":962,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":578,"tokens_out":395,"duration_ms":3761,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:35:49.929278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DFUZZ on a fixed set of matched (edge case, API) pairs and count how often a transferred edge case actually triggers a crash, abort, segfault, runtime error, or CPU/GPU inconsistency; if this transfer success rate is no higher than applying the same edge cases to APIs with mismatched type signatures, the central transferability assumption collapses.","supporting_citations":[{"cited_title":"Docter: Documentation-guided fuzzing for testing deep learning api functions","cited_arxiv_id":null,"evidence_quote":"Provides the accepted criterion that DL API functions should not crash on invalid inputs, used to classify DFUZZ's bugs as critical."},{"cited_title":"https://openai.com/chatgpt, 2024","cited_arxiv_id":null,"evidence_quote":"The LLM whose accuracy in extracting edge cases was measured in the pilot study and used throughout the main experiments."},{"cited_title":"https://github.com/meta-llama/llama, 2024","cited_arxiv_id":null,"evidence_quote":"Provides open-source LLMs used to check whether edge-case mutation works beyond the commercial LLM."}],"review_version":1}