{"id":"83addfdb-3ed0-4c23-b566-e49b449d8239","arxiv_id":"2412.18371","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"This study defines eight defect types for LLM-based agents and presents Agentable, a CPG-plus-LLM static analysis tool that detects them with reported precision of 88.79% and recall of 91.03%.","lead":"The paper analyzes StackOverflow posts to define eight defect types in LLM-based agents, then builds a static analysis tool called Agentable that combines code property graphs with LLM calls to detect them. It reports 88.79% precision and 91.03% recall on two self-built datasets, though the code and data are not yet released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recall estimate is circular because AgentTest's ground truth was built and verified using Agentable itself.","rationale":"The central empirical claim has two parts: precision on AgentSet and recall on AgentTest. The precision number is plausible as far as it goes, though it is based on self-labeled sampling. The recall number is the load-bearing weakness because the dataset used to measure it is constructed and validated with Agentable itself. This matches the reader's weakest_assumption about the base project being defect-free and each project containing exactly one defect, and sharpens it: the paper's own text says Agentable was used to confirm the base is defect-free, which is circular. A conditional verdict is appropriate: the taxonomy and tool idea are valuable, but the evaluation evidence for 91.03% recall must be independently reproducible before the claim can be accepted.","tokens_in":21576,"tokens_out":5446,"duration_ms":54466,"concrete_test":"Ask the authors to release AgentTest, the diffs for the three fixes to the base project, and the injection scripts. Then have at least two annotators who were not involved in the paper and are blind to Agentable's outputs independently list all eight defect types in (a) the fixed base project and (b) each of the 78 variants. Recompute recall against the intersection of the annotators' labels. If any defect is found in the base, if any variant has zero or more than one defect, or if the intersection-based recall differs materially from 91.03%, the reported recall is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section V-A, AgentTest is constructed from a single base project. The authors manually found three defects, fixed them, and then 'ran Agentable for detection, confirming that no defects were present.' This validates the negative ground truth with the tool under test. If Agentable has any false negatives—the very quantity recall is meant to measure—residual defects remain in the base, and each modified project can contain more than the one injected defect. The reported recall of 91.03% is computed as 71/78 (Table IV, with text in Section V-D), assuming exactly 78 true defects in 78 projects. Any unremoved defect increases the true denominator and lowers real recall. The positive cases are likewise derived from the same pipeline: the injected scenarios were 'based on the detection results of RQ1' (Section V-A), so the positive ground truth is aligned with the patterns Agentable was designed to recognize. The precision estimate on AgentSet (Section V-C) is less affected because it rests on manual labeling of a random sample, but the headline recall figure is not an independent measurement of detection ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first systematic study of code defects in LLM-based autonomous agents. It derives a taxonomy of eight defect types (ADAL, IETI, LOPE, TRE, ALS, MNFT, LARD, EPDD) from 331 StackOverflow posts using open card sorting, and then proposes Agentable, a static analysis tool combining Code Property Graphs and LLM reasoning to detect these defects. The tool is evaluated on AgentSet (84 real-world agent projects, reporting 889 detections with 88.79% precision estimated from a randomly sampled subset) and AgentTest (78 hand-injected defect projects, reporting 91.03% recall). The paper claims that this is the first work to define and detect LLM-agent code defects and to provide a large-scale empirical analysis of their prevalence in real projects.","tokens_in":21925,"tokens_out":5034,"duration_ms":47045,"significance":"If the evaluation is sound, this is a useful and timely contribution to the emerging area of LLM-agent reliability. The proposed taxonomy is grounded in developer posts and covers components that prior security-focused work has not systematically addressed. The design of Agentable—using CPGs for structural analysis and LLMs for semantic/generalization checks—is a reasonable and transferable approach, and the construction of a real-world dataset of 84 agent projects is a valuable resource for the community. However, the central recall claim rests on a benchmark whose negative ground truth is verified by the very tool being evaluated, which substantially weakens the credibility of the 91.03% figure as an independent measure of detection ability.","major_comments":[{"comment":"The claim that the base project is defect-free after fixing the three manually identified defects is validated by 'ran Agentable for detection, confirming that no defects were present.' This is circular for the purpose of measuring recall: if Agentable has any false negatives, residual defects remain in the base project, so the denominator of 78 (one injected defect per project) is incorrect and detections of residual defects would be misclassified as false positives. Please re-validate the negative ground truth with an independent method (e.g., multiple independent human inspections or a different detection tool) or, if no such validation is feasible, explicitly analyze and report how this circularity affects the reported recall rate.","section":"Section V-A, paragraph 'Dataset Construction of AgentTest'"},{"comment":"The positive scenarios in AgentTest are constructed 'based on the detection results of RQ1,' meaning they are derived from the outputs of Agentable itself. This aligns the benchmark with the specific patterns the tool was designed to recognize, so the measured recall of 91.03% reflects the tool's ability to find defects similar to those it already finds, not necessarily its ability to detect the full range of defect manifestations in the wild. I recommend constructing positive cases from independent sources—for example, real defect reports from GitHub issues or a held-out set of projects not used in RQ1—and reporting how the injection scenarios were selected.","section":"Section V-A, paragraph 'Dataset Construction of AgentTest', and Table IV"},{"comment":"The abstract and introduction repeatedly state that Agentable achieved an 'overall accuracy of 88.79%', but the computation in Section V-C and Table III is a precision (the fraction of reported defects that are true positives), and no true negatives are measured. Accuracy and precision are different metrics; using 'accuracy' without a definition that includes true negatives is misleading. Please replace 'accuracy' with 'precision' throughout, or explicitly define the metric used.","section":"Abstract, Section I, and Section V-C"},{"comment":"The detection oracles are described at a fairly high level, and the details needed for independent replication—such as the exact LLM prompts (beyond stating that chain-of-thought is used), the specific CPG query patterns for each defect type, and the value of the grouping parameter n—are not provided. Since the source code is not yet released, the evaluation is not reproducible as presented. Please include the prompts and pattern specifications in an appendix or make the code and datasets available with the submission, rather than only after acceptance.","section":"Section IV-D-2"}],"minor_comments":[{"comment":"Acronym usage is inconsistent: the text alternates between 'ADAL' and 'AAL', 'IETI' and 'ETE', 'LOPE' and 'LPE', 'LARD' and 'ARD', and 'EPDD' and 'EPDC'. Please standardize to the definitions in Table II.","section":"Entire manuscript"},{"comment":"The same code example in Figure 8 is used to illustrate both the ALS and MNFT defects; it would be clearer to explicitly annotate which lines correspond to which defect in the figure caption or in a separate listing.","section":"Section III-D(5) and (6), Figure 8"},{"comment":"The sentence 'in an TRE defect detection, the LLM sometimes flagged non-essential fault-tolerance aspects missing in the Tool code as IETI defects' appears to contain a typo ('TRE' should likely be 'IETI'), and the surrounding discussion of IETI errors should be checked for consistency.","section":"Section V-D, paragraph after Table IV"},{"comment":"The subsection heading 'Accuracy' should be renamed 'Precision' to match the metric actually reported in Table III and the text.","section":"Section V-C, subsection heading 'Accuracy'"},{"comment":"The reported LLM invocation cost of $24.2 and average cost of $0.027 per defect are useful details; consider also reporting the number of LLM calls made in total, which would help readers assess scalability.","section":"Section V-C, 'Overhead' paragraph"}],"recommendation":"major_revision","confidential_remarks":"The circularity in the recall evaluation is the main substantive concern; I would treat this as a blocking issue unless the authors can provide an independent verification of the AgentTest negative ground truth or substantially temper the recall claim. The precision evaluation, while based on author labeling, uses a reasonable sampling design with finite-population corrections, so I do not see it as fatal. The 'accuracy' vs 'precision' terminology error is also worth correcting in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is the first to offer a systematic taxonomy of code-level defects in LLM-based agents and a tool (Agentable) that combines Code Property Graphs with LLM calls to flag them. That part is genuinely useful. The eight defect types (tool registration gaps, missing fault tolerance, LLM API misuse, etc.) come from a manual analysis of 6,854 StackOverflow posts, and the examples are concrete and plausible. The tool design is also reasonable: build a CPG, locate agent/tool/LLM initialization, and use an LLM for semantic checks. On 84 real projects, the reported precision of 88.79% is supported by manual labeling of 339 sampled detections, which is a real, if self-reported, evaluation.\n\nThe soft spots are where the evidence gets weaker. The recall number of 91.03% is not an independent measurement. AgentTest is built from one base project that the authors manually fixed and then validated by running Agentable itself. Any false negatives Agentable has at that step stay in the base, so the 78 injected projects may carry more than the one intended defect, and the true denominator is unknown. The stress-test note is on target: the positive scenarios were also derived from the tool's RQ1 detections, so the test aligns with what Agentable was built to recognize. I would treat the recall as an upper bound until the artifacts are released and the ground truth is independently verified. Precision on AgentSet is less affected because it rests on manual labeling, not on the tool's own output.\n\nTwo smaller things. There is no baseline comparison, so we do not know whether the CPG+LLM combination beats a simpler LLM-only prompt. And the paper says it will release code and data only after acceptance, which makes it impossible to check the claims now. Also, the acronyms are inconsistent (ADAL vs AAL, IETI vs ETE, LOPE vs LPE) and the abstract says \"accuracy\" where it means precision.\n\nThe core contribution, the taxonomy and the general detection approach, holds up. The evaluation needs significant revision, not a desk rejection. This deserves a serious referee: the topic is timely, the taxonomy is a reference point, and the tool is a credible first step. With the datasets and code released and recall recomputed on an independently constructed ground truth, this could be a solid paper.","headline":"A useful first taxonomy and detection tool for LLM-agent code defects, but the headline recall figure is not independent because the test ground truth was validated with the tool itself.","tokens_in":22315,"tokens_out":2860,"would_cite":true,"duration_ms":27969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines eight recurring defect types in LLM-based agent code and presents Agentable, a static analyzer combining Code Property Graphs and LLMs that reports 88.79% precision and 91.03% recall.","keywords":["LLM agents","agent defect taxonomy","static analysis","Code Property Graph","LLM-based detection","empirical software engineering","fault tolerance","tool integration defects"],"falsifier":"Have an independent team audit the base agent project and the 78 AgentTest variants; if the base still contains a defect after the three fixes, or if any test project contains a defect beyond the one injected, then the 91.03% recall estimate is not a valid measure of true recall.","tokens_in":21402,"feed_emoji":"🐛","tokens_out":8019,"duration_ms":65369,"temperature":0.7,"pith_summary":"This paper claims that the code wiring LLMs into autonomous agents suffers from a small, nameable set of recurring defects that can be caught before runtime. The authors analyzed 6,854 developer Q&A posts to define eight defect types spanning tool registration, LLM output parsing, trigger-word handling, fault tolerance, API calls, and package dependencies. They built Agentable, a static analyzer that combines Code Property Graphs for code structure with LLM calls for semantic judgment, and evaluated it on 84 real-world agent projects and 78 deliberately defective ones. On the annotated set it reached 88.79% precision and 91.03% recall, and it flagged 889 suspected defects in the real projects. If correct, this gives agent developers a concrete defect checklist and a working detector instead of an open-ended debugging problem.","feed_headline":"Static analysis catches LLM-agent defects at 88.79 percent","feed_subtitle":"A code-graph-plus-LLM analyzer flagged 889 defects in 84 real agent projects and named eight recurring failure modes.","key_machinery":"The machinery is a three-module detection pipeline. A Code Property Graph is abstracted into a Unified Node Relationship Tree (UNRT) that treats classes and functions as equivalent nodes, so searches for LLM initialization, agent initialization, and tool instances work across different framework styles. A semantic enrichment module pulls in source code and abstract syntax trees to recover runtime-state information the graph loses. An LLM invocation module performs the generalization-heavy judgments, such as whether a tool's name, description, and implementation agree or whether fault tolerance wraps a call. Each of the eight defect types has its own detection oracle that decides which module combination to run.","core_discovery":"The paper's central claim is that LLM-based agent code has its own defect taxonomy, distinct from both ordinary bugs and security vulnerabilities. The eight types are: Adaptation Defect between Agent and LLM, Insufficient External Tool Information, LLM Output Parsing Error, Tool Return Error, Action Listener Setting, Missing Necessary Fault Tolerance, LLM API-related Defect, and External Package Dependency Defect. Each type is grounded in reports from developer Q&A posts and tied to a component of the agent workflow. The authors further claim that Agentable, a static analyzer combining Code Property Graphs with LLM-based semantic checks, detects these defects at the reported precision and recall, and that the 889 defects found in 84 real projects show the taxonomy matters in practice.","pith_inferences":["The paper leaves implicit that its taxonomy is anchored in projects and posts written mostly in Python, so agent ecosystems in other languages may need additional defect categories.","A practical extension is to turn the warning-only External Package Dependency Defect reports into real checks by adding version-resolution analysis at build time.","The same detection oracles could be adapted to multi-agent systems, where inter-agent message mismatches may form a whole new defect class.","An empirical test of the taxonomy's completeness would be to run Agentable on a fresh sample of agent projects and measure how many reported anomalies fall outside the eight types."],"forward_implications":["Agent developers can use the eight-type taxonomy as a code-review checklist before deploying an agent.","The same detection patterns can be wired into continuous integration so tool registration, output parsing, and fault-tolerance mistakes are caught before release.","The defect distribution, dominated by LLM API-related and external-package-dependency issues, tells framework authors where to focus documentation and guardrails.","Each defect type comes with a proposed mitigation, so a detection report points toward a fix rather than just a warning.","The design's reliance on graph abstraction and LLM judgment is intended to transfer to agent frameworks beyond the ones that appear in its training data."],"supporting_citations":[{"why":"Defines the Code Property Graph concept that unifies syntax, control flow, and data flow for Agentable's analysis.","marker":"[31]"},{"why":"Supplies the CPG construction and query tool that Agentable calls to build its code graph.","marker":"[30]"},{"why":"The base open-source agent project whose manually fixed version seeds the AgentTest recall dataset.","marker":"[26]"},{"why":"The hybrid card sorting method used to derive the eight defect categories from collected developer posts.","marker":"[19]"},{"why":"Precedent for mining defect types from developer Q&A posts, guiding the data collection and filtering process.","marker":"[20]"},{"why":"The classification procedure followed for the open card sorting rounds that produced the taxonomy.","marker":"[44]"}],"fun_headline_variants":["New taxonomy names 8 failure modes in LLM agents","Static analyzer Agentable finds 889 defects in LLM agents","First study: 8 defect types plague LLM agents; tool catches them","Code Property Graphs + LLMs spot common agent failures","8 recurring defects in LLM agents, 889 found in 84 projects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 91.03% recall presupposes that the base agent project used to build AgentTest is defect-free after three manual fixes and that each of the 78 test projects contains exactly one true defect.","fun_headline_variants_meta":{"raw":{"variants":["New taxonomy names 8 failure modes in LLM agents","Static analyzer Agentable finds 889 defects in LLM agents","First study: 8 defect types plague LLM agents; tool catches them","Code Property Graphs + LLMs spot common agent failures","8 recurring defects in LLM agents, 889 found in 84 projects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001149,"raw_usage":{"total_tokens":4803,"prompt_tokens":1025,"completion_tokens":3778,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":3689}},"tokens_in":641,"tokens_out":3778,"duration_ms":23317,"temperature":1.0,"reasoning_tokens":3689,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:45.971590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent team audit the base agent project and the 78 AgentTest variants; if the base still contains a defect after the three fixes, or if any test project contains a defect beyond the one injected, then the 91.03% recall estimate is not a valid measure of true recall.","supporting_citations":[{"cited_title":"Adcpg: Classifying javascript code property graphs with explanations for ad and tracker blocking,","cited_arxiv_id":null,"evidence_quote":"Supplies the CPG construction and query tool that Agentable calls to build its code graph."},{"cited_title":"mpaepper,","cited_arxiv_id":null,"evidence_quote":"The base open-source agent project whose manually fixed version seeds the AgentTest recall dataset."},{"cited_title":"Making it tangible: hybrid card sorting within qualitative interviews,","cited_arxiv_id":null,"evidence_quote":"The hybrid card sorting method used to derive the eight defect categories from collected developer posts."}],"review_version":1}