{"id":"d7a2331b-3bee-4920-a8e5-9196139e5e49","arxiv_id":"2608.10530","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A PRISMA-based survey of 85 papers shows agentic LLM security research is attack-heavy and perception-focused, leaving action-layer and code-execution risks understudied.","lead":"This systematic review of 85 papers on security for LLM-based agents finds that research is concentrated on prompt-level attacks while the most consequential targets, action-layer tool use and code execution, are barely covered. It offers a four-layer vulnerability taxonomy and a list of seven open problems to guide where security work is needed.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own tables contradict its headline ratios: Table 4 lists 82 action-layer occurrences while Section 4.7 reports only 4 action-layer papers; Section 6.1 reports 17 defense papers while Section 5.1 uses 12 for the 3.9:1 ratio.","rationale":"The reader's weak spot was reproducibility of the corpus: missing search strings, undocumented inclusion criteria, and only 17% double-coding. Those are real, but my stress-test found a more direct problem: even using only the numbers printed in the paper, the headline ratios are not mutually consistent. Table 4 reports 82 component-level occurrences across the action layer while Section 4.7 reports 4 action-layer papers; Section 6.1 reports 17 defense-focused studies while Section 5.1 uses 12 to compute a 3.9:1 ratio. The distinction between primary-focus coding (used for the 4.7% and 3.9:1 claims) and component-occurrence coding (Table 4) is not stated or operationalized, so the central claim is ambiguous and potentially overstated. This is not a disqualifying flaw: a supplementary appendix with a per-paper coding table and a single counting rule could settle it. Because the concern is concrete, checkable, and addressable by revision, the appropriate verdict remains conditional. I agree only partially with the reader because their concern targets missing external artifacts; mine targets an internal numerical contradiction that would remain even if the search strings were provided. No ad hominem intended; the issue is with the reporting of the quantitative synthesis, not with the authors' intent.","tokens_in":29409,"tokens_out":7026,"duration_ms":56162,"concrete_test":"Recompute every headline ratio from the authors' coding spreadsheet with one explicit counting rule. For each of the 85 papers, list the primary layer (if any) and primary category (attack/defense/framework/benchmark). Then verify: (1) action-layer primary focus is 4 papers, not 82 occurrences; (2) Table 4's Cnt column is relabeled as occurrence/mention frequency, not paper count; (3) defense-focused count is either 12 or 17, with Section 6.1 reconciled to Section 5.1. If under any-mention counting the action layer appears in far more than 4.7% of papers, the headline must be reworded to 'primary focus' and the gap re-estimated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claims—the 14x perception/action gap and the 3.9:1 attack/defense ratio—do not reproduce from the paper's own tables. Section 4.7 says '4 papers (4.7%) discuss action layer security risks', but Table 4 lists component-level counts for the action layer of Tool Manipulation 54, Function Hijacking 7, Code Injection 13, and Sandbox Escape 8. The table note says these are occurrence counts and 'a single paper can contribute to multiple counts'; even allowing for multi-counting, 82 action-layer occurrences against only 4 primary action-layer papers requires an operational distinction between 'primary focus' and 'mentions' that the text never states. The abstract's phrase 'appear in only 4.7%' invites a prevalence reading that Table 4 undercuts. Independently, the attack/defense ratio is computed in Section 5.1 as 47 attack papers vs 12 defense papers (3.9:1), but Section 6.1 states 'Our review identified 17 defense-focused studies (12 detection and 5 mitigation-only)'; 47/17 is 2.8:1. The quantitative backbone is therefore internally inconsistent, independent of the missing search strings and coding artifacts the reader flagged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a systematic literature review, conducted under PRISMA 2020 guidelines, of 85 papers (2023--2025) on security vulnerabilities in agentic large language models. The authors propose a four-layer architectural taxonomy (perception, brain, action, interaction) covering 13 vulnerability types, report quantitative imbalances in the literature—notably a 3.9:1 attack-to-defense ratio and a roughly 14-fold gap between perception-layer and action-layer research coverage—and identify seven open research problems centered on containment. The paper positions itself as the first PRISMA-based systematic review dedicated to agentic LLM security and claims to provide a component-based foundation for future defense research.","tokens_in":29727,"tokens_out":4938,"duration_ms":42256,"significance":"If the quantitative claims were internally consistent, this would be a valuable contribution. The paper is commendable for attempting a systematic, PRISMA-structured review of a rapidly emerging area, for reporting inter-coder reliability statistics (Cohen's kappa 0.88 and 0.93), and for offering a taxonomy that organizes vulnerabilities by architectural function rather than by attack name. The seven open problems, especially the security of code-execution and embodied agents, are reasonable and useful for shaping future research. However, the headline numerical claims—the 3.9:1 attack-to-defense ratio and the 14x perception/action gap—are not reproducible from the paper's own tables and narrative, and the missing search strings and per-record coding details prevent independent audit. The central contribution is therefore currently undermined by reporting inconsistencies that must be resolved before the quantitative conclusions can be accepted.","major_comments":[{"comment":"The layer-coverage figures are internally inconsistent. Section 4.7 states that 56 of 85 papers (66%) discuss perception-layer risks, 35 (41%) brain-layer risks, 4 (4.7%) action-layer risks, and 24 (28%) interaction-layer risks, and the abstract and Table 11 repeat these paper-level counts. However, Table 4 reports component-level occurrence counts that sum to 95 for the perception layer (12+15+26+42), 109 for the brain layer (24+21+41+23), 82 for the action layer (54+7+13+8), and 17 for the interaction layer (12+5). The table note states that these are occurrence counts and that a single paper can contribute to multiple counts, but the paper never defines how occurrence counts are converted to the paper-level percentages used in Section 4.7 and the abstract. In particular, the claim that action-layer vulnerabilities 'appear in only 4.7%' of papers is not auditable when Table 4 alone shows 82 action-layer occurrences. The authors should report both units explicitly—unique-paper primary-layer counts and component-occurrence counts—and reconcile the abstract's prevalence language with the table's counting convention.","section":"Section 4.7, Table 4, Table 11"},{"comment":"The attack-to-defense ratio is not reproducible because the paper reports different defense counts in different places. Section 5.1 says 47 research papers (55.3%) focus on vulnerability discovery and attacks while 12 papers (14.1%) focus on defense and mitigation, yielding the headline 3.9:1 ratio; Table 11 repeats 47 and 12. Yet Section 6.1 states that the review identified 17 defense-focused studies (12 detection and 5 mitigation-only), and Section 7.1, Problem 4, refers to '10 detection methods vs 52 attacks' (a 5:1 ratio). These numbers cannot all describe the same corpus under one coding scheme. The authors must specify the mutually exclusive paper categories (attack, defense, framework, mitigation-only, detection-only), apply them consistently, and recompute the ratio; until then, the 3.9:1 claim in the abstract and conclusion is unsupported.","section":"Section 5.1, Section 6.1, Table 11, Section 7.1"},{"comment":"The review's reproducibility is insufficient for the strength of the quantitative claims. Section 2.4 describes the search strategy only at the level of query groups ('Group 1 AND Group 2 NOT Group 3') with illustrative terms, and it does not provide the full Boolean queries adapted to each of the six databases. PRISMA 2020 requires the complete search strategy, typically in an appendix or supplement, and the exclusion decisions are summarized only at the flow level (540 excluded at title/abstract, 71 at full-text) without per-record documentation. Because the inclusion criteria define the population from which every ratio in the paper is computed, the absence of these details is a load-bearing limitation. A supplement containing verbatim search strings, the full list of excluded records with reasons, and the coding protocol should be added.","section":"Section 2.4, Section 2.5"}],"minor_comments":[{"comment":"The organization paragraph states that the methodology is presented in Section 3, but the search and coding methodology actually appears in Sections 2.4--2.6; the cross-reference should be corrected.","section":"Section 1.6"},{"comment":"Figure 4 is described as a 'visual representation of Table 2,' but Table 2 maps existing agent frameworks to the four-layer taxonomy, while the figure appears to depict the vulnerability distribution in Table 3; the caption and table reference should be checked.","section":"Figure 4 caption"},{"comment":"Table 1 contains duplicate rows for references [81] and [125] with different years, which appears to be an editing artifact; the table should list each survey once with consistent metadata.","section":"Table 1"},{"comment":"The statement '10 detection methods vs 52 attacks' is not tied to any table or definition and does not match the 47 attack papers counted in Table 11; either define the unit of counting or correct the number.","section":"Section 7.1, Problem 4"},{"comment":"The text refers to 'LangeChain-based agents'; this should be 'LangChain-based agents.'","section":"Section 3.2"},{"comment":"The sentence 'This literature considered in this study spans January 2023 through December 2025' is ungrammatical; it should read 'The literature considered in this study spans...'.","section":"Section 1.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know: the paper's headline ratios do not reproduce from its own tables. Section 4.7 reports that only 4 papers (4.7%) discuss action-layer security risks, while Table 4 lists component-level occurrences: Tool Manipulation 54, Function Hijacking 7, Code Injection 13, Sandbox Escape 8 — 82 occurrences total. The table note explains that a single paper can contribute to multiple counts, but going from 82 occurrences to 4 primary papers requires an operational distinction between 'primary focus' and 'mentions' that the text never states. The abstract's 'appear in only 4.7%' invites a prevalence reading that Table 4 undercuts. Separately, Section 5.1 computes the attack/defense ratio as 47 attack papers vs 12 defense papers (3.9:1), but Section 6.1 states the review identified 17 defense-focused studies (12 detection plus 5 mitigation-only). 47/17 is 2.8:1. These are not stylistic quibbles; the quantitative synthesis is the paper's core contribution.\n\nWhat is genuinely useful: the four-layer architecture taxonomy (perception, brain, action, interaction) with 13 mapped vulnerability types is a sensible organizing device, and the paper correctly identifies underserved areas — code-execution agents, embodied agents, and tool-augmented systems. The authors report inter-rater agreement (kappa 0.88 and 0.93), which is more than most narrative surveys do. For a reader looking for a structured map of the space, the taxonomy alone is worth something.\n\nWhere it falls down: none of the quantitative claims can be audited as submitted. Search strings are described but not given verbatim (Section 2.4), inclusion/exclusion is summarized rather than documented per record, the list of 85 included papers is not provided, and there are duplicate references (e.g., [125] appears twice in Table 1; [27]/[28] are the same work in preprint and published forms). The internal contradictions above are the bigger issue. The stress-test note checks out; the paper's own tables are in tension with its abstract and body.\n\nBottom line: this deserves a serious referee because the topic matters and the taxonomy is useful, but the quantitative backbone needs a major repair. Add an appendix with search strings, full inclusion log, and per-paper codes, and reconcile the layer and defense counts. I would not cite the 3.9:1 or 14x numbers as they stand. Once fixed, the survey could be a solid reference.","headline":"The paper's headline ratios don't reproduce from its own tables; the taxonomy is useful but the quantitative synthesis needs major repair before it can be cited.","tokens_in":30214,"tokens_out":3264,"would_cite":false,"duration_ms":27315,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A systematic review of 85 papers on agentic LLM security finds the field structurally imbalanced: 66% of work targets perception-layer attacks, only 4.7% targets action-layer attacks, and attack research outnumbers defense research by 3.9…","keywords":["agentic LLM security","prompt injection","vulnerability taxonomy","systematic literature review","action-layer vulnerabilities","attack-defense imbalance","code-execution agents","containment"],"falsifier":"Re-run the same review with the full search strings, two independent coders on 100% of the corpus, and a pre-registered coding protocol; if the action-layer share rises above roughly 15% or the perception-layer share drops below 50%, the claimed fourteen-fold imbalance collapses. A cheaper check: take the 85 papers and count how many require a full execution environment (tools, code, sandbox) as part of their evaluation — if that number is large, the 'action layer is unstudied' claim is an artifact of how papers were classified.","tokens_in":29228,"feed_emoji":"🛡️","tokens_out":8117,"duration_ms":67138,"temperature":0.7,"pith_summary":"Agentic large language models — systems that plan, call tools, run code, and keep memory — can trigger irreversible real-world actions when a reasoning step is compromised. This paper argues that the security research community is not studying the risk in proportion to its consequences. A systematic review of 85 papers (2023–2025) finds that 66% focus on perception-layer attacks such as prompt injection and jailbreaking, while only 4.7% study action-layer vulnerabilities like tool misuse, code injection, and sandbox escape — the failures that actually change system state. Attack research outnumbers defense research by 3.9 to 1, and code-execution agents receive 3.5% of attention despite being the highest-consequence deployment. The paper offers a four-layer architectural taxonomy and a seven-problem research agenda centered on containment.","feed_headline":"AI agent security research skews 14-to-1 away from real risk","feed_subtitle":"Review of 85 papers: attack work tops defense 3.9:1 while risky execution-layer flaws get only 4.7% of attention.","key_machinery":"The load-bearing object is a four-layer architectural taxonomy — perception, brain, action, interaction — built by open-coding the 85-paper corpus into 13 vulnerability types, with every agent component assigned to exactly one layer by its primary function. The taxonomy's core move is to count, per layer, how many papers study that layer's vulnerabilities, which converts narrative review into quantitative gap analysis: the perception/action ratio, the attack/defense ratio, and the near-zero coverage of code-execution and embodied agents are all produced by this counting scheme. The companion mechanism is a systematic screening pipeline (743 records screened down to 85 studies) that defines the population the counts describe.","core_discovery":"The central claim is that agentic LLM security research is structurally misaligned with real-world risk. Coding 85 retained papers by primary attack mechanism, the authors find that perception-layer vulnerabilities (direct and indirect prompt injection, jailbreaking, adversarial perturbations) appear in 65.9% of papers, while action-layer vulnerabilities (tool manipulation, function hijacking, code injection, sandbox escape) appear in only 4.7% — a fourteen-fold gap. Defense-focused papers number 12 against 47 attack-focused ones, a 3.9:1 ratio, and code-execution agents appear in just 3 (3.5%) papers while embodied agents appear in none. The authors attribute the imbalance to methodological accessibility: perception attacks can be studied black-box with API access, whereas action-layer attacks require full execution environments. They interpret the insecurity itself as architectural coupling: because agent layers feed one another, a compromise at any layer can propagate, so the field's singular focus on input filtering is misplaced, and containment boundaries across all four layers are what needs building.","pith_inferences":["The 14-fold gap probably measures experimental convenience as much as genuine neglect: perception attacks need only an API, while action-layer research requires building full agent environments, so the ratio may narrow quickly once standardized execution-aware benchmarks become the norm.","A testable extension is to run the same coding scheme on the 2026 literature; if the action-layer share rises sharply, the imbalance is a lagging indicator, but if it stays below 10%, accessibility, not field maturity, is the binding constraint.","The taxonomy's component-based framing predicts that the next high-impact vulnerabilities will appear in the parts of the stack with no coverage at all — inter-agent message authentication and tool-invocation protocols — where the paper finds zero or near-zero studies.","The paper's own quality assessment rates real-world applicability as the weakest of its four quality dimensions, so many published defenses are unlikely to survive deployment and the reported success rates should be read as upper bounds."],"forward_implications":["If the imbalance is real, defensive research priorities are inverted: the attack classes with the most irreversible consequences (code injection, sandbox escape, tool misuse) currently receive the least investigation.","Tool-augmented agents, at 12% of the corpus despite being the dominant production deployment pattern, are a pending security liability as their adoption scales.","The absence of standardized benchmarks and adaptive-adversary evaluation means published detection and defense numbers are not comparable across studies and likely overstate robustness.","Implementing the paper's seven open problems amounts to a containment strategy: sandboxing code execution, privilege separation for tools, runtime policy engines, human-in-the-loop governance, and authenticated inter-agent communication.","Because failures propagate across layers, input filtering alone cannot secure an agent; defenses must be placed at each architectural boundary, including the action layer where verification is currently absent."],"supporting_citations":[{"why":"Supplies the thought-action-observation agent architecture that the four-layer taxonomy organizes and that most of the corpus builds on.","marker":"[149]"},{"why":"Benchmark for indirect prompt injection in tool-integrated agents; anchors the perception-layer attack class that dominates the corpus.","marker":"[155]"},{"why":"Dynamic benchmark environment for prompt-injection attacks and defenses; one of the datasets the review counts when quantifying perception-layer coverage.","marker":"[25]"},{"why":"Formalizes attack and defense categories for LLM-based agents and documents the absence of standardized evaluation protocols and cross-layer failure propagation.","marker":"[157]"},{"why":"Prior survey that the review positions against, and a cited source for code injection and sandbox escape as action-layer vulnerability examples.","marker":"[28]"},{"why":"Empirical study showing multi-agent systems can be made to execute arbitrary malicious code; the pivotal example for the action-layer coverage gap.","marker":"[120]"}],"fun_headline_variants":["AI agent security papers ignore real risks","Agentic AI risks: research split 14-to-1","Security research on agents: attacking perception, ignoring action","Agent vulnerability research skews toward easy targets","AI agent security: research gap vs real-world threats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ratios are only as good as the paper collection they count; the exact search strings used across the six databases are not reported, inclusion and exclusion decisions are summarized rather than documented per record, and only 17% of the corpus was independently coded by a second reviewer, so a different search or a different coder could shift the headline percentages.","fun_headline_variants_meta":{"raw":{"variants":["AI agent security papers ignore real risks","Agentic AI risks: research split 14-to-1","Security research on agents: attacking perception, ignoring action","Agent vulnerability research skews toward easy targets","AI agent security: research gap vs real-world threats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1303,"prompt_tokens":1020,"completion_tokens":283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":636,"tokens_out":283,"duration_ms":3419,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:17:59.758949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same review with the full search strings, two independent coders on 100% of the corpus, and a pre-registered coding protocol; if the action-layer share rises above roughly 15% or the perception-layer share drops below 50%, the claimed fourteen-fold imbalance collapses. A cheaper check: take the 85 papers and count how many require a full execution environment (tools, code, sandbox) as part of their evaluation — if that number is large, the 'action layer is unstudied' claim is an artifact of how papers were classified.","supporting_citations":[],"review_version":1}