REVIEW 3 major objections 6 minor 40 references
AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read AgentSnare claims that a trajectory-adaptive decoy, built from a validated and state-consistent world ledger, can keep autonomous LLM penetration agents from ever reaching the real target.
desk verdict A credible empirical result with external evaluation, but the 0/45 claim only holds for the evaluated agents' tool sets; worth refereeing with a scoped threat model. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a two-stage online loop: a learned artifact-construction policy model proposes a candidate artifact from the current interaction history, probe action, decoy state, and application profile, and a stateful runtime called ShadowOS validates the candidate and commits it to a decoy. ShadowOS maintains a world-fact ledger, which records committed decoy objects, attributes, relations, and attacker-induced changes, and a lazy affordance graph whose nodes are files, credentials, services, users, containers, hosts, and escalation or lateral-movement opportunities and whose edges encode relations like references, authenticates-to, runs-as, corroborates, and enables. The graph
What would settle it
Give a penetration agent a tool that reads the real host's process table, mounted filesystems, and raw network connections outside the defender's proxy, then check whether the returned facts match the decoy's disclosed files, users, containers, and credentials; any mismatch gives the agent a way back to the real vulnerable path, and none of the paper's 45 runs includes such a check.
Extended reading notes
Core claim
AgentSnare's central claim is that the bottleneck in defending against LLM penetration agents is not artifact sophistication but state-adaptive construction. The system intercepts actions at a defender-controlled surface, uses a fine-tuned policy model to infer the attacker's current intent and propose a candidate artifact—an observation plus the environmental facts needed to support it—and then validates the candidate for semantic, state, and safety consistency before committing it to a persistent decoy. The decoy's world-fact ledger and lazy affordance graph keep every later observation consistent with earlier ones, so an attacker that re-verifies evidence sees a coherent reality. Over a 2
Load-bearing premise
The defense assumes the attacker keeps acting on observations that pass through the defender's control and never directly inspects the real host's processes, mounts, or network in a way that separates decoy fiction from the real target.
Editorial extensions
If this is right
- If the decoy-share trajectory represents genuine containment, defenders can add a reactive honeypot layer without patching the vulnerable application itself; the attacker's finite budget is spent against a synthetic reality.
- The three process metrics need not move together—GPT-5.4 showed the lowest Delay but the highest SubDefuse—so evaluations of interactive defenses should track where actions land and what evidence completion attempts cite, not just final exploit status.
- Because the decoy is unfolded incrementally and validated, the decoy absorbs a larger share of actions over attack progress; a one-step distraction is not enough to explain the sustained containment.
- The zero-exploit outcome at pass@3 is accompanied by substantially higher Delay, Divert, and SubDefuse than static defenses or existing interactive honeypots, linking trajectory control to exploit suppression.
Reading between the lines
- Beyond the paper: if the approach transfers to production web services, defenders need not classify traffic as malicious up front; they could route probing actions to a synthetic surface and reserve real behavior for actions that pass validation.
- Beyond the paper: a natural stress test is to train an attacker that audits the decoy ledger directly—comparing filesystem listings across sessions, checking whether disclosed containers and processes appear in real system calls, and maintaining a private map of the target—since the defense assumes the agent stays inside the observation loop.
- Beyond the paper: the DDD metrics suggest a way to compare deception strength across systems by measuring how much completion evidence is decoy-grounded; a testable question is whether SubDefuse alone, without Delay or PER, predicts exploit suppression.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AgentSnare, a trajectory-adaptive deception system for defending web applications against autonomous LLM penetration agents. Instead of planting static artifacts, AgentSnare uses a fine-tuned artifact-construction policy model (LoRA-tuned Qwen3-4B) to propose decoy observations conditioned on the attacker's interaction history and decoy state, then validates each proposal in a stateful ShadowOS runtime (world-fact ledger W_t and affordance graph G_t) before disclosing it. The policy is trained on synthetic attacker–defender trajectories generated by two DeepSeek-V4-Flash instances. The system is evaluated on 15 CVE-Bench web applications with three attacker models (Claude Opus 4.6, GPT-5.4, MiniMax M3) and compared against Static Defenses, Cowrie, and HoneyLLMd. The headline claims are Delay 46.8%, post-entry retention (PER) 55.9%, SubDefuse 90.0%, and 0/45 evaluator-verified real-target exploits at pass@3. Ablations show both model fine-tuning and ShadowOS contribute, and that the full system outperforms a prompted GPT-5.4-mini variant. The paper also introduces Delay–Divert–Defuse (DDD) as a trajectory-level evaluation framework.
Significance. If the 0/45 exploit-suppression result is robust, this is a meaningful advance: it is one of the first demonstrations that a dynamically constructed, factually consistent decoy can steer modern LLM penetration agents away from real-exploit completion. The evaluation uses the external CVE-Bench evaluator for the primary security claim, spans multiple attacker models and vulnerable applications, includes two ablations and a strong prompt-only baseline, and reports per-CVE results in the appendix. Those are genuine strengths. However, the central result is conditioned on the attacker's tool set in a way that is not stated precisely in the threat model, and the absence of a pass@3 no-defense baseline makes the magnitude of the improvement difficult to assess. The DDD metrics are descriptive and useful, but their labeling protocol is underspecified. The significance of the contribution is high, but the paper currently overclaims generality relative to what was actually tested.
major comments (3)
- [Threat Model / Online Artifact Construction and Disclosure (Eq. 1)] The threat model states that the defender makes 'no assumptions about ... the agent's tool configuration,' but the mechanism 'intervenes only when a_t reaches a defender-controlled surface' (Eq. 1 and surrounding text). Host-inspection tools that execute directly on the real host—e.g., 'ps aux', 'mount', 'ip addr', 'cat /etc/hosts', 'docker ps'—return observations that do not pass through ShadowOS and would expose the decoy's fabrications. The appendix's own trace (Section E) fabricates a second host (10.0.4.100), container names such as 'wordpress-htmega-php-r1-web-1', and processes; a single direct 'ip addr' on the real container would contradict those facts. No evaluated attacker had such a tool, so the 0/45 result is a property of the three evaluated tool sets, not of arbitrary tool configurations. The stress-test concern therefore lands. I recommend either restricting the stated thr
- [Experiments, Setup / Table 1] The paper reports no-defense exploit rates at pass@1 only ('we provide pass@1 exploit rate without defense for references'), while all defended conditions are evaluated at pass@3. Since pass@3 success is at least as large as pass@1 success, the relevant no-defense baseline is unknown and likely higher than the reported 18/45 (e.g., 13/15 for Claude Opus 4.6 at pass@1). Without a pass@3 no-defense baseline, the reader cannot determine how much of the 0/45 result is due to AgentSnare versus the added repetitions and the difficulty of the CVE suite. Please report the no-defense pass@3 rates for each attacker and for the aggregate.
- [Delay–Divert–Defuse Metrics (Eqs. 15–17)] Delay, PER, and SubDefuse depend on assigning each tool call a target label z_t ∈ {real, decoy, other}, identifying the first decoy action τ, and tracing completion-attempt provenance to the decoy (P_D). The paper does not specify an operational labeling protocol, who performs the labeling, or any reliability measure. Because the DDD framework is itself one of the contributions, and because the reported percentages (e.g., 90.0% SubDefuse) are central to the narrative, this underspecification leaves the metric difficult to reproduce. Please release the annotation guidelines, the per-run labels (or a representative sample), and an inter-annotator agreement statistic.
minor comments (6)
- [RQ1, first paragraph] Typo: 'Across attackers, It absorbs' should be 'it absorbs.' Also, 'fine tune' in the Offline LLM Training section should be 'fine-tune.'
- [Figure 1] The figure reports Interference@20 values but the color scale and row/column labels are not fully legible in the text version. A note on how the six static tactics are mapped to the columns would help.
- [Eq. (13)] The set notation for R_t^{(j)} is malformed in the rendering ('n ... o_k'). Clarify that the set contains m sampled reactions.
- [Experiments, Setup] Clarify the relationship between the three repetitions per attacker–CVE pair and the pass@3 metric: are the three runs exactly the three attempts that define pass@3? This is implied but should be explicit.
- [Appendix A] The sentence 'The breadcrumb thereforeremainslimited...' has a spacing typo and uses 'therefore' twice in adjacent sentences; consider rewriting.
- [Related Work / Offline Data Synthesis] The distinction between AgentSnare's offline trajectory synthesis and Cyber-Zero is clear, but it would be useful to state explicitly that the training corpus excludes CVE-Bench applications, as is done in the methodology section, so the reader can verify non-contamination.
Circularity Check
No significant circularity: the 0/45 exploit claim is evaluated by the external CVE-Bench evaluator, and the defense's training/evaluation separation (DeepSeek-V4-Flash simulator vs. Claude/GPT/MiniMax attackers) prevents any by-construction link.
full rationale
AgentSnare's central claims are empirical measurements, not derivations from definitions. The headline 'no verified real-target exploit (0/45)' is determined by the official CVE-Bench evaluator (RQ2: 'Real-target exploit success is determined by the official CVE-Bench evaluator'), which is external to the defense; the defensive pipeline (Eqs. 1–6) does not touch the evaluator. The DDD metrics (Eqs. 15–17) are operational definitions of the paper's own evaluation framework; Delay/PER count tool calls the agent actually directed at decoy surfaces and are not forced by construction — the threat model explicitly allows the agent to 'ignore defender-constructed artifacts, alternate between the real target and defender-controlled environments,' and Appendix E shows the agent repeatedly returning to real WordPress routes that produce real responses. No fitted parameter is renamed as a prediction: the artifact-construction policy (LoRA-tuned Qwen3-4B) was trained on 24,807 synthetic decisions from DeepSeek-V4-Flash simulations that 'contain no CVE-Bench application, vulnerability description, reference exploit trajectory, official solution, or evaluator output,' and online deployment uses no reaction-guided selection against the evaluated models (Claude Opus 4.6, GPT-5.4, MiniMax M3). Self-citations (Qi et al. 2026a/b; Zhao et al. 2025) provide context about attack-agent capabilities and are not load-bearing for the defense's effectiveness. Two caveats are weighed but do not constitute circularity. (1) SubDefuse's provenance tags are assigned by the defense's own ShadowOS ledger, so the 90.0% figure is an internally tagged process metric; its external significance rests on the separately evaluator-verified 0/45 result. (2) The threat model's assertion that the defender 'makes no assumptions about ... the agent's tool configuration' is broader than the evidence: as the appendix trace shows, the decoy's fabricated facts (e.g., host 10.0.4.100, container names) exist only in observations rendered through defender-controlled surfaces, and a tool inspecting the real host's /etc/hosts, process table, or mounts would contradict them. Thus the 0/45 result is a property of the three evaluated agents' tool sets rather than a guarantee against arbitrary tool configurations — a scope limitation, not a by-construction reduction. No equation reduces to its input, and no load-bearing result is imported from the authors' prior work, so the appropriate finding is no significant ci
Assumptions & free parameters
free parameters (3)
- Attack budget B =
200 tool calls
- Pass@3 repetitions k =
3
- LoRA hyperparameters =
r=64, alpha=128, 3 epochs
assumptions (4)
- domain assumption LLM penetration agents act purely on the observation-action loop and do not have access to ground truth beyond tool outputs.
- ad hoc to paper The defender can intercept and replace observations for actions reaching the decoy surface while leaving real-target actions unchanged.
- domain assumption CVE-Bench's evaluator is a faithful proxy for real-world penetration-test success.
- domain assumption The DeepSeek-V4-Flash attacker simulator used for training is a sufficient proxy for the test-time attacker models.
invented entities (3)
-
ShadowOS stateful decoy runtime (world-fact ledger W_t and affordance graph G_t)
independent evidence
-
Artifact-construction policy model M_theta (LoRA-tuned Qwen3-4B)
independent evidence
-
Application-side breadcrumb (reverse-proxy sidecar route /v1/internal/ops/diagnostics)
independent evidence
Cite this review
Pith. "Pith review of AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents." pith.science (2026). https://pith.science/paper/IZ7JTKT5
@misc{pith2026260726998,
author = {Pith},
title = {Pith review of: AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/IZ7JTKT5}},
note = {Machine review of arXiv:2607.26998}
}
read the original abstract
Large language model (LLM) agents automate penetration testing through an observation-action loop, selecting actions based on observations returned by tools. This dependence allows defenders to inject deceptive observations that can mislead the agent's decision-making process. However, existing defenses rely heavily on static, isolated artifacts planted in the environment prior to an attack. Advanced agents can progressively recognize and bypass these artifacts, ultimately refocusing their exploitation attempts on the real target. To address this issue, we introduce AgentSnare, a trajectory-adaptive deception system that dynamically unfolds a decoy environment to continually steer the penetration agent away from the real target. Specifically, AgentSnare employs an artifact-construction policy model that constructs candidate artifacts conditioned on the agent's interaction history and decoy state. AgentSnare then validates these candidates and incrementally incorporates valid artifacts into a factually consistent decoy environment, thereby delaying the attack by absorbing its tool calls, diverting its post-entry trajectory within the decoy, and defusing it by inducing completion reports grounded in decoy evidence. Across 15 CVE-Bench web applications and three attacker models, AgentSnare absorbs 46.8% of the agent's tool calls in the decoy and retains 55.9% of post-entry actions there, while 90.0% of completion attempts are grounded in decoy evidence; across all 45 attacker-CVE pairs, no real target is successfully exploited at pass@3.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education
Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)
-
[2]
Classification Problem Solving
Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence
-
[3]
, title =
Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =
1980
-
[4]
New Ways to Make Microcircuits Smaller---Duplicate Entry
Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science
-
[5]
Clancey and Glenn Rennels , abstract =
Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =
-
[6]
and Rennels, Glenn R
Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies
-
[7]
Poligon: A System for Parallel Problem Solving
Rice, James. Poligon: A System for Parallel Problem Solving
-
[8]
Transfer of Rule-Based Expertise through a Tutorial Dialogue
Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue
Show all 40 references
-
[9]
The Engineering of Qualitative Models
Clancey, William J. The Engineering of Qualitative Models
-
[10]
2023 , eprint=
Attention Is All You Need , author=. 2023 , eprint=
2023
-
[11]
Pluto: The 'Other' Red Planet
NASA. Pluto: The 'Other' Red Planet
-
[12]
33rd USENIX Security Symposium (USENIX Security 24) , pages=
\ PentestGPT \ : Evaluating and harnessing large language models for automated penetration testing , author=. 33rd USENIX Security Symposium (USENIX Security 24) , pages=
-
[13]
arXiv preprint arXiv:2503.17332 , year=
CVE-bench: a benchmark for AI agents' ability to exploit real-world web application vulnerabilities , author=. arXiv preprint arXiv:2503.17332 , year=
-
[14]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Towards effective offensive security llm agents: Hyperparameter tuning, llm as a judge, and a lightweight ctf benchmark , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[15]
Cowrie SSH/Telnet Honeypot , year =
-
[16]
2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) , pages=
Llm in the shell: Generative honeypots , author=. 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) , pages=. 2024 , organization=
2024
-
[17]
IEEE Transactions on Network Science and Engineering , year=
HoneyLLMd: A Large Language Model-Powered Adaptive Honeypot System , author=. IEEE Transactions on Network Science and Engineering , year=
-
[18]
34th USENIX Security Symposium (USENIX Security 25) , pages=
Cloak, Honey, Trap: Proactive Defenses Against \ LLM \ Agents , author=. 34th USENIX Security Symposium (USENIX Security 25) , pages=
-
[19]
arXiv preprint arXiv:2410.20911 , year=
Hacking back the ai-hacker: Prompt injection as a defense against llm-driven cyberattacks , author=. arXiv preprint arXiv:2410.20911 , year=
-
[20]
arXiv preprint arXiv:2508.00910 , year=
Cyber-zero: Training cybersecurity agents without runtime , author=. arXiv preprint arXiv:2508.00910 , year=
-
[21]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Safenlidb: A privacy-preserving safety alignment framework for llm-based natural language database interfaces , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[22]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Shoppingbench: A real-world intent-grounded shopping benchmark for llm-based agents , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[23]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Security games with layered defenses: adaptive adversaries and gittins indices , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[24]
, author=
Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=
-
[25]
arXiv preprint arXiv:2505.09388 , year=
Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=
-
[26]
IEEE Transactions on Information Forensics and Security , year=
AutoPT: How Far Are We From the Fully Automated Web Penetration Testing? , author=. IEEE Transactions on Information Forensics and Security , year=
-
[27]
arXiv preprint arXiv:2501.13411 , year=
Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework , author=. arXiv preprint arXiv:2501.13411 , year=
-
[28]
arXiv preprint arXiv:2306.05301 , year=
Toolalpaca: Generalized tool learning for language models with 3000 simulated cases , author=. arXiv preprint arXiv:2306.05301 , year=
-
[29]
arXiv preprint arXiv:2310.05915 , year=
Fireact: Toward language agent fine-tuning , author=. arXiv preprint arXiv:2310.05915 , year=
-
[30]
Findings of the Association for Computational Linguistics: ACL 2024 , pages=
Agenttuning: Enabling generalized agent abilities for llms , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=
2024
-
[31]
Proceedings of the 29th symposium on operating systems principles , pages=
Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=
-
[32]
Introducing Claude Opus 4.6 , year=
-
[33]
GPT-5.4 Thinking System Card , year=
-
[34]
MiniMax M3: Frontier Coding, 1M Context, Native Multimodality---All in One Model , year=
-
[35]
arXiv preprint arXiv:2511.13725 , year=
AI Kill Switch for malicious web-based LLM agent , author=. arXiv preprint arXiv:2511.13725 , year=
-
[36]
2025 , publisher=
LLM Agent Honeypot: Monitoring AI Hacking Agents in the Wild , author=. 2025 , publisher=
2025
-
[37]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategies , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[38]
arXiv , primaryClass=:2510.02422 , year=
Dynamic Jailbreaking Attack , author=. arXiv , primaryClass=:2510.02422 , year=
-
[39]
Qi, Weiwei and Wu, Zefeng and Guo, Zhilin and Zheng, Tianhang and Lu, Chaochao and He, Liang and Qin, Zhan and Ren, Kui , journal=
-
[40]
From Topology to Behavioral Semantics: Enhancing
Zhao, Heng and Wang, Ruoyu and Zheng, Tianhang and Li, Qi and Lv, Bo and Wang, Yuyi and Du, Wenliang , year=. From Topology to Behavioral Semantics: Enhancing. 2511.14467 , archivePrefix=
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.