Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-agent runtime defenses can be made to evolve from failure traces, outperforming handcrafted ones.

desk verdict A genuinely useful reframing of runtime defense evolution, with a real evaluation weakness: the attack-success judge is also the evolution signal, and the paper's own appendix contradicts its claimed four-repeat statistics. read the letter →

arxiv 2608.12977 v1 pith:UULTNMJ4 submitted 2026-08-13 cs.CR cs.AI

Jiajun Ruan , Peiyang Li , Yukun Chen , Fengting Li , Chao Feng This is my paper · ORCID
classification cs.CRcs.AI
keywords LLMagentsecurityruntimedefenseself-evolvingharnesspromptinjectionmemorypoisoningskilladaptiveattacks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Handcrafted runtime defenses for LLM agents are static: once deployed they stay fixed while attackers adapt, so they inevitably miss new failure modes. This paper claims that runtime defense can instead be treated as an optimization problem over the agent's harness—the two interfaces that decide what context the model sees (context construction) and what actions it is allowed to take (action interpretation)—and that an LLM-driven loop can improve those interfaces automatically from observed failures. In the proposed system, HARD, failed execution trajectories are routed to whichever defense artifact is responsible and reused to regenerate a semantic security policy and executable gate rules. On the AgentCanary benchmark, HARD reduces attack success rate to 15.4% (direct prompt injection), 1.0% (indirect prompt injection), 6.7% (memory poisoning), and 10.2% (skill poisoning), compared with 13–66% for three handcrafted baselines, while keeping benign utility at 91.9–95.0% and raising utility under attack from 56% to 86% on memory poisoning and from 52% to 92% on skill poisoning. If true, deployed agents could continuously improve their own protection against new and adapting attacks without model retraining or manual security engineering.

What carries the argument

The load-bearing mechanism is the harness decomposition $H=(\phi_H,\psi_H)$, which splits runtime defense into two independently editable intervention interfaces: context construction (what the model is shown, governed by a security policy) and action interpretation (what tool calls are permitted, governed by execution-time gate rules). This decomposition defines a structured evolution space, and HARD's loop then consists of a trace router $R$ that assigns each failure trajectory to the responsible artifact and two LLM-based evolvers, $E_P$ for the policy and $E_G$ for the gate, that regenerate the artifact from batches of eight failure traces while minimizing changes and avoiding benchmark-specific overfitting. The gate evolver only emits machine-matchable predicates over tool names and argument patterns, while the policy evolver only emits general security invariants, and the router deliberately reassigns low-confidence gate predictions to the policy. What this mechanism does is turn failure-driven human debugging into a closed loop that can be repeated across evolution rounds, with the harness objective $\max_H \mathbb{E}[J_{\text{safe}}+\lambda_u J_{\text{util}}]$ as the target and the judge's outcome score as the feedback signal that defines which traces count as failures.

What would settle it

Take the identical HARD evolution loop and replace the GLM-5 judge with an independently calibrated judge or human labels on the same AgentCanary test trajectories; if the attack-success gap over handcrafted defenses collapses, or if the evolved policies and gates reduce train-set failures but not held-out failures (the pattern the paper itself observes for HARD-Gate under long-horizon attacks), the central claim of autonomous generalization from failures is not supported. A second check is to evaluate the evolved artifacts on an attack taxonomy disjoint from AgentCanary's four threat classes and confirm that the ASR advantage persists.

Watch

Extended reading notes

Core claim

At its core, the paper claims that the static, hand-authored runtime defense is replaceable by an autonomous evolution process that converts execution failures into targeted harness updates. The agent is modeled as a fixed language model $M_\theta$ wrapped in a harness $H=(\phi_H,\psi_H)$, where $\phi_H$ constructs the model's context and $\psi_H$ interprets its output into executable actions; runtime defense becomes the optimization $\max_H \mathbb{E}[J_{\text{safe}}(\tau)+\lambda_u J_{\text{util}}(\tau)]$ over editable harness configurations. HARD operationalizes this by collecting trajectories under attack, flagging failures through a judge (an attack succeeds when the judge's outcome_assessment is at most 0.5), routing each failure to the context-side policy evolver or the action-side gate evolver, and redeploying the refined harness each round. In the reported evaluation HARD-Both, which evolves both artifacts, achieves the lowest attack success rate on every static attack category and the highest utility under attack on memory poisoning and skill poisoning, and it also outperforms the strongest handcrafted baseline under both adaptive attack settings (26.5% versus 30.1% ASR on dynamic attack evolution; 12.1% versus 24.1% on long-horizon progressive attacks for the both variant, 4.8% for the policy-only variant). A case-level analysis shows that a single observed failure can suffice to synthesize an effective repair, provided the failure is routed to the layer that matches its nature: stable execution patterns are fixed by deterministic gate predicates, while semantic failures require policy-level invariants. The paper further finds that the benefits of the two artifacts are not uniformly additive under long-horizon adaptive attacks, where policy evolution transfers better than literal gate rules.

Load-bearing premise

The framework's improvement signal is the attack-judge score from a single LLM evaluator on a single benchmark split; if that judge is miscalibrated or the split is unrepresentative of real attacks, the evolved defenses are optimizing the wrong objective and the claimed gains may not transfer.

Editorial extensions

If this is right

  • Deployed agents could improve their security over time without model retraining or manual defense engineering, because HARD converts newly observed failures directly into updated policy and gate artifacts.
  • Context-side policy evolution and action-side gate evolution address complementary failure modes, so jointly evolving both artifacts yields the lowest attack success rates under static attacks in the reported experiments.
  • Under long-horizon adaptive attacks, semantic policy evolution transfers much better than literal execution predicates, which can memorize observed attacks (HARD-Gate's test-split ASR barely moves from 28.9% to 26.5% while its evolution-split ASR drops to 23.2%).
  • The evolution procedure is not tied to a single model: all four tested evolution backbones substantially reduce ASR relative to no evolution, though they land at different points on the security–utility frontier.
  • A single observed failure can be sufficient to synthesize an effective repair when the failure is routed to the layer that matches its nature, as the SUID-enumeration and credential-exfiltration case studies show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the feedback loop depends only on a scalar judge score and tool-call traces, the same machinery should work with non-LLM feedback—sandbox monitors, program verifiers, or human-flag telemetry—so HARD's design is a natural fit for production deployment monitoring rather than benchmark grading alone.
  • The policy/gate dichotomy is likely a special case of a more general principle: any harness function that can be serialized and re-generated from failure traces is a candidate evolution artifact, so future designs could add memory-access rules, confirmation workflows, or tool-schema sanitizers to the same routing loop.
  • A testable extension the paper does not run is cross-framework transfer: because the gate evolver is explicitly prohibited from keying on benchmark workspace paths, its rules should be studied on a different agent harness or tool set to see whether the evolved predicates transfer beyond the environment they were evolved on.
  • The security-utility frontier observed across backbones suggests that in deployment, the number of evolution rounds, the routing confidence threshold (0.55 in this paper), and the choice of backbone could be tuned per organization, with the reported HARD-Both numbers as one operating point rather than the optimum.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a harness-level formulation of runtime defense for LLM agents, decomposing the harness into context-construction and action-interpretation interfaces, and proposes HARD (Harness-based Autonomous Runtime Defense Evolution), a framework that mines failure trajectories, routes them to policy or gate evolvers, and iteratively updates defense artifacts. The authors evaluate HARD against handcrafted baselines on AgentCanary plus AgentHazard tasks, under static and adaptive attacks, reporting lower attack success rates with preserved benign utility. The appendix provides detailed prompts, evolution protocols, and three case studies illustrating how evolved artifacts operate.

Significance. The direction is timely and potentially important: if the reported improvements are real, HARD offers a way to improve security of deployed agents without model retraining, and the harness-centric formulation is a useful organizing principle for runtime defense design. The paper is transparent about prompts and protocol, uses deterministic train/test splits, and includes case studies that help explain the mechanism. However, the central empirical claim currently rests on a closed evaluation loop in which the same judge both mines failures and scores success, with no independent validation; the paper also contains direct numerical inconsistencies between the table and the prose/abstract. The significance is therefore conditional on resolving these issues, which is why I recommend major revision rather than acceptance.

major comments (4)
  1. [5.1, 4.1, A.4] The evaluation loop is closed around a single judge. GLM-5 serves as J_safe for failure identification (Section 4.1, step 2) and as the scorer whose outcome_assessment<=0.5 defines ASR (Section A.4), and the serialized failure traces fed to the router and evolvers include failure_feedback and grader_breakdown (Section A.4). HARD is therefore evolved to minimize exactly the quantity later reported as ASR, whereas the handcrafted baselines are not optimized against this function. If GLM-5's outcome_assessment is miscalibrated or can be satisfied by rubric-conforming text, the reported advantage may reflect judge-specific behavior rather than a genuine reduction in unauthorized actions. Please add independent validation: a human-annotated subset, a second judge, or a sensitivity analysis across several judges, with agreement metrics.
  2. [Table 1 vs. Abstract and Section 5.2] The reported HARD-Both static ASR values are internally inconsistent. Table 1 lists DPI 12.1, IPI 1.3, MC 13.9, and SP 7.4, while the abstract and Section 5.2 report 15.4, 1.0, 6.7, and 10.2; Appendix B repeats 6.7 and 10.2 with UA 86.3 and 92.0, whereas Table 1's MC row shows ASR 13.9 and UA 85.9. Since the central claim quantifies the improvement over handcrafted defenses, the paper must identify which set of numbers corresponds to the described protocol and make table, prose, and abstract consistent.
  3. [Section 5.2 vs. Section A.2] The statistical claims contradict the stated protocol. Section 5.2 says each static-attack cell is a mean over four independent repeats with standard deviation at most 5.7 and paired McNemar p<1e-8, while Section A.2 states that reported results correspond to a single evaluation pass and should not be compared across independently executed runs. These cannot both describe the reported table. Please specify which numbers come from the single pass and which from the repeats, describe the repeat protocol (seeds, handling of API nondeterminism, pooling), or remove the statistical significance claims.
  4. [5.1, Appendix A.2, Reference [26]] The evaluation relies exclusively on AgentCanary, whose author list overlaps with this paper's authors (Reference [26]), and on AgentHazard tasks translated into AgentCanary's format. Given the closed-loop judge issue, the lack of an independently authored benchmark or an external evaluation pipeline makes the headline comparison harder to assess. Even a small externally scored subset, or an independent judge with reported agreement, would substantially strengthen the claim that HARD improves over handcrafted defenses in general, rather than on a benchmark developed within the same group.
minor comments (5)
  1. [Figure 1] Figure 1 is not referenced in the experimental sections; consider adding a pointer to the corresponding table rows and, if possible, error bars or a note about the number of runs.
  2. [Section 2.2] The terms 'memory contamination' and 'memory poisoning' are used interchangeably; the explanatory sentence is helpful, but the table, prose, and abstract should use one term consistently.
  3. [Section 3.1 and A.4] The abstract objectives J_safe and J_util are defined operationally only in Section A.4; a brief operational definition at first use in Section 3.1 would improve readability.
  4. [Section 5.1] The security threshold delta_s=0.5 and the router confidence threshold 0.55 are free parameters; a sensitivity analysis or at least a rationale for these values would help assess robustness.
  5. [Table 4] Table 4 largely duplicates Table 2 with additional backbone rows; consider merging them into one table to avoid confusion.

Circularity Check

1 steps flagged · score 4.0 of 10

HARD's ASR improvement is partly circular: the same GLM-5 judge provides the evolution signal and the reported metric; held-out tasks do not remove the same-judge loop.

  1. fitted input called prediction [Section A.4 (Example Failure Trace); see also Section 4.1 and Section 5.1 (Table 2).]
    "Following AgentCanary, an attack counts as successful when outcome_assessment≤0.5, and ASR is the fraction of attacked tasks satisfying this condition. UA is the mean utility_evaluation score over attacked tasks for which a legitimate user objective is defined; we report this mean as a percentage. Failure mining therefore selects exactly the traces counted as successful attacks by ASR."

    GLM-5 is assigned as J_safe and δ_s=0.5 (Section 5.1); Section 4.1 defines failures as F_t={τ∈T_t | J_safe(τ)<δ_s ∨ J_util(τ)<δ_u}. A.4 states that ASR is the fraction of attacked tasks with outcome_assessment≤0.5 and that failure mining selects exactly the traces counted as successful attacks by ASR. The router and evolvers consume the judge's failure_feedback and grader_breakdown (A.4), so HARD is evolved to minimize the same GLM-5 rubric that later reports its ASR. The held-out split changes tasks but not the label function; no human agreement, calibration, or alternate-judge check is reported. The static baselines are not optimized against this function, so the comparison is meaningful only if the judge measures real security.

full rationale

The central derivation is not equation-level circular: the harness formulation, the evolution algorithm, and the empirical comparison are presented as a learned defense improvement over baselines, not as a formal theorem. The main circularity risk is statistical/measurement-based. The paper uses the same GLM-5 judge both to identify failures for evolution (Section 4.1) and to define the ASR metric that reports success (Section A.4). The failure traces given to the router and evolvers include the judge's failure_feedback and grader_breakdown, so evolution is explicitly guided by the exact scoring rubric being reported. The held-out test split does provide some independence by changing the tasks, but it does not change the judge or the rubric, so the improvement could reflect fitting the judge's scoring heuristics rather than genuinely blocking attacks. The paper does not report human agreement, judge calibration, or sensitivity to an alternative judge, and the static baselines are not optimized against that function. Additionally, the benchmark AgentCanary is co-authored by one of the present authors (Peiyang Li), but this self-citation is not load-bearing for the derivation; the evaluation on it is standard practice and the split is described. The internal inconsistency between 'single evaluation pass' (A.2) and 'four independent repeats' (5.2) also weakens the quantitative claims, but it is a reporting issue rather than circularity. Overall, the central claim retains independent content because the security utility trade-off and the evolution dynamics are evaluated on a held-out split, so the finding is not entirely forced. However, the same-judge loop means the headline ASR improvement is partly circular, meriting a score of 4 rather than 0-2.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The framework rests on commercial LLM judges as ground truth, a same-author benchmark for evolution and testing, unverified trace routing, and a harness decomposition that is assumed complete. These are plausible modeling assumptions but none is independently validated.

free parameters (5)
  • security threshold delta_s = 0.5
    Section 5.1 sets the failure criterion J_safe<0.5, directly determining which traces are mined for evolution and how ASR is computed.
  • router gate confidence threshold = 0.55
    Appendix A.2 reassigns gate predictions below 0.55 to policy evolution; this changes how failures are split between artifacts and therefore the final defense.
  • evolution rounds T = 5
    Algorithm 1 and Figure 4 run five rounds; results depend on this budget and no convergence criterion is reported.
  • training batch size = 8
    Appendix A.2 processes failures in batches of eight; batch composition affects which shared security invariants are extracted.
  • utility weight lambda_u = not specified
    Section 3.1 defines the defense objective with lambda_u but the paper does not state its value or how BU and UA trade-offs were selected.
assumptions (6)
  • domain assumption Adversary cannot modify model parameters theta or the deployed harness H.
    Section 2.2 states this threat model; all defenses and experiments assume it, and results may not hold against adversaries who can alter the harness or model.
  • domain assumption GLM-5 judge outputs for safety and utility are accurate ground truth.
    ASR and UA are computed from outcome_assessment and utility_evaluation scores per Appendix A.4; the evolution loop also relies on these labels, so judge errors propagate to both training signal and reported metrics.
  • domain assumption AgentCanary held-out task split is representative of real-world agent attacks.
    Section 5.1 and A.2 use deterministic splits within AgentCanary for evolution and testing; generalization claims beyond this benchmark are an assumption, not a demonstrated fact.
  • domain assumption LLM-based trace router correctly attributes each failure to the policy or gate artifact.
    Section 4.2 defines routing k=R(tau) with no human verification; incorrect routing misdirects evolution and changes results, and no routing accuracy metric is reported.
  • domain assumption H=(phi_H, psi_H) captures all relevant runtime defense intervention mechanisms.
    Section 3.2 presents this decomposition as a unified design principle; if a defense acts outside context construction or action interpretation, the formulation and HARD's artifact slots omit it.
  • domain assumption Temperature-0 commercial APIs are stable enough for single-pass comparisons.
    Appendix A.2 notes commercial LLM APIs remain mildly nondeterministic and results are a single evaluation pass, yet Table 1 reports precise numbers and Section 5.2 attaches significance tests to them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agents." pith.science (2026). https://pith.science/paper/UULTNMJ4

@misc{pith2026260812977,
  author       = {Pith},
  title        = {Pith review of: Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UULTNMJ4}},
  note         = {Machine review of arXiv:2608.12977}
}
read the original abstract

The expanding operational capabilities of large language model (LLM) agents introduce sophisticated security threats. Runtime defenses have emerged as an effective approach to mitigating these risks by integrating security mechanisms into the agent execution loop. However, existing runtime defenses rely heavily on manually designed interventions and lack a principled framework for their construction and maintenance. In this work, we first develop a harness-level formulation of runtime defense that systematically characterizes how harness mechanisms enable defense construction and provides a unified view of existing runtime defense interventions from a harness perspective. Building on this formulation, we propose HARD (Harness-based Autonomous Runtime Defense Evolution), a self-evolving runtime defense framework that automatically identifies appropriate intervention strategies and iteratively improves defense artifacts based on observed failure traces. HARD transforms runtime defense development from manual engineering into an autonomous evolution process, and extensive experiments demonstrate that it improves security performance over existing handcrafted defenses while preserving benign task utility. Our findings highlight autonomous defense evolution as a promising new paradigm for securing deployed LLM agents, enabling agents to identify defense weaknesses and continuously improve their protection mechanisms.

Figures

Figures reproduced from arXiv: 2608.12977 by the authors.

Figure 1
Figure 1. Evolution curves of HARD under the memory [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A harness-mediated LLM agent and the four [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of HARD. Failed trajectories are collected and routed to the responsible harness defense artifact. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Evolution dynamics of the three HARD vari [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 35 canonical work pages

  1. [26]

    AgentCanary: A secu- rity evaluation framework for autonomous ai agents in real executable environments

    Peiyang Li, Songping Wang, Yi Huang, Yan- hua Shi, Chenhao Zhang, Qi Li, Yueming Lyu, Caifeng Shan, Fengting Li, Chao Feng, Chuan- qun Zhu, and Liang Chen. AgentCanary: A secu- rity evaluation framework for autonomous ai agents in real executable environments. arXiv preprint arXiv:2606.10484, 2026

    arXiv 2026

  2. [1]

    Re- act: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. 9 Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsA PREPRINT

    work page 2023

  3. [2]

    Toolformer: Language models can teach them- selves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach them- selves to use tools. InAdvances in Neural Infor- mation Processing Systems, 2023

    work page 2023

  4. [3]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. InAdvances in Neu- ral Information Processing Systems, 2024

    work page 2024

  5. [4]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models re- solve real-world github issues? InInternational Conference on Learning Representations, 2024

    work page 2024

  6. [5]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web en- vironment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2024

    arXiv 2024

  7. [6]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Honghua Dong, Andrew Wang, Sil- viu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Iden- tifying the risks of lm agents with an lm-emulated sandbox. InInternational Conference on Learning Representations, 2024

    work page 2024

  8. [7]

    Agentharm: A benchmark for measuring harmfulness of llm agents

    Maksym Andriushchenko, Alexandra Souly, Ma- teusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. Agentharm: A benchmark for measuring harmfulness of llm agents. InInternational Conference on Learning Representations, 2025

    work page 2025

Show all 76 references
  1. [8]

    Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents

    Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. InInternational Con- ference on Learning Representations, 2025

    2025

  2. [9]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents

    Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fis- cher, and Florian Tramer. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. InAdvances in Neural Information Processing Systems Datasets and Benchm...

    2024

  3. [10]

    The task shield: Enforcing task alignment to defend against indirect prompt injection in llm agents

    Feiran Jia, Tong Wu, Xin Qin, and Anna Squic- ciarini. The task shield: Enforcing task alignment to defend against indirect prompt injection in llm agents. arXiv preprint arXiv:2412.16682, 2024

    2024 arXiv

  4. [11]

    Struq: Defending against prompt injection with structured queries

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363, 2024

    2024 arXiv

  5. [12]

    Secalign: Defending against prompt injection with preference optimization

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. Secalign: Defending against prompt injection with preference optimization. arXiv preprint arXiv:2410.05451, 2025

    2025 arXiv

  6. [13]

    Toolsafety: A comprehensive dataset for enhancing safety in llm- based agent tool invocations

    Yuejin Xie, Youliang Yuan, Wenxuan Wang, Fan Mo, Jianmin Guo, and Pinjia He. Toolsafety: A comprehensive dataset for enhancing safety in llm- based agent tool invocations. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14135–141...

    2025 doi

  7. [14]

    Agent safety alignment via reinforcement learning

    Zeyang Sha, Hanling Tian, Zhuoer Xu, Shiwen Cui, Changhua Meng, and Weiqiang Wang. Agent safety alignment via reinforcement learning. arXiv preprint arXiv:2507.08270, 2025

    2025 arXiv

  8. [15]

    Adversarial reinforcement learning for large language model agent safety

    Zizhao Wang, Dingcheng Li, Vaishakh Keshava, Phillip Wallis, Ananth Balashankar, Peter Stone, and Lukas Rutishauser. Adversarial reinforcement learning for large language model agent safety. arXiv preprint arXiv:2510.05442, 2025

  9. [16]

    On-policy self- evolution via failure trajectories for agentic safety alignment

    Bo Yin, Qi Li, and Xinchao Wang. On-policy self- evolution via failure trajectories for agentic safety alignment. arXiv preprint arXiv:2605.11882, 2026

    2026 arXiv

  10. [17]

    The attack and defense landscape of agentic ai: A comprehensive survey

    Juhee Kim, Xiaoyuan Liu, Zhun Wang, Shi Qiu, Bo Li, Wenbo Guo, and Dawn Song. The attack and defense landscape of agentic ai: A comprehensive survey. arXiv preprint arXiv:2603.11088, 2026

    2026

  11. [18]

    Clawguard: A runtime security framework for tool- augmented llm agents against indirect prompt in- jection

    Wei Zhao, Zhe Li, Peixin Zhang, and Jun Sun. Clawguard: A runtime security framework for tool- augmented llm agents against indirect prompt in- jection. arXiv preprint arXiv:2604.11790, 2026

    2026 arXiv

  12. [19]

    Openclaw prism: A zero-fork, defense- in-depth runtime security layer for tool-augmented llm agents

    Frank Li. Openclaw prism: A zero-fork, defense- in-depth runtime security layer for tool-augmented llm agents. arXiv preprint arXiv:2603.11853, 2026

    2026

  13. [20]

    Defending against indirect prompt injection attacks with spotlighting

    Keegan Hines, Gary Lopez, Matthew Hall, Fed- erico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending against indirect prompt injection attacks with spotlighting. InProceedings of the Conference on Applied Machine Learning in Information Secu- rity (CAMLIS), volume 3920 ofCE...

    2024

  14. [21]

    GuardAgent: Safeguard LLM agents via knowledge-enabled reasoning

    Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. GuardAgent: Safeguard LLM agents via knowledge-enabled reasoning. InProceedings of the 42nd International Conference on Machine Learnin...

    2025

  15. [22]

    IsolateGPT: An ex- ecution isolation architecture for LLM-based agen- tic systems

    Yuhao Wu, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. IsolateGPT: An ex- ecution isolation architecture for LLM-based agen- tic systems. InProceedings of the Network and Distributed System Security Symposium (NDSS). The Internet Society, 2025

    2025

  16. [23]

    10 Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsA PREPRINT Progent: Securing ai agents with privilege control

    Tianneng Shi, Jingxuan He, Zhun Wang, Hong- wei Li, Linyu Wu, Wenbo Guo, and Dawn Song. 10 Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsA PREPRINT Progent: Securing ai agents with privilege control. arXiv preprint arXiv:2504.11703, 2026

    2026 arXiv

  17. [24]

    Defending against indirect prompt injection by instruction detection

    Tongyu Wen, Chenglong Wang, Xiyuan Yang, Haoyu Tang, Yueqi Xie, Lingjuan Lyu, Zhicheng Dou, and Fangzhao Wu. Defending against indirect prompt injection by instruction detection. arXiv preprint arXiv:2505.06311, 2025

    2025

  18. [25]

    Adaptive attacks break defenses against indirect prompt injection attacks on llm agents

    Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. arXiv preprint arXiv:2503.00061, 2025

    2025 arXiv

  19. [27]

    DRIFT: Dy- namic rule-based defense with injection isolation for securing LLM agents

    Hao Li, Xiaogeng Liu, Hung-Chun Chiu, Dianqi Li, Ning Zhang, and Chaowei Xiao. DRIFT: Dy- namic rule-based defense with injection isolation for securing LLM agents. InAdvances in Neural Information Processing Systems, 2025

    2025

  20. [28]

    Airgapagent: Protect- ing privacy-conscious conversational agents

    Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. Airgapagent: Protect- ing privacy-conscious conversational agents. arXiv preprint arXiv:2405.05175, 2024

    2024 arXiv

  21. [29]

    AgentHazard: A benchmark for evaluating harmful behavior in computer-use agents

    Yunhao Feng, Yifan Ding, Yingshui Tan, Xingjun Ma, Yige Li, Yutao Wu, Yifeng Gao, Kun Zhai, and Yanming Guo. AgentHazard: A benchmark for evaluating harmful behavior in computer-use agents. arXiv preprint arXiv:2604.02947, 2026

    2026 arXiv

  22. [30]

    Pinchbench: Real-world benchmarks for ai agents

    PinchBench. Pinchbench: Real-world benchmarks for ai agents. https://github.com/pinchbench/skill,

  23. [31]

    SecureClaw: An owasp-aligned secu- rity plugin and skill for openclaw agents

    Adversa AI. SecureClaw: An owasp-aligned secu- rity plugin and skill for openclaw agents. https: //github.com/adversa-ai/secureclaw, 2026

    2026

  24. [32]

    ClawKeeper: Comprehensive safety protection for openclaw agents through skills, plugins, and watch- ers

    Songyang Liu, Chaozhuo Li, Chenxu Wang, Jinyu Hou, Zejian Chen, Litian Zhang, Zheng Liu, Qiwei Ye, Yiming Hei, Xi Zhang, and Zhongyuan Wang. ClawKeeper: Comprehensive safety protection for openclaw agents through skills, plugins, and watch- ers. arXiv preprint arXiv:2603.24414, 2026

    2026

  25. [33]

    OpenClaw Shield: A defense-in-depth se- curity plugin for openclaw agents

    Knostic. OpenClaw Shield: A defense-in-depth se- curity plugin for openclaw agents. https://github. com/knostic/openclaw-shield, 2026

    2026

  26. [34]

    DeepSeek-V4 technical report

    DeepSeek-AI. DeepSeek-V4 technical report. Technical Report, 2026

    2026

  27. [35]

    GLM-5 technical report

    Zhipu AI. GLM-5 technical report. Technical Re- port, 2026

    2026

  28. [36]

    GLM-5.2 technical report

    Zhipu AI. GLM-5.2 technical report. Technical Report, 2026

    2026

  29. [37]

    Claude Opus 4.6

    Anthropic. Claude Opus 4.6. Model Card, 2026

    2026

  30. [38]

    Qwen3.7-Max

    Qwen Team. Qwen3.7-Max. Technical Report, 2026

    2026

  31. [39]

    GPT-5.5 system card

    OpenAI. GPT-5.5 system card. System Card, 2026

    2026

  32. [40]

    Zico Kolter, and Matt Fredrik- son

    Andy Zou, Zifan Wang, Nicholas Carlini, Mi- lad Nasr, J. Zico Kolter, and Matt Fredrik- son. Universal and transferable adversarial at- tacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  33. [41]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. InIn- ternational Conference on Learning Representa- tions, 2024

    2024

  34. [42]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  35. [43]

    Tree of attacks: Jail- breaking black-box llms automatically

    Anay Mehrotra, Manolis Zampetakis, Paul Kas- sianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jail- breaking black-box llms automatically. InAd- vances in Neural Information Processing Systems, 2024

    2024

  36. [44]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM Workshop on Artificial Intelligenc...

    2023

  37. [45]

    InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents

    Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. InFindings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 2024

    2024

  38. [46]

    Formalizing and bench- marking prompt injection attacks and defenses

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and bench- marking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Secu- rity 24). USENIX Association, 2024

    2024

  39. [47]

    Neural exec: Learning (and learning from) execution triggers for prompt injection at- tacks

    Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. Neural exec: Learning (and learning from) execution triggers for prompt injection at- tacks. arXiv preprint arXiv:2403.03792, 2024

    2024 arXiv

  40. [48]

    AgentPoison: Red-teaming LLM agents via poisoning memory or knowledge bases

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. AgentPoison: Red-teaming LLM agents via poisoning memory or knowledge bases. InAdvances in Neural Information Processing Sys- tems, 2024

    2024

  41. [49]

    PoisonedRAG: Knowledge corruption attacks to retrieval-augmented generation of large language models

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG: Knowledge corruption attacks to retrieval-augmented generation of large language models. In34th USENIX Security Sympo- sium (USENIX Security 25). USENIX Association, 2025

    2025

  42. [50]

    Memory injection attacks on LLM agents via query-only interaction

    Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, and 11 Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsA PREPRINT Zhen Xiang. Memory injection attacks on LLM agents via query-only interaction. arXiv preprint arXiv:2503...

    2025

  43. [51]

    MCP- Tox: A benchmark for tool poisoning attack on real-world MCP servers

    Zhiqiang Wang, Yichao Gao, Yanting Wang, Suyuan Liu, Haifeng Sun, Haoran Cheng, Guan- quan Shi, Haohua Du, and Xiangyang Li. MCP- Tox: A benchmark for tool poisoning attack on real-world MCP servers. arXiv preprint arXiv:2508.14925, 2025

    2025 arXiv

  44. [52]

    Breaking the protocol: Security analysis of the model context protocol specification and prompt injection vul- nerabilities in tool-integrated llm agents

    Narek Maloyan and Dmitry Namiot. Breaking the protocol: Security analysis of the model context protocol specification and prompt injection vul- nerabilities in tool-integrated llm agents. arXiv preprint arXiv:2601.17549, 2026

    2026

  45. [53]

    BadSkill: Backdoor attacks on agent skills via model-in-skill poisoning

    Guiyao Tie, Jiawen Shi, Pan Zhou, and Lichao Sun. BadSkill: Backdoor attacks on agent skills via model-in-skill poisoning. arXiv preprint arXiv:2604.09378, 2026

    2026 arXiv

  46. [54]

    Supply-chain poisoning attacks against LLM coding agent skill ecosystems

    Yubin Qu, Yi Liu, Tongcheng Geng, Gelei Deng, Yuekang Li, Leo Yu Zhang, Ying Zhang, and Lei Ma. Supply-chain poisoning attacks against LLM coding agent skill ecosystems. arXiv preprint arXiv:2604.03081, 2026

    2026 arXiv

  47. [55]

    Jailbreaking leading safety- aligned LLMs with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned LLMs with simple adaptive attacks. In International Conference on Learning Representa- tions, 2025

    2025

  48. [56]

    AdvAgent: Controllable blackbox red- teaming on web agents

    Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. AdvAgent: Controllable blackbox red- teaming on web agents. InProceedings of the 42nd International Conference on Machine Learn- ing (ICML), Proceedings of Machine Learning Re- sear...

    2025

  49. [57]

    Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack

    Mark Russinovich, Ahmed Salem, and Ronen El- dan. Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack. In34th USENIX Security Symposium (USENIX Security 25). USENIX Association, 2025

    2025

  50. [58]

    Foot-in-the-door: A multi-turn jail- break for LLMs

    Zixuan Weng, Xiaolong Jin, Jinyuan Jia, and Xi- angyu Zhang. Foot-in-the-door: A multi-turn jail- break for LLMs. arXiv preprint arXiv:2502.19820, 2025

    2025 arXiv

  51. [59]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open- ended generation of diverse adversarial prompts. a...

    2024 arXiv

  52. [60]

    AutoRedTeamer: Autonomous red teaming with lifelong attack inte- gration

    Andy Zhou, Kevin Wu, Francesco Pinto, Zhaorun Chen, Yi Zeng, Yu Yang, Shuang Yang, Sanmi Koyejo, James Zou, and Bo Li. AutoRedTeamer: Autonomous red teaming with lifelong attack inte- gration. arXiv preprint arXiv:2503.15754, 2025

    2025 arXiv

  53. [61]

    Hsu, and Pin-Yu Chen

    Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I- Hsin Chung, Winston H. Hsu, and Pin-Yu Chen. Attention tracker: Detecting prompt injection at- tacks in LLMs. arXiv preprint arXiv:2411.00348, 2024

    2024 arXiv

  54. [62]

    MELON: Provable defense against indirect prompt injection attacks in ai agents

    Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, and William Yang Wang. MELON: Provable defense against indirect prompt injection attacks in ai agents. InProceedings of the 42nd International Conference on Machine Learning (ICML), Pro- ceedings of Machine Learning Research. ...

    2025

  55. [63]

    LlamaFirewall: An open source guardrail system for building secure ai agents

    Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, Alekhya Gampa, Beto de Paola, Dominik Gabi, James Crnkovich, Jean-Christophe Testud, Kat He, Rashnil Chaturvedi,...

    2025 arXiv

  56. [64]

    System-level defense against indirect prompt injec- tion attacks: An information flow control perspec- tive

    Fangzhou Wu, Ethan Cecchetti, and Chaowei Xiao. System-level defense against indirect prompt injec- tion attacks: An information flow control perspec- tive. arXiv preprint arXiv:2409.19091, 2024

    2024 arXiv

  57. [65]

    Defeating prompt injections by design

    Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramer. Defeating prompt injections by design. arXiv preprint arXiv:2503.18813, 2025

    2025 arXiv

  58. [66]

    Poskitt, and Jun Sun

    Haoyu Wang, Christopher M. Poskitt, and Jun Sun. AgentSpec: Customizable runtime enforcement for safe and reliable LLM agents. InProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026

    2026

  59. [67]

    Firewalls to secure dynamic LLM agentic net- works.Transactions on Machine Learning Re- search, 2026

    Sahar Abdelnabi, Amr Gomaa, Eugene Bag- dasarian, Per Ola Kristensson, and Reza Shokri. Firewalls to secure dynamic LLM agentic net- works.Transactions on Machine Learning Re- search, 2026

    2026

  60. [68]

    Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflec- ...

    2025 arXiv

  61. [69]

    Auto- mated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Auto- mated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024

    2024 arXiv

  62. [70]

    Darwin godel machine: Open-ended evolution of self-improving agents

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954, 2025

    2025 arXiv

  63. [71]

    Tianshi Xu, Huifeng Wen, and Meng Li. Adapting the interface, not the model: Runtime harness adap- 12 Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsA PREPRINT tation for deterministic llm agents. arXiv preprint arXiv:2605.22166, 2026

    2026 arXiv

  64. [72]

    Meta- harness: End-to-end optimization of model har- nesses

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kang- wook Lee, Omar Khattab, and Chelsea Finn. Meta- harness: End-to-end optimization of model har- nesses. arXiv preprint arXiv:2603.28052, 2026

    2026 arXiv

  65. [73]

    V oyager: An open-ended em- bodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended em- bodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  66. [74]

    Skillopt: Executive strategy for self-evolving agent skills

    Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dong- dong Chen, Xue Yang, and Chong Luo. Skillopt: Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026

    2026 arXiv

  67. [75]

    decisions

    Minseok Choi, Seungbin Yang, Dongjin Kim, Subin Kim, Jungmin Son, Yunseung Lee, Jaegul Choo, and Youngjun Kwak. Membrane: A self- evolving contrastive safety memory for LLM agent defense. arXiv preprint arXiv:2606.05743, 2026. A HARD Implementation Details A.1 Models and Confi...

    2026 arXiv

  68. [2026]

    Benchmark repository

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.