REVIEW 3 major objections 5 minor 46 references
Unsafe agent behavior can be stopped as natural-language token flows before they hit memory, tools, or external sinks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 08:07 UTC pith:FMJJIWER
load-bearing objection Useful systems framing for pre-sink agent mediation; the real win is latency/HR trade-off, not the 1.1-point ASR edge over ClawKeeper. the 3 major comments →
Token-Flow Firewall: Semantic Runtime Auditing for Persistent AI Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors claim that security for persistent agents should be formulated as containment of semantic token flows—natural-language payloads about to cross context, authority, or capability boundaries—and that a local hierarchical auditor over those flows can cut attack success on CIK-Bench to 12.5% while preserving a 97.4% benign pass rate without human confirmation and with only 0.69 s extra latency on benign cases.
What carries the argument
Token-flow abstraction: each security-relevant transfer is written as f = (payload spans, source, sink, runtime metadata, boundary), then mediated pre-effect by deterministic precheck, a small local auditor that can rewrite separable risk, and selective large-model fallback.
Load-bearing premise
The defense only works if every security-critical change appears as a natural-language transfer that the instrumented runtime can see before it is committed, and the attacker cannot bypass or disable that mediation layer.
What would settle it
An attack case on the same OpenClaw-style runtime that reaches a protected sink with residual actionable harm after TokenWall mediation, or a matched benign owner-authorized transfer that is systematically blocked, deferred, or delayed beyond the reported rates under the same fixed thresholds and judge protocol.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TokenWall, a runtime semantic firewall for OpenClaw-style persistent AI agents. It models security-relevant state transitions as token flows f=(x,s,t,c,b) that cross context, authority, and capability boundaries, and mediates them before sinks via deterministic precheck, a local small-model auditor with structured scores and optional rewrite, and selective large-model arbitration. On CIK-Bench (88 attack / 38 benign cases), TokenWall reports 12.5% overall ASR (vs 14.7% for ClawKeeper), 97.4% benign pass rate with 0% human rate and 0.69 s extra benign latency, with architecture, report-schema, auditor-size, rewrite, and uncertainty ablations supporting the hierarchical design.
Significance. If the result holds, the paper offers a useful systems framing for persistent-agent security: pre-transfer mediation of natural-language token flows rather than post-hoc action or remote full-trajectory review. Strengths include a clear threat model, an implementable hierarchical pipeline with fail-closed rewrite validation, head-to-head comparison against multiple OpenClaw ecosystem defenses, and multi-axis evaluation (ASR by surface, RR/HR, latency, benign PR) plus ablations that show components are load-bearing. The practical claim—stronger containment than the best full-runtime baseline at lower latency and less human escalation—would matter for deployable agent runtimes even if absolute gains remain modest.
major comments (3)
- Abstract and Table 1: the headline security claim is a 12.5% vs 14.7% overall ASR gap over ClawKeeper on only 88 attack cases (~1 case absolute). No confidence intervals, multi-seed runs, or judge-sensitivity analysis are reported, despite GPT-5.5 LLM-as-judge residual-actionability labels (Section 4; Appendix C.1). Given that Figures 3–4 and Tables 8–10 show ASR is sensitive to fallback, report fields, and rewrite completeness, the ranking over the strongest baseline is not yet shown to be stable; at minimum report uncertainty (e.g., bootstrap/case-level CI) and a small judge- or threshold-perturbation check before claiming superiority.
- Section 2 and Section 3.1: the evaluation and threat model assume all security-relevant transitions are interceptable as instrumented natural-language token flows at pre-effect OpenClaw boundaries, with an adversary unable to bypass the firewall or act outside the normal path. This is a reasonable systems assumption but is load-bearing for the claim of full-coverage pre-execution mediation. The manuscript should state more explicitly which attack classes fall outside instrumented surfaces (non-text tool channels, host-level side channels, uninstrumented plugins) and whether any CIK-Bench residual failures are boundary-misses rather than auditor errors.
- Table 5 and Appendix B.3: escalation uses many fixed scalar thresholds and surface-specific Φtype(f) conditions chosen a priori. Ablations confirm that removing uncertainty, surface predicates, or fallback raises ASR substantially, so the operating point is policy-dependent. A short sensitivity sweep (or leave-one-threshold-out) on the default configuration would strengthen the claim that the hierarchical design—not a particular threshold set—drives the reported trade-off.
minor comments (5)
- Figure 1 caption and body: clarify that TokenWall’s allow/rewrite/defer/block decisions are enforced at transfer time, not only illustrated conceptually, so readers can map the figure to Algorithm 1.
- Table 1 latency column mixes end-to-end case times that include task-model work; Appendix C.1 defines Small Lat. vs Overall Lat., but the main table would benefit from a one-line note distinguishing defense overhead from full case runtime.
- Equation (1)–(2) and Appendix B.2: residual_risk_types and rewrite_completeness are central to escalation; a short inline example of an accepted vs rejected rewrite would make the contracts easier to audit.
- Related Work: several 2026 OpenClaw defenses are contemporaneous; ensure citation dates and public availability are accurate and that CIK-Bench authorship is clearly distinguished from TokenWall.
- Limitations correctly notes host compromise and explicit user authorization of harmful acts are out of scope; consider also noting dependence on LLM-judge agreement for residual actionability.
Circularity Check
Empirical systems paper: TokenWall’s ASR/PR/latency claims are measured outcomes on an external benchmark, not results forced by definition or self-citation.
full rationale
TokenWall is a runtime defense design evaluated on CIK-Bench (Zijun Wang et al., 2026—different author set) and matched benign cases, with ASR, PR, HR, LCR, and latency reported against independent baselines (OpenGuardrails, ClawKeeper, etc.). The core formulation—intercept natural-language token flows at pre-effect source–sink boundaries—is a systems abstraction and instrumentation choice, not a mathematical derivation that defines the target metric in terms of fitted constants. Escalation thresholds (Table 5) and Φtype(f) are free policy parameters fixed a priori; they are not re-labeled as predictions of a theory. Ablations (Figs. 3–4, Tables 8–10) test component necessity rather than closing a definitional loop. There is no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no fitted parameter renamed as a first-principles forecast. Ordinary same-ecosystem tooling and LLM-as-judge evaluation raise correctness/stability questions, not circularity. Score 0 is appropriate.
Axiom & Free-Parameter Ledger
free parameters (3)
- Escalation scalar thresholds (Table 5)
- Local auditor model size/choice (default Qwen3-4B)
- Fallback arbiter and task/judge models
axioms (4)
- domain assumption Most security-critical agent interactions are transmitted as natural-language token sequences that can be intercepted before sink commit.
- domain assumption Adversary cannot compromise the host, bypass/tamper with the firewall, modify audit logs, or execute outside the agent’s normal path.
- ad hoc to paper Structured small-auditor scores (risk, uncertainty, exploitability, impact, rewrite completeness) plus surface-specific Φtype(f) suffice to decide local vs escalated enforcement for most flows.
- domain assumption LLM-as-judge labels of whether a mediated transfer remains actionable at the sink are adequate for reporting ASR.
invented entities (2)
-
Token-flow f = (x, s, t, c, b) with audit-relevant spans
independent evidence
-
TokenWall hierarchical firewall (precheck + local small auditor + large arbiter)
independent evidence
read the original abstract
Persistent AI agents extend large language models (LLMs) beyond single-turn interaction into long-lived software systems. Unlike traditional chat assistants, unsafe content in these agents can propagate through persistent state, reusable skills, and tool-mediated interactions, creating a substantially larger semantic attack surface. We observe that most security-critical interactions in such agents are transmitted through natural-language token flows, including memory updates, tool arguments, retrieved files, and inter-component communications. This observation enables a new security formulation: unsafe behavior can be intercepted as risky semantic flows before reaching privileged runtime sinks. Based on this insight, we propose TokenWall, a runtime defense framework that acts as a semantic firewall over agent token flows. TokenWall performs boundary-aware semantic auditing over these flows, constructing structured source-sink audit records, applying lightweight local inspection before execution, and selectively escalating ambiguous high-risk cases to stronger arbitration modules. Unlike prior approaches that rely on sparse auditing or remote large-model oversight, TokenWall enables full-coverage pre-execution mediation while reducing remote arbitration and latency. Experiments on CIK-Bench show that TokenWall reduces attack success rate to 12.5% while maintaining a 97.4% benign executable pass rate without human confirmation. TokenWall further introduces only 0.69 seconds of additional latency on benign cases, demonstrating that semantic runtime containment can achieve a practical security-utility trade-off for persistent AI agents.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
Publications Manual , year = "1983", publisher =
work page 1983
-
[3]
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
- [4]
-
[5]
Dan Gusfield , title =. 1997
work page 1997
-
[6]
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
work page 2015
-
[7]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[8]
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Ruan, Yangjun and Dong, Haoran and Wang, Andrew and Pitis, Silviu and Zhou, Yongchao and Ba, Jimmy and Dubois, Yann and Maddison, Chris J. and Hashimoto, Tatsunori , year =. Identifying the Risks of. 2309.15817 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
- [9]
-
[10]
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Andriushchenko, Maksym and Souly, Alexandra and Dziemian, Mateusz and Duenas, Derek and Lin, Maxwell and Wang, Justin and Hendrycks, Dan and Zou, Andy and Kolter, Zico and Fredrikson, Matt and Winsor, Eric and Wynne, Jerome and Gal, Yarin and Davies, Xander , year =. 2410.09024 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
Agent-SafetyBench: Evaluating the Safety of LLM Agents
Zhang, Zhexin and Cui, Shiyao and Lu, Yida and Zhou, Jingzhuo and Yang, Junxiao and Wang, Hongning and Huang, Minlie , year =. 2412.14470 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
Greshake, Kai and Abdelnabi, Sahar and Mishra, Shailesh and Endres, Christoph and Holz, Thorsten and Fritz, Mario , year =. Not What You've Signed Up For: Compromising Real-World. 2302.12173 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[13]
and Pinto, Jeremy and Khan, Anaum and Bouchard, Louis-Fran
Schulhoff, Sander V. and Pinto, Jeremy and Khan, Anaum and Bouchard, Louis-Fran. Ignore This Title and. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =
work page 2023
-
[14]
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models , author =. 2023 , eprint =
work page 2023
-
[15]
Defending Against Indirect Prompt Injection Attacks with Spotlighting , author =. 2024 , archivePrefix =
work page 2024
-
[16]
StruQ: Defending Against Prompt Injection with Structured Queries
Chen, Sizhe and Piet, Julien and Sitawarin, Chawin and Wagner, David , year =. 2402.06363 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[17]
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, Eric and Xiao, Kai and Leike, Reimar and Weng, Lilian and Heidecke, Johannes and Beutel, Alex , year =. The Instruction Hierarchy: Training. 2404.13208 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[18]
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and Tontchev, Martin and Hu, Qing and Fuller, Brian and Testuggine, Davide and Khabsa, Madian , year =. 2312.06674 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[19]
NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails
Rebedea, Traian and Dinu, Razvan and Sreedhar, Makesh Narsimhan and Parisien, Christopher and Cohen, Jonathan , editor =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , month = dec, year =. doi:10.18653/v1/2023.emnlp-demo.40 , pages =
-
[20]
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , year =. Judging. 2306.05685 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[21]
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , year =. 2303.16634 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[22]
Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
Wang, Zijun and Tu, Haoqin and Zhang, Letian and Chen, Hardy and Wu, Juncheng and Liu, Xiangyan and Yuan, Zhenlong and Pang, Tianyu and Shieh, Michael Qizhe and Liu, Fengze and Zheng, Zeyu and Yao, Huaxiu and Zhou, Yuyin and Xie, Cihang , year =. Your Agent, Their Asset: A Real-World Safety Analysis of. 2604.04759 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[23]
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url =
work page 2023
-
[24]
Schick, Timo and Dwivedi-Yu, Jane and Dessi, Roberto and Raileanu, Roberta and Lomeli, Maria and Hambro, Eric and Zettlemoyer, Luke and Cancedda, Nicola and Scialom, Thomas , booktitle =. 2023 , url =
work page 2023
-
[25]
Generative Agents: Interactive Simulacra of Human Behavior , author =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , year =. doi:10.1145/3586183.3606763 , url =
-
[26]
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, Guanzhi and Xie, Yuqi and Jiang, Yunfan and Mandlekar, Ajay and Xiao, Chaowei and Zhu, Yuke and Fan, Linxi and Anandkumar, Anima , year =. 2305.16291 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[27]
LlamaFirewall: An open source guardrail system for building secure AI agents
Chennabasappa, Sahana and Nikolaidis, Cyrus and Song, Daniel and Molnar, David and Ding, Stephanie and Wan, Shengye and Whitman, Spencer and Deason, Lauren and Doucette, Nicholas and Montilla, Abraham and Gampa, Alekhya and de Paola, Beto and Gabi, Dominik and Crnkovich, James and Testud, Jean-Christophe and He, Kat and Chaturvedi, Rashnil and Zhou, Wu an...
work page internal anchor Pith review Pith/arXiv arXiv
-
[28]
Prompt Injection attack against LLM-integrated Applications
Liu, Yi and Deng, Gelei and Li, Yuekang and Wang, Kailong and Wang, Zihao and Wang, Xiaofeng and Zhang, Tianwei and Liu, Yepang and Wang, Haoyu and Zheng, Yan and Zhang, Leo Yu and Liu, Yang , year =. Prompt Injection Attack Against. 2306.05499 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[29]
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Qiusi and Liang, Zhixiang and Ying, Zifan and Kang, Daniel , year =. 2403.02691 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[30]
Formalizing and Benchmarking Prompt Injection Attacks and Defenses , author =. 2023 , eprint =
work page 2023
-
[31]
Chen, Sizhe and Zharmagambetov, Arman and Wagner, David and Guo, Chuan , year =. 2507.02735 , archivePrefix =
-
[32]
Liu, Songyang and Li, Chaozhuo and Wang, Chenxu and Hou, Jinyu and Chen, Zejian and Zhang, Litian and Liu, Zheng and Ye, Qiwei and Hei, Yiming and Zhang, Xi and Wang, Zhongyuan , year =. 2603.24414 , archivePrefix =
-
[33]
Wang, Thomas and Li, Haowen , year =. 2510.19169 , archivePrefix =
-
[34]
Pirch, Lukas and Horlboge, Micha and Gro. Toward Securing. 2026 , eprint =
work page 2026
-
[35]
Kasselman, P. and Lombardo, J. and Rosomakho, Y. and Campbell, B. and Steele, N. , year =
-
[36]
Authenticated Delegation and Authorized AI Agents
South, Tobin and Marro, Samuele and Hardjono, Thomas and Mahari, Robert and Whitney, Cedric Deslandes and Greenwood, Dazza and Chan, Alan and Pentland, Alex , year =. Authenticated Delegation and Authorized. 2501.09674 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[37]
Owner-Harm: A Missing Threat Model for AI Agent Safety
Zhang, Dongcheng and Jiang, Yiqing , year =. Owner-Harm: A Missing Threat Model for. 2604.18658 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[38]
Agent-Sentry: Bounding LLM Agents via Execution Provenance
Sequeira, Rohan and Damianakis, Stavros and Iqbal, Umar and Psounis, Konstantinos , year =. Agent-Sentry: Bounding. 2603.22868 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
- [39]
- [40]
- [41]
- [42]
-
[43]
AgentWard: A Lifecycle Security Architecture for Autonomous AI Agents
Zhang, Yixiang and Deng, Xinhao and Wu, Jiaqing and Xiao, Yue and Xu, Ke and Li, Qi , year =. 2604.24657 , archivePrefix =
work page internal anchor Pith review Pith/arXiv arXiv
-
[44]
2026 , howpublished =
work page 2026
-
[45]
Munda, Sandro , year =
- [46]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.