Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Timing metric exposes 2.3x gap in rogue-agent detection

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:44 UTC pith:XH2FD2HE

load-bearing objection Timeliness is a real missing axis in agent-safety benchmarks, and StepShield's dataset is a useful step, but the EIR metric as defined doesn't match the results and the headline gap isn't yet supported. the 4 major comments →

arxiv 2601.22136 v2 pith:XH2FD2HE submitted 2026-01-29 cs.LG cs.AIcs.CRcs.SE

StepShield: When, Not Whether to Intervene on Rogue Agents

Gloria Felicia (University of Virginia) , Zitha Sasindran (Indian Institute of Science Bangalore) , Jinfeng He (Cornell University) , Michael Eniolade (University of the Cumberlands) , Hemant Kumar (University of Arizona) , Milan Hussain Angati (California State University Northridge) This is my paper
classification cs.LG cs.AIcs.CRcs.SE
keywords agent safetydetection timelinessEarly Intervention Raterogue agentsruntime monitoringbenchmarkLLM-as-judgeguardrails
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that agent-safety benchmarks have been asking the wrong question: they ask whether a monitor detects harmful behavior, not when. It introduces StepShield, a benchmark of code-agent trajectories with step-level labels of the first rogue action, and a metric—Early Intervention Rate (EIR)—that measures the fraction of rogue traces flagged before or at that step. On this benchmark, an LLM-based judge catches violations 2.3 times earlier than a regex-based static analyzer, even though ordinary accuracy and F1 barely distinguish them. The paper concludes that pattern-based monitors detect syntax, not intent, and therefore cannot intervene in time, making the entire deployed class of regex guardrails unsuited for real-time oversight.

Core claim

StepShield's central discovery is that detection timeliness is a measurable property that standard binary metrics cannot see. On 9,213 incident-grounded code-agent trajectories (8.1% rogue rate in test), the Early Intervention Rate separates detectors that accuracy ties: LLMJudge reaches 0.59 EIR while StaticGuard reaches 0.26, a 2.3x gap with comparable accuracy and F1. The paper identifies a 'Forensics Trap': over three-quarters of StaticGuard's alerts fire on benign prefix code before any violation, so its detections are statistically indistinguishable from random timing. This is presented as structural: regex guardrails match surface patterns, so they cannot locate the moment intent turn

What carries the argument

The key object is the Early Intervention Rate (EIR), defined as the fraction of rogue trajectories for which the detector fires at or before the ground-truth first rogue step. It is supported by a 9,213-trajectory step-level annotated dataset built from six categories of real-world incidents, and by three additional temporal metrics (Intervention Gap, Tokens Saved). EIR does the work of separating timing quality from coverage, exposing the gap that accuracy hides.

Load-bearing premise

The entire timing comparison rests on the reliability of the hand-annotated ground-truth 'first rogue step' labels and on the assumption that synthetic trajectories built from incident write-ups resemble real production agent behavior.

What would settle it

If a pattern-based detector (pure regex/keyword rules) were shown to achieve EIR comparable to an LLM judge on a real production log of agent executions—or if re-annotation of the StepShield test set with a different 'first rogue step' definition erased the 2.3x gap—the central structural claim would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Future agent-safety benchmarks should adopt timing metrics (EIR, Intervention Gap) as primary evaluation criteria, not just binary accuracy.
  • Deployments relying on regex- or keyword-based guardrails for real-time oversight should expect late detection on semantic violations; semantic judges are needed for early intervention.
  • The economic model shows early detection can cut monitoring and execution costs substantially—75% in the paper's cascade—so timely intervention has direct ROI.
  • Step-level annotation enables training process reward models for safety and developing predictive detectors that anticipate rogue behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The structural claim about pattern-based monitors likely extends beyond code agents: any detector that matches surface syntax rather than intent should struggle on violation types defined by context, so EIR-style benchmarking could be applied to web, API, and multi-agent settings.
  • The EIR gap may be sensitive to the definition of 'first rogue step'; if annotation noise shifts t_rogue by even one step, the 2.3x ratio could shrink—a testable robustness check on the benchmark.
  • The economic savings projection under uniform token pricing is described by the authors as a lower bound; the real cost curve is superlinear, so the actual savings could be larger, but this depends on the accuracy of the token-cost model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. StepShield introduces a benchmark and three temporal metrics (EIR, Intervention Gap, Tokens Saved) for evaluating when a monitor detects rogue behavior in LLM code-agent trajectories, rather than only whether it detects it. The dataset contains 9,213 (or 9,429, per the abstract) incident-grounded trajectories with step-level labels. The paper evaluates four detectors (StaticGuard, ConstraintGuard, LLMJudge, HybridGuard) and reports that LLMJudge achieves EIR 0.59 versus StaticGuard's 0.26, a 2.3x gap claimed to be invisible to accuracy metrics. It further claims that pattern-based regex guardrails are structurally unsuited for real-time oversight and that early detection yields large cost savings, culminating in a $108M five-year projection.

Significance. The core idea—that detection timeliness is a distinct and deployment-relevant axis of evaluation—is valuable and timely. A benchmark with step-level ground-truth labels, paired rogue/clean trajectories, and released code/data would be a useful community resource if the numbers are correct. The paper also correctly identifies that trajectory-level accuracy conflates early intervention with post-mortem analysis. However, the central quantitative claims are currently undermined by a definitional flaw in EIR and by multiple internal inconsistencies in the reported results. The qualitative distinction between timing and accuracy is plausible, but the specific 2.3x gap and the stronger structural conclusion about regex guardrails are not yet supported by the manuscript as written.

major comments (4)
  1. [§4.1, Eq. (1)] EIR as defined counts any trajectory where t_detect ≤ t_rogue, regardless of whether the alert at t_detect is a true positive. The abstract states that over three-quarters of StaticGuard's alerts fire on benign prefix code before any violation occurs; under Eq. (1) those would count as early detections, yet StaticGuard's reported EIR is 0.26, below the random-timing baseline of 0.23–0.24. The only way to reconcile these statements is that the implemented EIR differs from the definition—for example, by conditioning on true-positive alerts or by using a different t_detect—but the paper never specifies this. The 2.3x gap is therefore uninterpretable, and the 'Forensics Trap' conclusion is unsupported.
  2. [Table 4 vs. Appendix F; Abstract vs. §3.4] The paper contains multiple inconsistent reported values. Table 4 gives HybridGuard EIR = 0.41, while Appendix F reports its 95% CI as [0.60, 0.66]. The abstract says 9,429 trajectories, while §3.4 and Appendix A say 9,213. The abstract says '4x EIR gap,' while §5.2 says 2.3x (0.59/0.26 ≈ 2.3). The abstract also says StaticGuard's EIR is 0.23 versus random 0.24, while Table 4 reports 0.26. These inconsistencies make the headline numbers unverifiable from the text.
  3. [Abstract vs. §4.1] The EIR definition is described differently in the abstract and in Eq. (1). The abstract defines EIR as 'the fraction of detected rogue trajectories where the alert fires within a k-step window after the divergence point,' but Eq. (1) is simply t_detect ≤ t_rogue, with no k-step window and no k defined. If the implemented metric actually uses a window after the divergence point, then all EIR values in Tables 4–6 and Appendix F need to be recomputed and reported under that definition. The current text does not specify which definition was used.
  4. [§5.2 and Abstract] The paper claims the EIR gap is 'entirely invisible' to accuracy, but Table 4 shows accuracies of 0.63 and 0.56 for LLMJudge and StaticGuard—a non-negligible difference. More importantly, the abstract asserts that no existing method achieves high recall, low false-positive rate, and timely intervention, yet Table 4 reports no recall or false-positive rate, only accuracy and F1. The structural conclusion that regex guardrails 'detect syntax, not intent' and are therefore unsuited for real-time oversight rests entirely on the EIR gap, which is not supportable without a false-positive-adjusted timing metric.
minor comments (6)
  1. [Eq. (1)] The notation D_rogue is not formally defined; state that it is the set of rogue trajectories in the evaluation set. Also, the subscript formatting of t_detect and t_rogue is inconsistent.
  2. [Table 4] The column header 'Saved' is ambiguous; it should be labeled 'Tokens Saved' to match Eq. (3), and the units (%) should be stated.
  3. [Appendix D.1, Table 9] The column 'ms' should specify whether it is latency per step or per trajectory, since Table 4 reports per-step latency.
  4. [Abstract vs. Appendix A] The abstract and Section 8 state that code and data are released under Apache 2.0, while Appendix A states data is released under CC BY 4.0. Clarify the license for each component.
  5. [Figure 1] Figure 1 is referenced in the text but not included in the provided manuscript. Ensure the figure is present in the camera-ready version.
  6. [§4.4, Theorem 1] The proof is a construction and is correct, but it relies on the implicit assumption that both monitors have the same trajectory-level labeling behavior on clean trajectories. State this assumption explicitly.

Circularity Check

0 steps flagged

No circular derivation: EIR is computed from externally labeled first-rogue-step timestamps and detector alert times; no self-citations or fitted parameters manufacture the gap.

full rationale

The paper's central claim is an empirical benchmark comparison, not a derivation from its own definitions. EIR (Eq. 1) is defined as the fraction of rogue trajectories with t_detect <= t_rogue, where t_rogue comes from external step-level annotations (§3.4, Appendix G) and t_detect from independent detectors; the metric is not fitted to detector outputs and does not encode the measured 2.3x gap. Theorem 1 is an illustrative construction, not an assumed conclusion. There are no self-citations, and no 'uniqueness theorem' or imported ansatz is load-bearing. The statement that regex guardrails 'detect syntax, not intent' is a characterization of StaticGuard's implementation, and the empirical EIR gap is the evidence cited for the unsuitability conclusion; even if that conclusion overgeneralizes, it is not a circular reduction. The reviewing rule also asks me to flag internal inconsistencies and limitations: the HybridGuard EIR is 0.41 in Table 4 but its Appendix F CI is [0.60,0.66], and Eq. 1 does not condition on alert correctness, so an always-fire detector would trivially get EIR=1. These are correctness/reproducibility concerns about the benchmark's validity, not cases where a prediction is identical to its input by construction. Therefore no circular step can be exhibited under the required standard.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 0 invented entities

The central measurement rests on ground-truth step labels on synthetic trajectories. Economic projections rest on assumed scale, growth, and cost inputs. Cascade threshold is tuned, not predicted.

free parameters (6)
  • EIR k-step window = undefined in body
    Abstract defines EIR with a k-step window; Eq. 1 defines t_detect <= t_rogue. The parameter k is never specified, making the metric ambiguous.
  • cascade_threshold = 0.7
    Tuned by ablation on the training set (Appendix D.4, Table 12); changes HybridGuard EIR/cost but not the core timing-vs-accuracy point.
  • token backfill model = not specified
    Section 6.4 says token counts were estimated by a statistical model calibrated against real API calls; coefficients are not reported, so Tokens Saved and cost projections cannot be independently recomputed.
  • enterprise starting scale = 100M trajectories/year
    Assumption in Appendix B for the $108M savings projection; sensitivity ranges 50M–200M.
  • growth rate = 35%/yr
    Midpoint of 25–50% market estimates, Appendix B.
  • baseline cost per trajectory = $0.10
    Assumes 8k tokens at GPT-4.1-mini pricing, Appendix B.
axioms (3)
  • domain assumption First-rogue-step labels are consistent and correct.
    EIR (Eq. 1) is defined relative to ground-truth t_rogue; κ=0.82 is reported only on 500 of 9k+ trajectories, and ambiguous steps are systematically labeled benign (Appendix G), which can shift t_rogue later.
  • domain assumption Synthetic LLM-generated trajectories represent real production code-agent behavior.
    Section 3.4/Appendix A describe GPT-4/Claude-3 generation from incident reports, not collection from real deployments; external validity of the benchmark depends on this transfer.
  • standard math Self-attention O(n^2) cost is a valid production token-cost proxy.
    Appendix H uses asymptotic complexity to claim later tokens cost up to 64x more; this is a modeling assumption, not a measured price schedule.

pith-pipeline@v1.3.0-alltime-deepseek · 122 in / 16188 out tokens · 225818 ms · 2026-08-03T06:44:04.467415+00:00 · methodology

0 comments
read the original abstract

Agent safety benchmarks measure whether a monitor detects harm, not when. Yet timing is the difference between intervention and autopsy. We introduce StepShield, the first benchmark that treats detection timeliness as a first-class metric. On 9,429 incident-grounded code-agent trajectories, we define the Early Intervention Rate (EIR): the fraction of detected rogue trajectories where the alert fires within a k-step window after the divergence point, isolating timing quality from coverage. This metric exposes what we call the Forensics Trap: a pattern-based guardrail with 847 rules achieves 86% recall yet is statistically indistinguishable from random timing on EIR (0.23 vs. 0.24; p = 0.66, one-sided binomial; difference within CI), because over three-quarters of its alerts trigger on benign prefix code before any violation occurs. The 4x EIR gap between rule-based and semantic detectors is completely invisible to accuracy, recall, or F1. Our finding is structural: regex guardrails detect syntax, not intent, and therefore cannot distinguish the moment an agent turns rogue, rendering the entire deployed class of pattern-based monitors unsuited for real-time oversight. No existing method simultaneously achieves high recall, low false-positive rate, and timely intervention, establishing step-level rogue detection as genuinely unsolved.

Figures

Figures reproduced from arXiv: 2601.22136 by Gloria Felicia (University of Virginia), Hemant Kumar (University of Arizona), Jinfeng He (Cornell University), Michael Eniolade (University of the Cumberlands), Milan Hussain Angati (California State University Northridge), Zitha Sasindran (Indian Institute of Science Bangalore).

Figure 1
Figure 1. Figure 1: The core insight of StepShield: detection timing determines intervention capability. Early [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Economic impact of early detection. (a) Annual cost by detection approach. (b) Cumulative [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ProjGuard: Safety Monitoring for Computer-Use Agents via Low-Dimensional Projections

    stat.CO 2026-05 unverdicted novelty 5.0

    ProjGuard monitors agent trajectories with low-dimensional projections to cut unsafe actions from 16% to 3% and raise task completion from 59% to 65% on OS-Harm.

Reference graph

Works this paper leans on

24 extracted references · 8 linked inside Pith · cited by 1 Pith paper

  1. [1]

    AgentHarm : A benchmark for measuring harmfulness of LLM agents

    Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. AgentHarm : A benchmark for measuring harmfulness of LLM agents. In International Conference on Learning Representations (ICLR), 2025

    2025

  2. [2]

    ShieldAgent : Shielding agents via verifiable safety policy reasoning

    Zhaorun Chen, Mintong Kang, and Bo Li. ShieldAgent : Shielding agents via verifiable safety policy reasoning. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  3. [3]

    AI coding tool wiped our database, says startup in catastrophic failure

    Fortune Staff . AI coding tool wiped our database, says startup in catastrophic failure. Fortune, July 2025

    2025

  4. [4]

    Sleeper agents: Training deceptive LLMs that persist through safety training

    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    On the computational complexity of self-attention

    Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention. In Proceedings of the 34th International Conference on Algorithmic Learning Theory (ALT), pages 597--619, 2023

    2023

  6. [6]

    Specification gaming: the flip side of AI ingenuity

    Victoria Krakovna et al. Specification gaming: the flip side of AI ingenuity. DeepMind Blog, 2020

    2020

  7. [7]

    SHADE-Arena : Evaluating sabotage and monitoring in LLM agents

    Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shlegeris, and Joe Benton. SHADE-Arena : Evaluating sabotage and monitoring in LLM agents. arXiv preprint arXiv:2506.15740, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    AgentBench : Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench : Evaluating LLMs as agents. In International Conference on Learning Representations (ICLR), 2024

    2024

  9. [9]

    GAIA : A benchmark for general AI assistants

    Gr \'e goire Mialon, Roberto Dess \`i , Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozi \`e re, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. GAIA : A benchmark for general AI assistants. In International Conference on Learning Representations (ICLR), 2024

    2024

  10. [10]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Identifying risks of LM agents with an LM -emulated sandbox

    Yangjun Ruan et al. Identifying risks of LM agents with an LM -emulated sandbox. In NeurIPS, 2024

    2024

  12. [12]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \`i , Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  13. [13]

    SafeArena : Evaluating the safety of autonomous web agents

    Alexander Daniel Tur, Nicholas Meade, Xing Han L \`u , Alejandra Zambrano, Arkil Patel, et al. SafeArena : Evaluating the safety of autonomous web agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  14. [14]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    2017

  15. [15]

    GuardAgent : Safeguard LLM agents via knowledge-enabled reasoning

    Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. GuardAgent : Safeguard LLM agents via knowledge-enabled reasoning. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025

  16. [16]

    OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Xu, Yufan Song, Boxuan Li, Yuxuan Ding, Jiayi Zou, Wangchunshu Zhao, Hao Peng, Daniel Fried, and Graham Neubig

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Ding, Jiayi Zou, Wangchunshu Zhao, Hao Peng, Daniel Fried, and Graham Neubig. TheAgentCompany : Benchmarking LLM agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    ReAct : Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  19. [19]

    SafeAgentBench : A benchmark for safe task planning of embodied LLM agents

    Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, et al. SafeAgentBench : A benchmark for safe task planning of embodied LLM agents. arXiv preprint arXiv:2412.13178, 2024

    arXiv 2024

  20. [20]

    R-Judge : Benchmarking safety risk awareness

    Tongxin Yuan et al. R-Judge : Benchmarking safety risk awareness. arXiv preprint arXiv:2401.10019, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Agent-SafetyBench : Evaluating the safety of LLM agents

    Zhexin Zhang et al. Agent-SafetyBench : Evaluating the safety of LLM agents. arXiv preprint arXiv:2412.14470, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Judging LLM -as-a-judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM -as-a-judge with MT-Bench and Chatbot Arena . arXiv preprint arXiv:2306.05685, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    WebArena : A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. WebArena : A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), 2024

    2024

  24. [24]

    Agent-as-a-judge: Evaluate agents with agents

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and J \"u rgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    2025