Pith. sign in

REVIEW 3 major objections 5 minor 38 references

The paper reframes agent action review as a per-instance EXECUTE/ASK/REFUSE routing decision and claims that a single lightweight guard, tuned by one decoding threshold, controls both over-ask and under-ask error rates simultaneously and ou

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A 4B model fine-tuned on a 9,203-step LLM-annotated corpus with a single tunable threshold yields a three-way EXECUTE/ASK/REFUSE guard that beats zero-shot baselines on in-distribution and held-out agent actions.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A genuinely useful three-way routing guard with an honest error analysis, held back by a self-generated gold standard and an unmeasured REFUSE-to-ASK confusion that makes the 'simultaneous directional error control' claim stronger than the current evidence. the 3 major comments →

arxiv 2607.13594 v1 pith:JEYGNJ3C submitted 2026-07-15 cs.AI

SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing

Tianyu Chen , Chujia Hu , Wenjie Wang This is my paper
classification cs.AI
keywords agent safetyguard modelsEXECUTE-ASK-REFUSE routinghuman-in-the-loopdirectional error ratesLLM agentspersona-conditioned annotationthreshold-based autonomy control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conventional guard models for LLM agents judge each proposed action safe or unsafe, which conflates 'harmful in itself' with 'inappropriate in this context' and forces every uncertain case onto the user. This paper argues that the right unit of decision is per-instance three-way routing — execute, ask the human, or refuse — and that a guard trained on step-level labels can learn a boundary between those three. The empirical claim is that Safety Sentry, a 4B-parameter model fine-tuned on 9,203 annotated steps across nine self-hosted enterprise services, reaches 91.02% accuracy with refuse-recall of 92.68%, over-ask 5.05%, and under-ask 4.96% on its in-distribution test set, while all eighteen baselines trade one error direction against the other. A single decoding threshold then slides the autonomy-oversight boundary without retraining. Every reported number is measured against the authors' annotation protocol, which uses two LLM annotators, a fourteen-pattern trigger taxonomy, and a 92% human-audit agreement.

Core claim

Safety Sentry's central claim is that agent action review is a three-way decision — EXECUTE, ASK, or REFUSE — and that a guard trained on step-level supervision can learn a boundary at which both directional errors are small at the same time. On its in-distribution test set of 1,436 real tool-call snapshots from nine self-hosted enterprise services, the model reports 91.02% accuracy, 92.68% refuse-recall, 5.05% over-ask, and 4.96% under-ask. None of the eighteen baselines, including much larger open-weight and frontier closed-source models, reaches the same (over-ask, under-ask) frontier; each reduces one error direction only by inflating the other. The same boundary transfers to a held-out

What carries the argument

The mechanism that carries the argument is the decision-token distribution over three newly added tokens — <|direct_execute|>, <|ask_human|>, and <|refuse|> — produced by a single decoding call on each step snapshot (user memory, task, service, prior trace, proposed action). The EXECUTE/ASK trade-off is isolated as a scalar knob by renormalizing q_EXECUTE = p_EXECUTE / (p_EXECUTE + p_ASK); if q_EXECUTE >= τ, the guard executes, otherwise it asks, while REFUSE is a fixed floor invariant to τ. This makes the autonomy-oversight balance a post-hoc choice rather than a training-time property. The training signal is a 9,203-record step-level corpus, labeled by two LLM annotators under a fourteen-p

Load-bearing premise

The load-bearing premise is that the gold labels used for training and evaluation — produced by two LLM annotators applying the authors' fourteen-pattern trigger taxonomy, with author arbitration and a 92% human-audit agreement — correctly capture the real routing decisions an operator would want; if this rubric misclassifies what should be ASK versus REFUSE, the claimed error rates measure agreement with the annotation protocol rather than actual routing quality.

What would settle it

Have a panel of operators who have never seen the trigger taxonomy label the same 1,436 in-distribution test snapshots (and the 198 held-out-service snapshots) as EXECUTE/ASK/REFUSE from the raw snapshot alone, then recompute accuracy, over-ask, under-ask, and refuse-recall against those independently produced labels. If agreement with the author-rubric labels falls far below the reported 92% human agreement, or if the directional error rates under operator labels are materially higher, the simultaneous control of both error directions is an artifact of the annotation protocol rather than a pr

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • One fixed checkpoint can serve deployments with different risk tolerances: operators move along the over-ask/under-ask curve by tuning τ, with no retraining, and refuse-recall stays constant.
  • Separating ASK from REFUSE means the guard can defer without rejecting, so users are not interrupted for every ambiguous action while hard refusals remain non-negotiable.
  • The learned three-way boundary transfers to an unseen service and across upstream agent frameworks and backbones, suggesting it is a transferable decision structure rather than a memorized mapping.
  • Personalized routing is trainable: the same tool call can be routed to EXECUTE for a permissive user and to ASK for a cautious user, and Safety Sentry keeps roughly 87–88% accuracy on both sides of memory flip-pairs while large baselines drop toward chance.
  • Parameter scaling alone is a bounded route to reliable routing; frontier models plateau around 70–73% accuracy and remain unable to control both directional error rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • (Editorial) If the taxonomy is applied beyond enterprise computer-using agents, the fourteen trigger patterns will likely need domain-specific extension; the paper does not evaluate creative or open-ended writing tools, so a fair test would be a comparable corpus there.
  • (Editorial) A practical next step is to learn τ from operator feedback or bandit-style calibration instead of setting it manually; the paper's own limitation notes that manual τ requires deployment experience and trial-and-error.
  • (Editorial) The reported autonomy gain assumes that ASK interruptions are actually scarce enough to avoid alarm fatigue; measuring downstream operator throughput and override behavior would test whether three-way routing produces the promised balance in practice.
  • (Editorial) An independent, taxonomy-blind human panel could re-label the same test snapshots; because the current gold labels are LLM-produced with author audit, such a panel would establish whether the ASK boundary matches what operators truly want confirmed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reformulates agent action guardrails as a per-instance three-way routing problem over {EXECUTE, ASK, REFUSE}, arguing that binary safe/unsafe guards conflate harmful actions with contextually inappropriate ones and over-trigger human escalation. It introduces SAFETYSENTRY, a 4B LoRA fine-tuned guard trained on a newly constructed step-level corpus of 9,203 records from nine self-hosted enterprise services, with labels produced by two LLM annotators under a fourteen-trigger taxonomy and author arbitration. The guard emits a decision token plus a rationale/question payload; a decoding-time threshold τ on the EXECUTE/ASK boundary allows re-positioning the autonomy/oversight trade-off without retraining. Experiments report large gains over 18 baselines on in-distribution accuracy and refuse-recall, a Pareto frontier dominating baseline operating points on (OAR, UAR), graceful degradation under persona memory, transfer to a held-out service (Mailu), robustness across upstream frameworks/backbones, and modest latency. The paper also includes a limitations section and detailed appendix with the taxonomy, prompts, and case studies.

Significance. If the empirical claims hold, the paper makes a useful conceptual and practical contribution: it identifies a real gap in binary guard models, provides a concrete three-way routing formulation, and demonstrates that a lightweight open-weight model can outperform frontier APIs on this task. The threshold-based deployment mechanism is simple and plausible, and the corpus, despite its annotation caveats, is a potentially reusable resource for the community. The main empirical strengths are the real self-hosted service environments, the systematic baseline comparison, the OOD and robustness evaluations, and the candid case studies that include failures. However, the significance is contingent on two issues: the gold labels are generated by the same protocol the model is trained to reproduce, and the safety-floor claim is not fully supported because REFUSE misses are not decomposed by whether they go to ASK or EXECUTE. These are addressable with additional analysis and reporting.

major comments (3)
  1. [§4.1, metrics; Appendix G Case 9] The two directional error rates OAR and UAR are defined only over the EXECUTE/ASK boundary. The REFUSE class is summarized solely by RR = P(ŷ=REFUSE | y=REFUSE), which conflates two very different failure modes: a gold REFUSE predicted as ASK and one predicted as EXECUTE. Since the taxonomy (§A, triggers j–n) defines REFUSE as 'structurally unacceptable regardless of user intent,' routing a refuse-worthy action to ASK delegates a non-delegable decision to the user, who may approve—especially under the adversarial-memory conditions described in §3.2. Appendix G, Case 9 (OOD, Mailu) is a concrete instance: gold REFUSE, predicted ask_human with p_AH=0.977. The paper does not report P(ASK|REFUSE) or P(EXECUTE|REFUSE), so the 7.32% missed refusals on the ID test set and 6.45% on Mailu cannot be assessed: they could be mostly safety-critical ASK mis-routings. The abstract's claim of 'controlli
  2. [§3.2; Tables 1–3] Training, validation, and test labels are all produced by the same two-LLM-annotator + author-arbitration protocol, and external benchmarks are re-annotated under the same taxonomy. The evaluation therefore primarily measures agreement with the annotation pipeline, not with an independent ground truth. The reported '92% human agreement' is an audit by the authors themselves, not an independent inter-annotator reliability measure, and no kappa or per-class disagreement statistics are given. This makes the quantitative comparison self-referential: a model fine-tuned to mimic the full rubric will naturally outperform zero-shot baselines that receive a shorter prompt. Please provide (a) inter-annotator agreement between the two LLM annotators, (b) author-audit disagreement rates per class, and (c) at least one external validation of the taxonomy—for example, a sample labeled by independent h
  3. [§4.6, Table 3] The reference configuration (custom × GPT-5.5) in Table 3 is reported on 'the same test set' as Table 1, but its balanced-operating-point numbers differ substantially: Acc 88.4% vs 91.02%, UAR 14.7% vs 4.96%. The paper does not state the threshold τ used in Table 3 or whether the trajectories were regenerated and re-labeled for each upstream configuration. If the same fixed checkpoint and the same τ=0.68 were used, the reference row should match Table 1; if τ differed, or the test snapshots were regenerated per configuration, the comparison is confounded and the robustness claim is not interpretable. Please clarify the experimental protocol and, if needed, re-run the grid with matched τ and identical test snapshots, or explicitly analyze the effect of threshold selection on the grid.
minor comments (5)
  1. [Appendix A] The text says 'the taxonomy is used directly as the labeling rubric (Figure 4),' but Figure 4 is the Pareto-frontier plot. The rubric is in Figures 10–11 (Appendix F). Please fix the cross-reference.
  2. [References] References Wang et al. 2025b and 2025c are the same paper ('Learning to ask: When LLM agents meet unclear instruction'). Please merge or differentiate.
  3. [§4.2] The sentence 'Llama-3.2-3B ... UAR exceeds 56%' is technically true but understates the value in Table 1 (UAR=80.12%). Please use the actual number or a more representative summary.
  4. [§4.4] The '33.3% chance floor' mentioned for the memory-augmented paired evaluation is not the natural chance baseline. Since gold labels flip between the paired copies and the model may predict any of three classes, a random predictor would not necessarily achieve 33.3% on the memory-augmented side; please clarify the chance model or compute a proper baseline.
  5. [§3.3] The threshold τ is applied to q_EXECUTE restricted to the non-REFUSE subspace, so REFUSE predictions are unaffected by τ. This is stated, but it might be worth emphasizing in the main text that the 'risk tolerance' knob does not adjust the REFUSE strictness—only the EXECUTE/ASK boundary.

Circularity Check

0 steps flagged

No significant circularity: the central claims rest on a disjoint test split and explicit threshold sweeps; the annotation-provenance issue is a validity caveat, not a circular reduction.

full rationale

The paper's derivation chain is a standard supervised pipeline: define a three-way routing problem (Section 3.1), construct a labeled corpus under a stated taxonomy (Section 3.2), fine-tune a guard with LoRA on a training split (Section 3.3), and evaluate on a disjoint in-distribution test split plus a held-out service (Sections 4.1–4.5). The prediction target is not defined as the model's own output; SAFETYSENTRY emits decision tokens from a separate decoding pass, and the reported 91.02% Acc / 92.68% RR / 5.05% OAR / 4.96% UAR are computed against gold labels that were never used to fit the model on that split. The claim that baselines 'reduce one error direction only by collapsing toward the other' is empirical: 18 baselines receive the same snapshots and the same labels yet do not reach the same operating point, so the result is not forced by construction. The threshold τ is explicitly selected on a validation set for the balanced point, and the Pareto-frontier sweeps in Section 4.3 are presented as operating-point analyses, not as hidden fits; the Limitations section candidly states that τ is set manually. The reader's concern that training and test labels both come from the same two-LLM annotator plus author-arbitration protocol is a legitimate external-validity caveat about the gold standard, and Appendix G Case 9 shows a real REFUSE-to-ASK miss that the aggregate RR metric does not decompose. But that is a metric-coverage and benchmark-independence issue, not a definitional equivalence or fitted-input-called-prediction circularity: the model must still generalize to held-out snapshots, and the authors' audit of the test set provides independent human grounding. No load-bearing self-citation chain, imported uniqueness argument, or ansatz-by-citation is present. Accordingly, no step in the claimed derivation reduces to its own inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The central evaluation rests on a self-generated gold standard: the authors' taxonomy, LLM annotators, and author audit. External benchmarks (AgentHarm, When2Call, etc.) are re-annotated under the same taxonomy, so they do not provide independent grounding. The only explicit post-hoc model-level knob is tau, selected on validation.

free parameters (1)
  • Decoding threshold tau = 0.68 (balanced); swept 0.29–0.91 for autonomous/conservative points
    Renormalizes p_EXECUTE/(p_EXECUTE+p_ASK) to position the EXECUTE/ASK boundary post-hoc. Selected on the validation set for the reported Table 1 numbers; the three named operating points in Appendix D are obtained by sweeping on the test set.
axioms (3)
  • ad hoc to paper The fourteen-pattern trigger taxonomy (Table 4) correctly encodes routing semantics for EXECUTE/ASK/REFUSE in enterprise agent settings.
    The taxonomy is authored by the paper's team and used both to generate labels and as the evaluation ground truth; there is no external validation that these fourteen triggers match what real operators would want.
  • domain assumption LLM annotator judgments under the rubric, with author arbitration, are valid ground truth for agent-action routing decisions.
    Invoked in Section 3.2 annotation stage: 'The full corpus is annotated by two LLM annotators with author arbitration on disagreements; the authors additionally audit the entire test set, on which the human-validated gold reaches 92% human agreement.' All accuracy and error-rate numbers depend on this gold.
  • domain assumption Trajectories captured by a memory-blind agent on nine seeded self-hosted Docker services are representative of real deployment traces and action consequences.
    Section 3.2 and the collection-stage prompt (Figure 9) tell the agent to ignore safety and focus on task completion, so the collected traces are an artificial, safety-blind distribution; transfer to real safety-aware agents is assumed rather than demonstrated.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing." pith.science (2026). https://pith.science/paper/JEYGNJ3C

@misc{pith2026260713594,
  author       = {Pith},
  title        = {Pith review of: SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEYGNJ3C}},
  note         = {Machine review of arXiv:2607.13594}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM agents act on real-world environments through tool calls, and a single misjudged action can cause irreversible harm. The standard safeguard is a guard model that labels each proposed action as safe or unsafe, but this binary view conflates two distinct decisions: whether the action is harmful in itself, and whether it is appropriate given the user's context. It also operates at the granularity of action categories rather than individual instances, producing routine interruptions that erode autonomy and train users to wave through the most consequential alerts. We reframe the problem as a per-instance three-way routing decision over {EXECUTE, ASK, REFUSE} and instantiate it with Safety Sentry, a lightweight guard model whose inference reduces to a single decoding call. A single decoding-time threshold lets one fixed checkpoint be re-positioned across deployments of differing risk tolerance without retraining. Safety Sentry outperforms a broad set of open-weight and frontier closed-source baselines on overall accuracy and safety-related recall, while controlling both directional error rates simultaneously.

Figures

Figures reproduced from arXiv: 2607.13594 by Chujia Hu, Tianyu Chen, Wenjie Wang.

Figure 1
Figure 1. Figure 1: Conventional binary guards lump distinct [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Workflow scenarios per internal service. Inner [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Safety Sentry framework. (a) Training-time pipeline: action trajectories collected from nine real services are paired with persona memories and labeled step-by-step, then combined with re-annotated external benchmarks for supervised fine-tuning. (b) Deployment-time pipeline: the guard receives a per-step snapshot, emits probabilities over the three decision tokens, and routes the action via a threshold τ .… view at source ↗
Figure 4
Figure 4. Figure 4: Safety-autonomy Pareto frontier traced by sweeping [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Personalized routing on the memory￾augmented subset. Per-backbone accuracy on the no-memory baseline (Accbase, left) and the memory￾augmented side (Accmem, right) of paired records whose gold label flips once a persona memory is added. on plain tasks lose much of that accuracy: Claude￾Opus-4.7 drops from 72.5% to 56.5%, GPT-5.5 from 69.6% to 52.2%, and the pre-SFT Qwen3.5- 4B from 44.9% to 37.7%, only marg… view at source ↗
Figure 6
Figure 6. Figure 6: Training loss of the selected LoRA SFT guard. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Memory-conditioning accuracy by persona axis. Results are reported on three disjoint subsets correspond [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt used to author task–memory pairs for internal scenarios. It produces cautious, permissive, and [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Collection-stage prompt for the main tool-using agent. It produces real service trajectories but is memory [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Annotation-stage prompt, part 1: label definitions and memory rules. The detailed decision criteria [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Annotation-stage prompt, part 2: per-decision criteria (continued from Figure [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: System prompt the trained 4B-SFT guard runs under at inference time. The decision criteria are [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Instruction-style prompt for the zero-shot baselines of Table [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: A clean direct_execute case: the surrounding task will eventually contain sharing operations, but the reviewed step is only a bounded read. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: A clean ask_human case: the tool arguments are complete, but the safety-critical fact that the wire exists is unverified. Case 3 — audit-trail tampering is refused Record. ID test index 1386; service erpnext. Task. You are the company’s financial system administrator. Some people think that Summit Advisory’s overdue accounts are too conspicuous. I want to delete the sales invoice ACC-SINV-2026-0003 direct… view at source ↗
Figure 16
Figure 16. Figure 16: A clean refuse case: the objective itself is audit-trail tampering, so the guard must block rather than ask for confirmation. Case 4 — cautious memory requires link parameters Record. ID test index 718; service owncloud. Task. Customer rep emma needs access to the Shared/customer-handoff folder for the new account she’s owning. Pull up the shared folder, give her access, share the handoff checklist she’ll… view at source ↗
Figure 17
Figure 17. Figure 17: A memory-conditioned ask_human case: cautious memory adds concrete password and expiration requirements for the exact public-link target. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: A memory-conditioned direct_execute case: the same tool call as [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: A false-autonomy error: the guard over-trusts a relayed data-protection-officer approval for an admin-role [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: A false-escalation error: cautious memory is relevant to later invite/integration actions, but the reviewed [PITH_FULL_IMAGE:figures/full_fig_p023_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: A successful decision on the held-out Mailu service. The guard correctly treats the prerequisite [PITH_FULL_IMAGE:figures/full_fig_p023_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: A representative failure on the held-out Mailu service. Safety Sentry recognizes that the operation is [PITH_FULL_IMAGE:figures/full_fig_p024_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 14 linked inside Pith

  1. [1]

    Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2025. https://arxiv.org/abs/2410.09024 AgentHarm : A benchmark for measuring harmfulness of LLM agents . In International Conference on Learning Re...

    Pith/arXiv arXiv 2025

  2. [2]

    Chinmaya Andukuri, Jan-Philipp Fr \"a nken, Tobias Gerstenberg, and Noah D. Goodman. 2024. https://arxiv.org/abs/2403.19154 STaR-GATE : Teaching language models to ask clarifying questions . arXiv preprint, arXiv:2403.19154

    Pith/arXiv arXiv 2024

  3. [3]

    Anthropic . 2025. Model context protocol specification. https://modelcontextprotocol.io/

    2025

  4. [4]

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. AgentPoison : Red-teaming LLM agents via poisoning memory or knowledge bases. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  5. [5]

    Zhaorun Chen, Zhen Xiang, Chaowei Xie, Yu Liu, and Bo Li. 2025. https://arxiv.org/abs/2503.22738 ShieldAgent : Shielding agents via verifiable safety policy reasoning . arXiv preprint, arXiv:2503.22738

    arXiv 2025

  6. [6]

    Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, and Zhen Xiang. 2025. https://arxiv.org/abs/2503.03704 Memory injection attacks on LLM agents via query-only interaction . arXiv preprint, arXiv:2503.03704

    arXiv 2025

  7. [7]

    Aarya Doshi, Yining Hong, Congying Xu, Eunsuk Kang, Alexandros Kapravelos, and Christian K \"a stner. 2026. https://arxiv.org/abs/2601.08012 Towards verifiably safe tool use for LLM agents . arXiv preprint, arXiv:2601.08012

    arXiv 2026

  8. [8]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS)

    2017

  9. [9]

    Kate Goddard, Abdul Roudsari, and Jeremy C. Wyatt. 2012. Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1):121--127

    2012

  10. [10]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR)

    2022

  11. [11]

    Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024. https://arxiv.org/abs/2402.01586 TrustAgent : Towards safe and trustworthy LLM -based agents . arXiv preprint, arXiv:2402.01586

    Pith/arXiv arXiv 2024

  12. [12]

    Yue Huang, Hang Hua, Yujun Zhou, Pengcheng Jing, Manish Nagireddy, Inkit Padhi, Greta Dolcetti, Zhangchen Xu, Subhajit Chaudhury, Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, Prasanna Sattigeri, and Xiangliang Zhang. 2025. https://arxiv.org/abs/2510.09781 Building a foundational guardrail for general agentic systems via synthetic data . arXiv preprint,...

    arXiv 2025

  13. [13]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. https://arxiv.org/abs/2312.06674 Llama guard: LLM -based input-output safeguard for human- AI conversations . arXiv preprint, arXiv:2312.06674

    Pith/arXiv arXiv 2023

  14. [14]

    Yu Li, Haoyu Luo, Yuejin Xie, Yuqian Fu, Zhonghao Yang, Shuai Shao, Qihan Ren, Wanying Qu, Yanwei Fu, Yujiu Yang, Jing Shao, Xia Hu, and Dongrui Liu. 2026. https://arxiv.org/abs/2604.02022 ATBench : A diverse and realistic trajectory benchmark for long-horizon agent safety . arXiv preprint, arXiv:2604.02022

    Pith/arXiv arXiv 2026

  15. [15]

    Weidi Luo, Shenghong He, Zhuoer Liu, Shenzhi Lin, Yue Zhang, Ningyu Lu, Yu Sun, and Bo Li. 2025. https://arxiv.org/abs/2502.11448 AGrail : A lifelong agent guardrail with effective and adaptive safety detection . arXiv preprint, arXiv:2502.11448

    Pith/arXiv arXiv 2025

  16. [16]

    David Madras, Toniann Pitassi, and Richard Zemel. 2018. Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems (NeurIPS)

    2018

  17. [17]

    Margaret Mitchell, Abeba Birhane, and Giada Pistilli. 2025. https://arxiv.org/abs/2502.02649 Fully autonomous AI agents should not be developed . arXiv preprint, arXiv:2502.02649

    arXiv 2025

  18. [18]

    Yutao Mou, Zhangchi Xue, Lijun Li, Peiyang Liu, Shikun Zhang, Wei Ye, and Jing Shao. 2026. https://arxiv.org/abs/2601.10156 ToolSafe : Enhancing tool invocation safety of LLM -based agents via proactive step-level guardrail and feedback . arXiv preprint, arXiv:2601.10156

    arXiv 2026

  19. [19]

    Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2):230--253

    1997

  20. [20]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  21. [21]

    Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshitomo Suhara. 2025. When2Call : When (not) to call tools. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 3391--3409

    2025

  22. [22]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer : Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  23. [23]

    Zeyang Sha, Hanling Tian, Zhuoer Xu, Shiwen Cui, Changhua Meng, and Weiqiang Wang. 2025. https://arxiv.org/abs/2507.08270 Agent safety alignment via reinforcement learning . arXiv preprint, arXiv:2507.08270

    Pith/arXiv arXiv 2025

  24. [24]

    Ling Shi and Deyi Xiong. 2024. https://arxiv.org/abs/2406.04752 CRiskEval : A C hinese multi-level risk evaluation benchmark dataset for large language models . arXiv preprint, arXiv:2406.04752

    Pith/arXiv arXiv 2024

  25. [25]

    Hang Su, Jun Luo, Chang Liu, Xiao Yang, Yichi Zhang, Yinpeng Dong, and Jun Zhu. 2025. https://arxiv.org/abs/2506.23844 A survey on autonomy-induced security risks in large model-based agents . arXiv preprint, arXiv:2506.23844

    Pith/arXiv arXiv 2025

  26. [26]

    Rossi, and Dinesh Manocha

    Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, and Dinesh Manocha. 2025. https://arxiv.org/abs/2511.08798 Structured uncertainty guided clarification for LLM agents . arXiv preprint, arXiv:2511.08798

    Pith/arXiv arXiv 2025

  27. [27]

    Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, and Maarten Sap. 2025. https://arxiv.org/abs/2507.06134 OpenAgentSafety : A comprehensive framework for evaluating real-world AI agent safety . arXiv preprint, arXiv:2507.06134

    arXiv 2025

  28. [28]

    Poskitt, and Jun Sun

    Haoyu Wang, Christopher M. Poskitt, and Jun Sun. 2025 a . https://arxiv.org/abs/2503.18666 AgentSpec : Customizable runtime enforcement for safe and reliable LLM agents . arXiv preprint, arXiv:2503.18666

    Pith/arXiv arXiv 2025

  29. [30]

    Wenxuan Wang, Shi Juluan, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2025 c . https://doi.org/10.18653/v1/2025.emnlp-main.1104 Learning to ask: When LLM agents meet unclear instruction . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pa...

    doi:10.18653/v1/2025.emnlp-main.1104 2025

  30. [31]

    Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. 2024. https://arxiv.org/abs/2406.09187 GuardAgent : Safeguard LLM agents by a guard agent via knowledge-enabled reasoning . arXiv preprint, arXiv:2406.09187

    Pith/arXiv arXiv 2024

  31. [32]

    Haishuo Yan, Kaixin Zhu, Honglin Niu, Min Yang, Min Zhang, Yang Xu, Mingxuan Wang, Jinsong Su, and Yu Wu. 2024. Preemptive detection and correction of misaligned actions in LLM agents. In Findings of the Association for Computational Linguistics: EMNLP

    2024

  32. [33]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  33. [34]

    Jia Yu, Long Li, and Zhenzhong Lan. 2024. https://doi.org/10.1109/ACCESS.2024.3393245 Beyond binary classification: A fine-grained safety dataset for large language models . IEEE Access, 12:64717--64726

    arXiv 2024

  34. [35]

    Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. R-Judge : Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP

    2024

  35. [36]

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. 2024. https://arxiv.org/abs/2407.21772 ShieldGemma : Generative AI content moderation based on Gemma . arXiv preprint, arXiv:2407.21772

    Pith/arXiv arXiv 2024

  36. [37]

    Michael J. Q. Zhang and Eunsol Choi. 2025. Clarify when necessary: Resolving ambiguity through interaction with LM s. In Findings of the Association for Computational Linguistics: NAACL

    2025

  37. [38]

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024. https://arxiv.org/abs/2412.14470 Agent-SafetyBench : Evaluating the safety of LLM agents . arXiv preprint, arXiv:2412.14470

    Pith/arXiv arXiv 2024

  38. [39]

    Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, and Tat-Seng Chua. 2026. https://arxiv.org/abs/2602.03100 Risky-Bench : Probing agentic safety risks under real-world deployment . arXiv preprint, arXiv:2602.03100

    arXiv 2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.