AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration

Jizhou Chen; Samuel Lee Cong

REVIEW 3 major objections 5 minor 3 cited by

AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AgentGuard claims an LLM agent's own orchestrator can serve as its safety evaluator, discovering unsafe tool workflows, validating them by real execution, and producing tested constraints for deployment.

desk verdict An honest, clearly written hackathon prototype with a genuinely interesting core idea—using the target orchestrator as its own safety evaluator—but the evidence is too thin and self-referential to call the feasibility case closed. read the letter →

arxiv 2502.09809 v1 pith:XXOJ3FF4 submitted 2025-02-13 cs.CR cs.AI

Jizhou Chen , Samuel Lee Cong This is my paper

classification cs.CRcs.AI

keywords LLMagentstoolorchestrationsafetyself-evaluationunsafeworkflowdiscoveryconstraintsSELinuxagentred-teamingpre-deploymenthardening

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AgentGuard is a proposed pre-deployment testing and hardening pipeline for LLM agents that use tools. Its central idea is to reuse the agent's own orchestrator as the safety evaluator: the orchestrator already knows the tools, can generate realistic workflows, and holds execution privileges, so the same entity that plans tool calls can be asked to plan unsafe ones, execute them, and confirm the harm. The pipeline runs four phases—identify unsafe workflows, validate them by real execution, generate safety constraints, and validate those constraints—and outputs an evaluation report with confirmed unsafe workflows, test cases, and constraints that are ready to enforce. The paper demonstrates feasibility with a prototype built around a coding agent, reporting that successful end-to-end runs occurred even though many runs failed because the evaluated LLM could not reliably produce applicable SELinux rules.

What carries the argument

The central object is the LLM-based orchestrator inside the target agent, repurposed as its own safety evaluator. It carries the argument because it already satisfies the three requirements the authors argue any such framework must have: internal knowledge of tool capabilities and usage, the ability to generate workflows that realistically match what the agent would actually do, and the privilege to invoke tools for concrete validation. The pipeline is coordinated by a centralized Prompting Proxy Agent and a Safety Constraint Expert Agent, with feedback loops that request regeneration when generated content is low quality or execution fails. Two prompt-level mechanisms were needed to make the prototype work: Role Augmentation to get past the orchestrator's initial refusal to perform the evaluation task, and Chain-of-Thought prompting, asking the orchestrator to list and reflect on tools before identifying workflows, to suppress hallucinated tool capabilities.

What would settle it

Take a coding agent with a known unsafe tool sequence that a manual red team can execute to overwrite a protected file; run AgentGuard on that exact configuration many times, and if the pipeline never validates that workflow and never applies a successfully blocking SELinux rule across all runs, the central feasibility claim for the hardening loop is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that an LLM-based agentic orchestrator can be repurposed, before deployment, into the safety evaluator of its own tool orchestration. The orchestrator is instructed to reflect on its tools, produce unsafe workflows that violate security principles in given task scenarios, turn those workflows into executable test cases, execute them to check whether the predicted harm actually occurs, and then re-execute them under generated safety constraints to confirm the constraints block the harm. The evaluation report thereby contains validated unsafe workflows, violated security principles, test cases, and validated constraints, which can be enforced as a deployment-time safety baseline. The empirical result is a feasibility proof-of-concept: with a coding agent as the target, the pipeline ran end to end successfully at least a few times, despite many runs failing at the step of applying LLM-generated SELinux rules, a failure the authors attribute to the evaluated LLM rather than to the design.

Load-bearing premise

The load-bearing premise is that the target agent's orchestrator is not compromised and will actively comply with requests to expose unsafe workflows, and in the prototype this premise failed initially—the orchestrator refused the task and had to be jailbroken with Role Augmentation—so if a real agent hides or distorts its unsafe workflows, or the same LLM cannot turn constraints into executable rules, the resulting evaluation report does not cover the agent's actual risk surface.

Editorial extensions

If this is right

A deployable agent can be shipped with a validated set of safety constraints, such as SELinux rules, that confine its behavior even if the agent is later compromised.
Re-running the four phases produces a quantitative measure of safety improvement by comparing unsafe-outcome occurrences before and after constraints are applied.
Collected unsafe workflows and test cases for one agent family can be pooled into a benchmark corpus and shared as threat intelligence for similar agents.
Accumulated evaluation reports can serve as training samples to fine-tune LLMs for safer tool orchestration.
Because the method rests on the general design of LLM agents, it is intended to transfer from software agents to agents that interact with the physical world, such as robotic systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to use the same self-evaluation loop continuously at deployment with a shadow orchestrator, turning each real interaction into a latent test case rather than auditing only once before release.
The initial refusal and tool hallucination observed in the prototype suggest a general precondition for self-auditing: the orchestrator's system prompt must be audited or temporarily stripped of defensive role assignments, and the actual tool registry should be fed back into the prompt before workflow generation.
A more robust design would separate the evaluator model from the target agent's model, or pair the orchestrator with a tool-knowledge retriever, to avoid relying on the same LLM for both planning and rule generation.
An attacker will not ask a compliant orchestrator politely, so the identification phase could be extended with goal-injection prompts that simulate adversarial hijacking, testing workflows the agent would only produce under compromise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a clearly written hackathon prototype with one genuinely new idea and very little evidential weight. The new idea is to repurpose the target agent's own LLM orchestrator as the safety evaluator: it proposes unsafe tool workflows, executes them in the real environment, and then generates and validates SELinux constraints to block them. That is different from TooLEmu, which emulates tools with an LM. Running against real SELinux on a real VM is also a step up from a simulated sandbox. The paper is honest about the rough edges: it reports that the orchestrator initially refused the task, that hallucination required Chain-of-Thought prompting, and that most generated SELinux rules were unusable. The Role Augmentation observation is a useful incidental finding.

The soft spots are real and load-bearing. The empirical base is one target agent (Aider), one model (ChatGPT-4o), one configuration, no reported counts, no code, no prompts, no logs. The paper says 'very few' end-to-end runs succeeded. More importantly, in this implementation the same model is the workflow proposer, test-case author, executor, outcome detector, SELinux rule writer, and post-constraint checker. A 'validated' unsafe workflow is therefore self-reported by the very model under test. The stress-test's 'self-consistent hallucination' phrasing is a bit strong when the SELinux denial is real, but the point stands: without an independent oracle reading the execution logs, a successful run cannot distinguish a real unsafe outcome from a bug in the model-written detection code, nor a genuine denial from the model changing its tool calls. The paper also had to jailbreak its own orchestrator before it would cooperate, so the entity evaluated is not identical to the moderated deployment agent. The abstract's 'baseline of safety guarantee' language overstates what a few self-validated workflows support.

The paper is for people working on LLM agent safety who want an early, honest sketch of a self-evaluating-agent approach. It is not a benchmark or a system paper. I would not cite it as evidence that AgentGuard works, but I might cite it as an early exploration of the idea. It deserves a serious referee if positioned as a workshop or exploratory paper; the core question is worth asking and the limitations are stated clearly. The referee request should insist on reproducible artifacts, quantitative success rates, and an independent log-based oracle. Without those, the central feasibility claim remains plausible but unproven.

Referee Report

3 major / 5 minor

Summary. The paper proposes AgentGuard, a pre-deployment safety testing and hardening framework for LLM agents. The key idea is to repurpose the agent's own LLM-based orchestrator as its safety evaluator: the orchestrator identifies unsafe tool-use workflows, generates and executes concrete test cases to validate that they cause unsafe outcomes, a Safety Constraint Expert Agent converts validated workflows into SELinux constraints, and the orchestrator re-executes the test cases to show that the constraints block the unsafe outcomes. The deliverable is an evaluation report containing unsafe workflows, test cases, and validated constraints. The authors evaluate the prototype on the Aider coding agent with ChatGPT-4o on an Ubuntu 22.04 VM with SELinux enabled, reporting that most runs failed at SELinux rule generation or application but that a few successful runs demonstrated proof-of-concept feasibility.

Significance. If the feasibility claim is substantiated, AgentGuard addresses a real need: a standardized pre-deployment process for discovering and mitigating unsafe tool orchestration in LLM agents. The paper's core design rationale is sensible and well articulated: the orchestrator has tool knowledge, has execution privileges, and can generate realistic workflows precisely because it is the same entity that will be exploited at deployment. The four-phase structure and the concrete SELinux embodiment make the proposal actionable, and the authors are transparent about failures, including the orchestrator's initial refusal and the difficulty of generating applicable SELinux rules. The comparison with TooLEmu is useful. However, the empirical evidence is qualitative and self-referential: one target agent, one model, one configuration, no success counts, no example logs, and no independent oracle for whether an unsafe outcome actually occurred and was actually blocked. The paper is best read as a position/exploratory report; its central feasibility claim needs significantly stronger evidence before it can support a general conclusion.

major comments (3)

[Section 4 and Section 6] The central feasibility claim, stated in Section 6 as 'we observed successful evaluation results indicating the feasibility of AGENT GUARD', is not supported by the reported evidence. The evaluation section reports qualitative observations (e.g., 'only a very few' rules could be applied) but provides no success counts, no number of identified versus validated workflows, no example test cases, no SELinux rule examples, and no logs of denials or outcome detections. A reader cannot tell how many end-to-end runs succeeded, which workflows were validated, or whether the constraints actually blocked the unsafe effects. At minimum, the paper needs one complete end-to-end trace with independent verification, for example OS-level audit logs showing the unsafe operation occurring before the constraint and being denied after it.
[Section 4, Unsafe Workflow Identification, versus Assumption 3 in Section 2.1] Assumption 3 states that the orchestrator is not compromised and will actively expose unsafe workflows as requested. In practice, the orchestrator initially refused the task and had to be jailbroken with Role Augmentation prompting; it also hallucinated non-existent tool capabilities until Chain-of-Thought prompting was added. This means the entity that was evaluated is not the compliant, unmodified orchestrator assumed by the design, nor is it the same as the deployment-time agent after moderation and hardening. The paper should either evaluate a genuinely compliant orchestrator, or explicitly analyze how the jailbroken state affects the validity and transferability of the generated safety constraints. Without this, the evaluation report may not cover the actual risk surface of the agent as it would behave at deployment.
[Section 3 and Section 4, Safety Constraint Validation] The validation loop is self-referential. As described in Section 3, the Safety Constraint Expert Agent is 'the orchestrator equipped with knowledge in SELinux from the underlying LLM', so the same model generates the workflows, writes the test cases, executes them, detects unsafe outcomes, writes the SELinux constraints, re-executes the test cases, and checks whether outcomes are blocked. A successful post-constraint run could therefore reflect a genuine SELinux denial, but it could also reflect the model's own changed choice of tool calls or a flawed outcome-detection check. There is no independent oracle on the execution logs. The authors should add at least one run in which an independent mechanism (e.g., a SELinux audit log, a filesystem snapshot, or a human reviewer) confirms both that the unsafe outcome occurred before the constraint and that the constraint, not the model's changed behavior, prevented it afterward.

minor comments (5)

[Section 4, Safety Constraint Generation] The statement that SELinux rule generation 'appeared to be decent' and 'yielded rules with correct syntax ... for a fair amount of times' is too vague for a scientific report; please provide the number of generated rules, the number syntactically valid, and the number actually applied and validated.
[Section 1.2 and Section 3] Figure 1 and Section 2.2 describe the Prompting Proxy Agent and the Safety Constraint Expert Agent as distinct components, but Section 3 states that the same underlying LLM currently plays both roles. Please clarify which components are actually separate in the implementation and which are the same model under different prompts, as this matters for interpreting the results.
[Section 4, Unsafe Workflow Identification] The claim that 'the vanilla underlying foundation model (i.e., ChatGPT 4o) does not have content moderation for such a benign task' is asserted without evidence and is difficult to verify from the outside; please either show the relevant experiments or rephrase as a hypothesis about implicit prompt-based moderation.
[Section 2.3 and Section 6] Section 2.3 lists 'the difference in the occurrences of unsafe outcomes before and after applying the safety constraints' as a quantitative improvement measure, but the evaluation section does not report this number for any workflow. Either report it for the successful runs or explicitly defer it to future work in Section 6.
[Throughout] The manuscript lacks reproducibility details: the exact system prompts, the Role Augmentation prompt template, the Chain-of-Thought prompt, the test-case code, and the SELinux rules are not included. Providing an appendix with these artifacts, or a repository link, would substantially strengthen the paper. There are also several grammatical slips (e.g., 'the orchestrator's should be good at generating tool-use plans' in Section 4) that should be corrected.

Circularity Check

2 steps flagged · score 6.0 of 10

The 'validated unsafe workflows' and 'validated safety constraints' are both produced and certified by the same orchestrator under evaluation; the constraint-validation step re-tests the exact test cases the constraints were generated from, so the feasibility proof is an in-sample self-consistency check.

self definitional [Section 2.2.2 (Unsafe Workflow Validation); Section 3 (Implementation)]
"the Prompting Proxy Agent instructs the orchestrator to 1) generate corresponding test cases including concrete orchestration plans (e.g., sequences of tool API calls) representing the workflows at the execution level utilizing its internal knowledge of tool use, and the corresponding unsafe outcome detection mechanism (e.g., checking if writing to restricted a directory successfully overwrites files in it) and 2) execute the test cases to validate that the identified workflows indeed can result in unsafe outcomes, utilizing its privileges to invoke the tools."

The orchestrator under evaluation is also the entity that proposes the unsafe workflow, writes the concrete test case, and writes the 'unsafe outcome detection mechanism' that determines whether an unsafe outcome occurred. A workflow is marked validated only when the orchestrator's own self-authored detector reports an unsafe outcome; no independent oracle inspects the execution logs or defines the unsafe predicate. The validated set is therefore self-defined by the model's own knowledge and hallucinations. Section 3 collapses the ostensible independent constraint expert into the same model: 'we currently use the orchestrator equipped with knowledge in SELinux from the underlying LLM as the Safety Constraint Agent.'
fitted input called prediction [Section 2.2.4 (Safety Constraint Validation), with Section 2.2.3]
"for each unsafe workflow, the Safety Constraint Expert Agent applies the safety constraints to the execution environment of the target agent, then instructs the orchestrator to re-execute the test cases and check if the unsafe outcomes are blocked. If so, the set of safety constraints generated for this unsafe workflow are validated and added to the report."

Section 2.2.3 says the constraint agent generates constraints by examining the observed unsafe outcomes from test case execution and correlating them with the tool invocations in the test case. The validation in Section 2.2.4 then re-executes those same test cases, with the same orchestrator checking whether the outcomes are blocked. The constraint is thus fitted to the exact test case it is later tested on; no held-out workflow, new orchestration, or independent verifier is used. The deploy-time 'safety guarantee baseline' claim (Section 2.3) is therefore a prediction on the training data, forced to succeed by construction once the SELinux rule is syntactically applicable.

full rationale

No citation-based circularity is present: the paper does not lean on self-citations or imported uniqueness theorems. The circularity is in the empirical validation chain. The orchestrator under test generates the unsafe workflows, the test cases, and the unsafe-outcome detectors (Section 2.2.2), and the same model is used as the Safety Constraint Expert in the implementation (Section 3). The constraint validation (Section 2.2.4) re-executes the same test cases the constraints were generated from, so a successful end-to-end run demonstrates self-consistency rather than independent safety coverage. The concrete execution in a real VM with SELinux provides some grounding (file operations either succeed or are denied by the OS), which is why this is partial circularity rather than a fully vacuous derivation. The role-augmentation jailbreak also means the evaluated orchestrator behavior differs from the moderated deployment agent, but that is an external-validity caveat, not a circular step. Overall score 6: the flagship feasibility result reduces in part to in-sample self-validation by the same model.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No mathematical derivation or fitted parameters; the ledger entries are the framework's stated assumptions in Section 2.1, the SELinux rule generation assumption in Section 2.2.3, and the execution-as-ground-truth assumption in Section 2.2.2.

assumptions (6)

domain assumption The target agent is hosted in a controlled environment during testing.
Stated in Section 2.1 Assumption 1; the paper excludes building a controlled environment, so the framework assumes one exists.
domain assumption AgentGuard runs before moderation, so the orchestrator is compliant with safety evaluation requests.
Stated in Section 2.1 Assumption 2; violated in practice, requiring Role Augmentation prompting in Section 4.
ad hoc to paper The orchestrator is not compromised and will actively expose unsafe workflows as requested.
Stated in Section 2.1 Assumption 3. This is the load-bearing self-evaluation premise; if the orchestrator hides risks, the report misses them.
domain assumption The orchestrator has enough security knowledge from LLM pretraining to evaluate tool workflows.
Stated in Section 2.1 Assumption 4; the paper's own hallucination and SELinux label failures in Section 4 show this is only partially satisfied.
domain assumption The safety constraint expert can generate applicable constraints, such as valid SELinux rules.
Required in Section 2.2.3; the paper reports in Section 4 that the LLM frequently generated rules with undefined custom types that could not be applied.
domain assumption Executing a generated test case and observing the unsafe outcome is a valid ground truth for workflow risk.
Used in Section 2.2.2 for validation; it assumes the detection mechanism fully captures harm and that test cases are realistic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration." pith.science (2026). https://pith.science/paper/XXOJ3FF4

@misc{pith2026250209809,
  author       = {Pith},
  title        = {Pith review of: AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXOJ3FF4}},
  note         = {Machine review of arXiv:2502.09809}
}

read the original abstract

The integration of tool use into large language models (LLMs) enables agentic systems with real-world impact. In the meantime, unlike standalone LLMs, compromised agents can execute malicious workflows with more consequential impact, signified by their tool-use capability. We propose AgentGuard, a framework to autonomously discover and validate unsafe tool-use workflows, followed by generating safety constraints to confine the behaviors of agents, achieving the baseline of safety guarantee at deployment. AgentGuard leverages the LLM orchestrator's innate capabilities - knowledge of tool functionalities, scalable and realistic workflow generation, and tool execution privileges - to act as its own safety evaluator. The framework operates through four phases: identifying unsafe workflows, validating them in real-world execution, generating safety constraints, and validating constraint efficacy. The output, an evaluation report with unsafe workflows, test cases, and validated constraints, enables multiple security applications. We empirically demonstrate AgentGuard's feasibility with experiments. With this exploratory work, we hope to inspire the establishment of standardized testing and hardening procedures for LLM agents to enhance their trustworthiness in real-world applications.

Figures

Figures reproduced from arXiv: 2502.09809 by the authors.

**Figure 1.** Overview of AGENTGUARD. AGENTGUARD has three key components: 1) The LLM-based orchestrator within the target agent under evaluation, 2) A Safety Constraint Expert Agent responsible for safety constraint generation, and 3) A centralized Prompting Proxy Agent to instruct the other two components to perform testing and hardening. AGENTGUARD works in four main phases: 1) Unsafe Workflow Identification, 2) Unsafe Workflo… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents
cs.CR 2025-09 conditional novelty 6.0 of 10

AgentSentinel combines system-level tracing with LLM-based auditing to block 79.6% of attacks in the authors' 60-scenario computer-use agent benchmark.
WebGuard: Building a Generalizable Guardrail for Web Agents
cs.AI 2025-07 conditional novelty 6.0 of 10

WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...
GAF-Guard: An Agentic Framework for Risk Management and Governance in Large Language Models
cs.CL 2025-07 conditional novelty 5.0 of 10

The paper presents GAF-Guard, a multi-agent governance framework that combines pre-deployment risk questionnaires, real-time drift and risk monitoring, and incident reporting for LLM applications.

Reference graph

Works this paper leans on

5 extracted references · 5 canonical work pages · cited by 3 Pith papers

[1]

Langchain: Building applications with llms

Harrison Chase and LangChain Contributors. Langchain: Building applications with llms. Accessed: 2024-12-06

work page 2024
[2]

Fastapi: Fastapi framework, high performance, easy to learn, fast to code, ready for production

FastAPI. Fastapi: Fastapi framework, high performance, easy to learn, fast to code, ready for production. Accessed: 2024-12-06

work page 2024
[3]

Aider: Ai pair programming in your terminal

Paul Gauthier and Aider Contributors. Aider: Ai pair programming in your terminal. Accessed: 2024-12-06

work page 2024
[4]

Security-enhanced linux (selinux), 2000

National Security Agency (NSA). Security-enhanced linux (selinux), 2000. Accessed: 2025-02-06

work page 2000
[5]

Maddison, and Tatsunori Hashimoto

Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox. In The Twelfth International Conference on Learning Repre- sentations, 2024. 5

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Langchain: Building applications with llms

Harrison Chase and LangChain Contributors. Langchain: Building applications with llms. Accessed: 2024-12-06

work page 2024

[2] [2]

Fastapi: Fastapi framework, high performance, easy to learn, fast to code, ready for production

FastAPI. Fastapi: Fastapi framework, high performance, easy to learn, fast to code, ready for production. Accessed: 2024-12-06

work page 2024

[3] [3]

Aider: Ai pair programming in your terminal

Paul Gauthier and Aider Contributors. Aider: Ai pair programming in your terminal. Accessed: 2024-12-06

work page 2024

[4] [4]

Security-enhanced linux (selinux), 2000

National Security Agency (NSA). Security-enhanced linux (selinux), 2000. Accessed: 2025-02-06

work page 2000

[5] [5]

Maddison, and Tatsunori Hashimoto

Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox. In The Twelfth International Conference on Learning Repre- sentations, 2024. 5

work page 2024