Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This position paper claims that MCP-powered LLM agents are genuinely vulnerable to attacks from third-party services that embed prompt injections in service descriptions or returned responses, and that simple detection cannot reliably…

desk verdict Useful position paper with a released framework, but the MCP-specific attribution is not yet established—the evidence shows prompt injection works on tool-using agents, which is real but not unique to MCP. read the letter →

arxiv 2506.13666 v1 pith:B5VW62PT submitted 2025-06-16 cs.LG cs.AI

Junfeng Fang , Zijun Yao , Ruipeng Wang , Haokai Ma , Xiang Wang , Tat-Seng Chua This is my paper
classification cs.LGcs.AI
keywords modelcontextprotocolMCPsafetypromptinjectionthird-partyservicesLLMagentsredteamingAIagentsecuritytool-use
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the Model Context Protocol (MCP), the emerging standard that lets LLM agents call external tools and data services, introduces a new class of safety risk: the third-party service provider is a new actor with economic incentives to attack the agent, beyond the usual user-LLM dyad. To ground this claim, the authors build a controlled evaluation framework, SAFEMCP, that simulates MCP services and lets attackers inject prompts into service descriptions and returned responses. In pilot experiments across eight LLM backbones and six attack methods, every implemented agent was susceptible to at least one attack, and direct attacks alone degraded task accuracy substantially. Simple defenses, such as asking the backbone LLM to flag malicious services or using content-moderation tools, missed several advanced attacks, while an active defense that extracts and sanitizes information reduced harm but also reduced helpfulness. The paper therefore advocates systematic research on MCP-specific red teaming, backbone LLM alignment, evaluation, data, service safeguard, and ecosystem governance.

What carries the argument

The central objects are the Model Context Protocol (MCP), a standard interface that tells an LLM agent how to call external tools and data services, and the tripartite interaction it creates among user, agent, and third-party service. The load-bearing mechanism is prompt injection through the two text surfaces a third-party service controls: its service description and its returned response. The paper's SAFEMCP framework operationalizes this by turning ten agent environments into MCP services and exposing six attack strategies (Direct, AutoDAN, CodeChameleon, DeepInception, CipherChat, ReNeLLM), while measuring relative accuracy loss, attack success rate, harm rate, and detection ratio. These components carry the argument because they convert the conceptual claim that third parties can attack agents into measurable vulnerability and defense outcomes.

What would settle it

Run the same six attack methods on a production MCP deployment where service descriptions are stripped from the context given to the agent and tool outputs are validated against typed schemas; if attack success and harm rates stay at pre-attack baselines, the demonstrated vulnerability is specific to the simulation setup rather than to MCP deployments that sanitize service text.

Watch

Extended reading notes

Core claim

The central claim is that MCP-powered agent systems are vulnerable to attacks from MCP-supported third-party services, stated in the paper as Advocation 1. Concretely, the paper reports that injecting attack prompts into a service's description or into the text a service returns can raise attack success rate and harm rate and lower accuracy across GPT, Qwen, and Doubao backbones; all tested agents fell to at least one strategy. A second finding is that passive detection, whether by asking the LLM to judge whether a service is malicious or by running dedicated moderation models, catches direct attacks but fails on advanced rewrites such as ReNeLLM, CodeChameleon, DeepInception, and CipherChat. The third finding is that active defense, which paraphrases and extracts useful content from service responses, reduces attack success rate and harm rate but causes relative accuracy loss, so it is promising but not free. Taken together, the paper argues that MCP safety is a distinct, non-trivial research problem rather than a special case of ordinary LLM jailbreaking.

Load-bearing premise

The load-bearing premise is that injecting prompts into MCP service descriptions and returned responses is the representative way a malicious third-party service attacks an agent, and that two simulated scenarios with 20 tasks capture real MCP deployments.

Editorial extensions

If this is right

  • Any production MCP deployment that mixes third-party services with privileged tools should assume that service descriptions and responses can carry prompt injections, and trust boundaries should be designed accordingly.
  • A whitelist or moderation pass that screens services before invocation is not a complete defense, because advanced injected prompts evade both backbone LLM judgment and dedicated moderation models.
  • Active sanitization of service outputs can lower attack success and harm, but only at the cost of degraded task accuracy, so it must be tuned per task rather than applied blindly.
  • MCP safety needs its own red-teaming methods, evaluation benchmarks, safety data, and service safeguards rather than being treated as a subcase of existing LLM jailbreaking research.
  • Future MCP-safe backbone LLMs should be trained or aligned to recognize malicious patterns in untrusted service text, since the implementations of third-party services are not inspectable in advance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the vulnerability generalizes beyond the simulated shopping and writing tasks, cross-service attacks, where one compromised service steers the agent into invoking another service, could make the blast radius larger than the two-surface injection reported here.
  • The paper's active defense is a single paraphrasing pipeline; a testable next step is to measure whether structured extraction that keeps only schema fields from tool outputs outperforms free-form paraphrase on both attack success and accuracy loss.
  • The moderation-based harm metrics may undercount task-dependent harm that is not overtly toxic; a falsifiable extension would rate harm with human judges on domain-specific outcomes such as wrong medical advice or financial loss.
  • The paper frames the attack as prompt injection into text surfaces; another plausible adversary would manipulate returned structured values, such as falsified database rows or tool-call sequences, which would test whether the vulnerability is specifically about natural-language injection or more generally about untrusted tool output.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper is a position paper arguing that the Model Context Protocol (MCP) introduces third-party service providers into LLM agent systems, creating safety risks distinct from the dyadic user-LLM setting. The authors contribute SAFEMCP, a simulation framework that injects prompt-attack content into MCP service descriptions and returned responses, and report pilot experiments on two agent environments (WebShop and TextCraft) with eight backbone LLMs, measuring relative accuracy loss (RAL), attack success rate (ASR), and harm rate (HR). They find all tested models are vulnerable to at least one attack, that simple moderation-based detection (OpenAI moderation API, LLaMA-Guard, or LLM verdicts) misses several advanced attacks, and that an active defense (LLM-based sanitization) reduces ASR/HR but also degrades task accuracy. The paper concludes with a six-part research roadmap for MCP safety.

Significance. If the empirical claims hold, the paper makes a timely and useful call to action: any MCP deployment that trusts third-party service text should assume prompt-injection risk through service descriptions and returned responses, and simple content filtering cannot be relied upon as a complete defense. The release of SAFEMCP as open-source code is a concrete contribution that should enable follow-up evaluations. However, the paper's own evidence is a limited pilot (20 tasks, two environments, native unaligned backbones), and the metric definitions and the lack of a non-MCP control condition need to be addressed before the stronger MCP-attribution claims are accepted. The conceptual distinction the paper draws—third-party actors with economic incentives, distinct from user-driven jailbreaks—is important even if the underlying attack mechanism is prompt injection.

major comments (4)
  1. [3.1 / Table 1] The central attribution to MCP is not yet supported. The experiments inject malicious text into MCP service descriptions/responses but do not include a control condition in which the same attack strings appear in a user message or in a non-MCP function-calling tool. Under this design, the observed vulnerability is equally consistent with general LLM context-injection susceptibility, the position the authors themselves address in Section 5.1. Since the title, Advocation 1, and the roadmap are all framed around MCP-specific third-party risk, the paper should either add such a control and compare attack effectiveness, or explicitly limit Advocation 1 to 'systems in which untrusted MCP service text reaches the model context.'
  2. [2 / 3.1] RAL is defined as 'the fraction of the accuracy on the agent benchmark before and after the attack' but the text then uses it inconsistently. In Section 3.1, a direct attack is said to 'cause 0.25 RAL' on G-o3-mini, while for stronger models 'the RAL is still 0.63, 0.85, and 0.53, respectively. This indicates that the attack successfully obstructs the agent's original intention.' If RAL is retention (after/before), a value of 0.85 indicates a mild attack, not an obstruction; if RAL is loss, the values above 0.5 imply a large loss and the sentence should be rephrased. The same ambiguity affects Figure 3, where 'RAL decreases after the active defense' is interpreted as a negative outcome, implying RAL is retention. Please define RAL precisely, report clean baseline accuracy, and correct the interpretive statements in Section 3.1.
  3. [2 / Table 1] ASR and HR are referred to as rates but the reported values are not probabilities. HR values in Table 1 run up to 3.36, which is impossible for a rate in [0,1]. The metrics section says ASR and HR are 'computed based on LLaMA-Guard and OpenAI-moderation API' but does not specify how a trial is scored, how multiple harmful outputs are counted, or how the two detectors are reconciled. Without this information, the magnitudes in Table 1 and Figures 2(b) cannot be interpreted, and the conclusion that attacks increase harm is not quantitatively grounded.
  4. [3.1 / Appendix B] The evidence base for Advocation 1 is narrow: 20 tasks from only WebShop and TextCraft, and the authors state that DeepSeek-R1 and LLaMA were excluded for function-calling reasons and that only native capabilities (no alignment or MCP-specific fine-tuning) were evaluated. These limitations appear in Appendix B, but the abstract and Advocation 1 state the vulnerability as a general property of 'MCP-powered agent systems.' Either provide additional scenarios/models, or qualify the claim to the tested conditions.
minor comments (5)
  1. [Appendix C, Tables 2 and 3] The tables are titled 'Transposed evaluation results of SOPIA on MCP-powered agent systems,' but the framework introduced in the paper is SAFEMCP; 'SOPIA' appears nowhere else and appears to be a typographical remnant.
  2. [3.3] The text says 'We first report the harmness related metrics, ASR and HR, before and after the active defense in Figure 2(a),' but Figure 2(a) shows detection ratio; the ASR/HR results are in Figure 2(b).
  3. [Throughout] Multiple typos and stylistic issues need correction, including 'emergenece,' 'delibratively,' 'succeptible,' 'persue,' 'futher,' 'harmness,' 'santized,' and the doubled 'without without' in Appendix B.
  4. [3.2] The paper reports that direct attacks are detected with 100% detection ratio yet Table 1 shows nonzero ASR/HR for direct attacks; a sentence clarifying that detection is performed as a separate judgment task rather than within the agent's execution would prevent an apparent contradiction.
  5. [References] References [28] and [29] are the same paper (Hou et al., arXiv:2503.23278) and should be merged.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims rest on direct empirical measurements, not on fitted parameters, self-cited theorems, or definitions that contain their own conclusions.

full rationale

The paper's argument chain is empirical rather than definitional: Section 2 constructs SAFEMCP as a testbed, Section 3.1 applies six pre-existing attack methods (Direct, AutoDAN, CodeChameleon, DeepInception, CipherChat, ReNeLLM) to eight backbone LLMs across 20 WebShop/TextCraft tasks, and Table 1 reports measured RAL, ASR, and HR values. No parameter is fitted to a subset of the target outcome and then renamed a prediction; no equation defines the conclusion into the inputs; and no uniqueness theorem or author-derived prior result is invoked to force the position. The few co-authored citations in the reference list appear in related-work surveys of agentic systems and are not load-bearing for the central advocations. Appendix B's explicit limitations (native capabilities only, baseline defenses only) and the absence of a non-MCP control are external-validity and attribution concerns, not circular reductions. Accordingly, the paper is self-contained as an exploratory empirical demonstration, and there is no significant circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper introduces SAFEMCP, a software framework, not a postulated physical or conceptual entity with independent falsifiable handles. No new particles, forces, mediators, or dimensions are invented, and no free parameters are fitted to data.

assumptions (6)
  • domain assumption MCP is becoming the de facto standard interface for LLM-agent interaction and is supported by frontier LLMs.
    Invoked in the Introduction to motivate the threat model; cited to an Anthropic blog rather than demonstrated empirically.
  • domain assumption Third-party MCP service providers have economic incentives to exploit vulnerabilities and sabotage user-agent interactions.
    Core threat model stated in the Introduction; no real-world evidence of malicious MCP services is provided.
  • domain assumption Injecting prompts into service descriptions and returned responses is a viable and representative attack vector for third-party MCP services.
    SAFEMCP is built on this vector in Section 2; the pilot experiments test it, but there is no validation against real MCP services.
  • domain assumption Attack success rate and harm rate computed via OpenAI-moderation API and LLaMA-Guard are valid proxies for harm.
    Metrics are defined in Section 2; moderation classifiers are known to be imperfect, and the paper itself notes their limitations for detection.
  • domain assumption Results on 20 tasks from WebShop and TextCraft generalize to other MCP workloads.
    Main experiments use two environments in Section 3.1; the Appendix only splits results by injection position and does not add new scenarios.
  • domain assumption The baseline models' native function-calling behavior in simulated sandboxes reflects deployed MCP agent behavior.
    The simulation setup is described in Section 2; no real MCP deployment was tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems." pith.science (2026). https://pith.science/paper/B5VW62PT

@misc{pith2026250613666,
  author       = {Pith},
  title        = {Pith review of: We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5VW62PT}},
  note         = {Machine review of arXiv:2506.13666}
}
read the original abstract

The development of large language models (LLMs) has entered in a experience-driven era, flagged by the emergence of environment feedback-driven learning via reinforcement learning and tool-using agents. This encourages the emergenece of model context protocol (MCP), which defines the standard on how should a LLM interact with external services, such as \api and data. However, as MCP becomes the de facto standard for LLM agent systems, it also introduces new safety risks. In particular, MCP introduces third-party services, which are not controlled by the LLM developers, into the agent systems. These third-party MCP services provider are potentially malicious and have the economic incentives to exploit vulnerabilities and sabotage user-agent interactions. In this position paper, we advocate the research community in LLM safety to pay close attention to the new safety risks issues introduced by MCP, and develop new techniques to build safe MCP-powered agent systems. To establish our position, we argue with three key parts. (1) We first construct \framework, a controlled framework to examine safety issues in MCP-powered agent systems. (2) We then conduct a series of pilot experiments to demonstrate the safety risks in MCP-powered agent systems is a real threat and its defense is not trivial. (3) Finally, we give our outlook by showing a roadmap to build safe MCP-powered agent systems. In particular, we would call for researchers to persue the following research directions: red teaming, MCP safe LLM development, MCP safety evaluation, MCP safety data accumulation, MCP service safeguard, and MCP safe ecosystem construction. We hope this position paper can raise the awareness of the research community in MCP safety and encourage more researchers to join this important research direction. Our code is available at https://github.com/littlelittlenine/SafeMCP.git.

Figures

Figures reproduced from arXiv: 2506.13666 by the authors.

Figure 1
Figure 1. The overall framework of a MCP safe agent system, including: (1) Upper Left: The differences between MCP introduced safety risks and traditional LLM safety risks. (2) Right: The overall architecture of SAFEMCP. (3) Bottom Left: Outlook for MCP safety. However, MCP also introduces new safety attack challenges that are different from pre-MCP era, where attack happens in the dyadic user-LLM interactions, as shown in th… view at source ↗
Figure 2
Figure 2. We show the safety performance with different defense strategies. (a) We show the detection [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. We show the relative accuracy loss (RAL) before and after the defense. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Combating Data Laundering in LLM Training

    cs.CR 2026-04 conditional novelty 6.0 of 10

    Data laundering collapses original-query memorization detectors; SDR recovers useful detection signals by synthesizing training-like rewrites of proprietary data via a goal-details search.

  2. SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    An automated red-teaming framework and 300-case benchmark show that a single unreliable website can induce unsafe responses in LLM search agents, with attack success rates up to 90.5%.

  3. A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis

    cs.CR 2025-06 conditional novelty 6.0 of 10

    The paper releases MCPCorpus, a large-scale annotated dataset of MCP servers and clients with over 20 normalized attributes, plus tooling for updates and exploration.

  4. BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning

    cs.LG 2026-01 conditional novelty 5.0 of 10

    BalDRO makes LLM unlearning more balanced by updating against a worst-case-weighted forget distribution, improving forget quality on TOFU/MUSE at stable utility.

  5. Quantifying Conversation Drift in MCP via Latent Polytope

    cs.CL 2025-08 reject novelty 4.0 of 10

    SecMCP flags MCP conversation drift by thresholding per-layer activation distances from benign anchors, reporting AUROC above 0.915 on Llama3, Vicuna, and Mistral.

Reference graph

Works this paper leans on

107 extracted references · 15 canonical work pages · cited by 5 Pith papers

  1. [1]

    Is my data in your retrieval database? membership inference attacks against retrieval augmented generation.arXiv preprint arXiv:2405.20446, 2024

    Maya Anderson, Guy Amit, and Abigail Goldsteen. Is my data in your retrieval database? membership inference attacks against retrieval augmented generation.arXiv preprint arXiv:2405.20446, 2024

    arXiv 2024

  2. [2]

    Introducing the model context protocol.https://www.anthropic.com/news/ model-context-protocol, November 2024

    Anthropic. Introducing the model context protocol.https://www.anthropic.com/news/ model-context-protocol, November 2024. Blog post

    2024

  3. [3]

    Foundational challenges in assuring alignment and safety of large language models.arXiv preprint arXiv:2404.09932, 2024

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models.arXiv preprint arXiv:2404.09932, 2024

    arXiv 2024

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [5]

    Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=gT5hALch9z

    2024

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    2020

  7. [7]

    Highlights from lex fridman’s interview of yann lecun, March

    Joel Burget. Highlights from lex fridman’s interview of yann lecun, March

  8. [8]

    A survey on evaluation of large language models.ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology, 15(3):1–45, 2024

    2024

Show all 107 references
  1. [9]

    Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large la...

    2024

  2. [10]

    Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2025

    Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases.Advances in Neural Information Processing Systems, 37:130185–130213, 2025

    2025

  3. [11]

    Safety-aware fine-tuning of large language models

    Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-aware fine-tuning of large language models. InNeurips Safe Generative AI Workshop 2024, 2024. URL https://openreview. net/forum?id=SqL94fLSM7

    2024

  4. [12]

    Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 10

    2024

  5. [13]

    Textworld: A learning environment for text-based games

    Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. InWorkshop on Computer Games, pages 41–75. Springer, 2018

    2018

  6. [14]

    DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, and Others. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501...

    2025 arXiv

  7. [15]

    Ai agents under threat: A survey of key security challenges and future pathways

    Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. Ai agents under threat: A survey of key security challenges and future pathways. ACM Computing Surveys, 57(7):1–36, 2025

    2025

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  9. [17]

    A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily.arXiv preprint arXiv:2311.08268, 2023

    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily.arXiv preprint arXiv:2311.08268, 2023

    2023 arXiv

  10. [18]

    Abul Ehtesham, Aditi Singh, Gaurav Kumar Gupta, and Saket Kumar. A survey of agent interoperability protocols: Model context protocol (mcp), agent communication protocol (acp), agent-to-agent protocol (a2a), and agent network protocol (anp).arXiv preprint arXiv:2505.02279, 2025

    2025 arXiv

  11. [19]

    Pawan Kumar, and Adel Bibi

    Francisco Eiras, Aleksandar Petrov, Philip Torr, M. Pawan Kumar, and Adel Bibi. Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/f...

    2025

  12. [20]

    Imprompter: Tricking llm agents into improper tool use.arXiv preprint arXiv:2410.14923, 2024

    Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K Gupta, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Imprompter: Tricking llm agents into improper tool use.arXiv preprint arXiv:2410.14923, 2024

    2024 arXiv

  13. [21]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  14. [22]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    2023 arXiv

  15. [23]

    Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680, 2024

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges.arXiv preprint arXiv:2402.01680, 2024

    2024 arXiv

  16. [24]

    Regulating chatgpt and other large generative ai models

    Philipp Hacker, Andreas Engel, and Marco Mauer. Regulating chatgpt and other large generative ai models. InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, 2023

    2023

  17. [25]

    A survey on large language models: Applications, challenges, limitations, and practical usage.Authorea Preprints, 2023

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage.Authorea Preprints, 2023

    2023

  18. [26]

    What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099, 2024

    Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety.arXiv preprint arXiv:2404.01099, 2024

    2024 arXiv

  19. [27]

    Red-teaming llm multi-agent systems via communication attacks.arXiv preprint arXiv:2502.14847, 2025

    Pengfei He, Yupin Lin, Shen Dong, Han Xu, Yue Xing, and Hui Liu. Red-teaming llm multi-agent systems via communication attacks.arXiv preprint arXiv:2502.14847, 2025. 11

    2025 arXiv

  20. [29]

    Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278, 2025

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278, 2025

    2025 arXiv

  21. [30]

    Advancing language model reasoning through reinforcement learning and inference scaling.arXiv preprint arXiv:2501.11651, 2025

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling.arXiv preprint arXiv:2501.11651, 2025

    2025 arXiv

  22. [31]

    Effects of scale on language model robustness.arXiv preprint arXiv:2407.18213, 2024

    Nikolaus Howe, Ian McKenzie, Oskar Hollinsworth, Michał Zajac, Tom Tseng, Aaron Tucker, Pierre-Luc Bacon, and Adam Gleave. Effects of scale on language model robustness.arXiv preprint arXiv:2407.18213, 2024

    2024 arXiv

  23. [32]

    A survey of safety and trustworthiness of large language models through the lens of verification and validation.Artificial Intelligence Review, 57(7):175, 2024

    Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, et al. A survey of safety and trustworthiness of large language models through the lens of verification and validation.Artificial Intelligence Review, 57(...

    2024

  24. [33]

    Babyai 1.1, 2020

    David Yu-Tung Hui, Maxime Chevalier-Boisvert, Dzmitry Bahdanau, and Yoshua Bengio. Babyai 1.1, 2020

    2020

  25. [34]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.CoRR, abs/2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations.CoRR, abs/2312.06674, 2023

    2023 arXiv

  26. [35]

    Mcp security notification: Tool poisoning attacks

    Lab Invariant. Mcp security notification: Tool poisoning attacks. https://invariantlabs. ai/blog/mcp-security-notification-tool-poisoning-attacks , April 2025. Blog post

    2025

  27. [36]

    Safechain: Safety of language models with long chain-of-thought reasoning capabilities.CoRR, abs/2502.12025, 2025

    Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities.CoRR, abs/2502.12025, 2025

    2025 arXiv

  28. [37]

    Flooding spread of manipulated knowl- edge in llm-based multi-agent communities.arXiv preprint arXiv:2407.07791, 2024

    Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongshen Liu. Flooding spread of manipulated knowl- edge in llm-based multi-agent communities.arXiv preprint arXiv:2407.07791, 2024

    2024 arXiv

  29. [38]

    Llm-mod: Can large language models assist content moderation? InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems

    Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. Llm-mod: Can large language models assist content moderation? InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2024

    2024

  30. [39]

    Watch your language: Investigating content moderation with large language models.Proceedings of the International AAAI Conference on Web and Social Media, 2024

    Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. Watch your language: Investigating content moderation with large language models.Proceedings of the International AAAI Conference on Web and Social Media, 2024

    2024

  31. [40]

    Mcp guardian: A security-first layer for safeguarding mcp-based ai system.arXiv preprint arXiv:2504.12757, 2025

    Sonu Kumar, Anubhav Girdhar, Ritesh Patil, and Divyansh Tripathi. Mcp guardian: A security-first layer for safeguarding mcp-based ai system.arXiv preprint arXiv:2504.12757, 2025

    2025 arXiv

  32. [41]

    How not to be stupid about ai, with yann lecun

    Steven Levy. How not to be stupid about ai, with yann lecun. https://www.wired.com/ story/artificial-intelligence-meta-yann-lecun-interview , December 2023. Interview with Yann LeCun

    2023

  33. [42]

    Commercial llm agents are already vulnerable to simple yet dangerous attacks.arXiv preprint arXiv:2502.08586, 2025

    Ang Li, Yin Zhou, Vethavikashini Chithrra Raghuram, Tom Goldstein, and Micah Goldblum. Commercial llm agents are already vulnerable to simple yet dangerous attacks.arXiv preprint arXiv:2502.08586, 2025

    2025 arXiv

  34. [43]

    Common 7b language models already possess strong math capabilities

    Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024. 12

    2024 arXiv

  35. [44]

    Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  36. [45]

    Deepin- ception: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191, 2023

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepin- ception: Hypnotize large language model to be jailbreaker.arXiv preprint arXiv:2311.03191, 2023

    2023 arXiv

  37. [46]

    The unlocking spell on base llms: Rethinking alignment via in-context learning.arXiv preprint arXiv:2312.01552, 2023

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning.arXiv preprint arXiv:2312.01552, 2023

    2023 arXiv

  38. [47]

    Understanding and enhancing the transferability of jailbreaking attacks

    Runqi Lin, Bo Han, Fengwang Li, and Tongling Liu. Understanding and enhancing the transferability of jailbreaking attacks. InICLR, 2025

    2025

  39. [48]

    Toolace: Winning the points of llm function calling.arXiv preprint arXiv:2409.00920, 2024

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. Toolace: Winning the points of llm function calling.arXiv preprint arXiv:2409.00920, 2024

    2024 arXiv

  40. [49]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. InICLR, 2024

    2024

  41. [50]

    Jailbreaking chatgpt via prompt engineering: An empirical study.arXiv preprint arXiv:2305.13860, 2023

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study.arXiv preprint arXiv:2305.13860, 2023

    2023 arXiv

  42. [51]

    Robustft: Robust supervised fine-tuning for large language models under noisy response, 2024

    Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. Robustft: Robust supervised fine-tuning for large language models under noisy response, 2024. URL https://arxiv.org/abs/2412.14922

    2024 arXiv

  43. [52]

    Codechameleon: Personalized encryption framework for jailbreaking large language models.arXiv preprint arXiv:2402.16717, 2024

    Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. Codechameleon: Personalized encryption framework for jailbreaking large language models.arXiv preprint arXiv:2402.16717, 2024

    2024 arXiv

  44. [53]

    Agentboard: An analytical evaluation board of multi-turn llm agents, 2024

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents, 2024

    2024

  45. [54]

    Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206, 2025

    Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206, 2025

    2025 arXiv

  46. [55]

    Agentsafe: Safeguarding large language model-based multi- agent systems via hierarchical data management.arXiv preprint arXiv:2503.04392, 2025

    Junyuan Mao, Fanci Meng, Yifan Duan, Miao Yu, Xiaojun Jia, Junfeng Fang, Yuxuan Liang, Kun Wang, and Qingsong Wen. Agentsafe: Safeguarding large language model-based multi- agent systems via hierarchical data management.arXiv preprint arXiv:2503.04392, 2025

    2025 arXiv

  47. [56]

    Jishnu Mukhoti, Yarin Gal, Philip Torr, and Puneet K. Dokania. Fine-tuning can crip- ple foundation models; preserving features may be the solution, 2024. URL https: //openreview.net/forum?id=VQ7Q6qdp0P

    2024

  48. [57]

    A comprehensive overview of large language models.arXiv preprint arXiv:2307.06435, 2023

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models.arXiv preprint arXiv:2307.06435, 2023

    2023 arXiv

  49. [58]

    GPT-4 technical report.CoRR, 2023

    OpenAI. GPT-4 technical report.CoRR, 2023

    2023

  50. [59]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  51. [60]

    Self-alignment of large language models via monopolylogue-based social scene sim- ulation

    Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. Self-alignment of large language models via monopolylogue-based social scene sim- ulation. InProceedings of the 41st International Conference on Machine Learning, pages 39416–39447, 2024. 13

    2024

  52. [61]

    A survey on agent-based modelling assisted by machine learning.Expert Systems, 42(1):e13325, 2025

    Alejandro Platas-López, Alejandro Guerra-Hernández, Marcela Quiroz-Castellanos, and Nican- dro Cruz-Ramirez. A survey on agent-based modelling assisted by machine learning.Expert Systems, 42(1):e13325, 2025

    2025

  53. [62]

    Adapt: As-needed decomposition and planning with language models.arXiv preprint arXiv:2311.05772, 2023

    Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models.arXiv preprint arXiv:2311.05772, 2023

    2023 arXiv

  54. [63]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  55. [64]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openr...

    2025

  56. [65]

    Mcp safety audit: Llms with the model context protocol allow major security exploits.arXiv preprint arXiv:2504.03767, 2025

    Brandon Radosevich and John Halloran. Mcp safety audit: Llms with the model context protocol allow major security exploits.arXiv preprint arXiv:2504.03767, 2025

    2025 arXiv

  57. [66]

    Tptu: Task planning and tool usage of large language model-based ai agents

    Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. Tptu: Task planning and tool usage of large language model-based ai agents. InNeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  58. [67]

    Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023

    2023

  59. [69]

    Large language model safety: A holistic survey

    Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, et al. Large language model safety: A holistic survey. arXiv preprint arXiv:2412.17686, 2024

    2024 arXiv

  60. [70]

    Welcome to the era of experience.Google AI, 2025

    David Silver and Richard S Sutton. Welcome to the era of experience.Google AI, 2025

    2025

  61. [71]

    Large language model (chatgpt) as a support tool for breast tumor board.NPJ Breast Cancer, 9(1):44, 2023

    Vera Sorin, Eyal Klang, Miri Sklair-Levy, Israel Cohen, Douglas B Zippel, Nora Balint Lahat, Eli Konen, and Yiftach Barash. Large language model (chatgpt) as a support tool for breast tumor board.NPJ Breast Cancer, 9(1):44, 2023

    2023

  62. [72]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023

    2023 arXiv

  63. [73]

    A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    2025 arXiv

  64. [74]

    Multi-agent collaboration mechanisms: A survey of llms.arXiv preprint arXiv:2501.06322, 2025

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms.arXiv preprint arXiv:2501.06322, 2025

    2025 arXiv

  65. [75]

    House Committee on Oversight and Accountability

    U.S. House Committee on Oversight and Accountability. Hearing wrap-up: Artificial intelligence poses great risks, but safe integration will yield positive results. https://oversight.house.gov/release/ hearing-wrap-up-artificial-intelligence-poses-great-risks-but-safe-integrati...

    2023

  66. [76]

    From allies to adversaries: Manipulating llm tool-calling through adversarial injection.arXiv preprint arXiv:2412.10198, 2024

    Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, and Qing Wang. From allies to adversaries: Manipulating llm tool-calling through adversarial injection.arXiv preprint arXiv:2412.10198, 2024

    2024 arXiv

  67. [77]

    Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment

    Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. InThe Thirty-eighth Annual Conference on Neural...

    2024

  68. [78]

    A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585, 2025

    Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585, 2025

    2025 arXiv

  69. [79]

    Science- world: Is your agent smarter than a 5th grader?, 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Science- world: Is your agent smarter than a 5th grader?, 2022. URL https://arxiv.org/abs/ 2203.07540

    2022 arXiv

  70. [80]

    G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems.arXiv preprint arXiv:2502.11127, 2025

    Shilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan, Fanci Meng, Chongye Guo, Kun Wang, and Yang Wang. G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems.arXiv preprint arXiv:2502.11127, 2025

    2025 arXiv

  71. [81]

    Augmenting language models with long-term memory.Advances in Neural Information Processing Systems, 36:74530–74543, 2023

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory.Advances in Neural Information Processing Systems, 36:74530–74543, 2023

    2023

  72. [82]

    Agentgym: Evolving large language model-based agents across diverse environments.arXiv preprint arXiv:2406.04151, 2024

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments.arXiv preprint arXiv:2406.04151, 2024

    2024 arXiv

  73. [83]

    The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    2025

  74. [84]

    Certifiably robust rag against retrieval corruption.arXiv preprint arXiv:2405.15556, 2024

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption.arXiv preprint arXiv:2405.15556, 2024

    2024

  75. [85]

    Bag of tricks: Benchmarking of jailbreak attacks on llms

    Zhao Xu, Fan Liu, and Hao Liu. Bag of tricks: Benchmarking of jailbreak attacks on llms. In NeurIPS, 2024

    2024

  76. [86]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  77. [87]

    Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36:71995–72007, 2023

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36:71995–72007, 2023

    2023

  78. [88]

    The second half

    Shunyu Yao. The second half. https://ysymyth.github.io/The-Second-Half/ , 2025

    2025

  79. [89]

    Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  80. [90]

    On the vulnerability of safety alignment in open-access llms

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access llms. InFindings of the Association for Computational Linguistics ACL 2024, pages 9236–9260, 2024

    2024

  81. [91]

    Netsafe: Exploring the topological safety of multi-agent networks.arXiv preprint arXiv:2410.15686, 2024

    Miao Yu, Shilong Wang, Guibin Zhang, Junyuan Mao, Chenlong Yin, Qijiong Liu, Qingsong Wen, Kun Wang, and Yang Wang. Netsafe: Exploring the topological safety of multi-agent networks.arXiv preprint arXiv:2410.15686, 2024

    2024 arXiv

  82. [92]

    A survey on trustworthy llm agents: Threats and countermeasures.arXiv preprint arXiv:2503.09648, 2025

    Miao Yu, Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pang, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, et al. A survey on trustworthy llm agents: Threats and countermeasures.arXiv preprint arXiv:2503.09648, 2025. 15

    2025 arXiv

  83. [93]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher.arXiv preprint arXiv:2308.06463, 2023

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher.arXiv preprint arXiv:2308.06463, 2023

    2023 arXiv

  84. [94]

    The good and the bad: Exploring privacy issues in retrieval- augmented generation (rag).arXiv preprint arXiv:2402.16893, 2024

    Shenglai Zeng, Jiankun Zhang, Pengfei He, Yue Xing, Yiding Liu, Han Xu, Jie Ren, Shuaiqiang Wang, Dawei Yin, Yi Chang, et al. The good and the bad: Exploring privacy issues in retrieval- augmented generation (rag).arXiv preprint arXiv:2402.16893, 2024

    2024 arXiv

  85. [95]

    Breaking agents: Compromising autonomous llm agents through malfunction amplification.arXiv preprint arXiv:2407.20859, 2024

    Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. Breaking agents: Compromising autonomous llm agents through malfunction amplification.arXiv preprint arXiv:2407.20859, 2024

    2024 arXiv

  86. [96]

    Cut the crap: An economical communication pipeline for llm-based multi-agent systems.arXiv preprint arXiv:2410.02506, 2024

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems.arXiv preprint arXiv:2410.02506, 2024

    arXiv

  87. [97]

    G-designer: Architecting multi-agent communication topologies via graph neural networks.arXiv preprint arXiv:2410.11782, 2024

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks.arXiv preprint arXiv:2410.11782, 2024

    arXiv

  88. [98]

    Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet.arXiv preprint arXiv:2502.04180, 2025

    2025 arXiv

  89. [99]

    On large language models safety, security, and privacy: A survey.Journal of Electronic Science and Technology, page 100301, 2025

    Ran Zhang, Hong-Wei Li, Xin-Yuan Qian, Wen-Bo Jiang, and Han-Xiao Chen. On large language models safety, security, and privacy: A survey.Journal of Electronic Science and Technology, page 100301, 2025

    2025

  90. [100]

    A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    2024 arXiv

  91. [101]

    Agent-safetybench: Evaluating the safety of llm agents.arXiv preprint arXiv:2412.14470, 2024

    Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-safetybench: Evaluating the safety of llm agents.arXiv preprint arXiv:2412.14470, 2024

    2024 arXiv

  92. [102]

    Weak-to-strong jailbreaking on large language models

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. InICML 2024 Next Generation of AI Safety Workshop, 2024

    2024

  93. [103]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024

    2024

  94. [104]

    Poisoning retrieval corpora by injecting adversarial passages.arXiv preprint arXiv:2310.19156, 2023

    Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. Poisoning retrieval corpora by injecting adversarial passages.arXiv preprint arXiv:2310.19156, 2023

    2023 arXiv

  95. [105]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  96. [106]

    A survey of multi-agent deep reinforcement learning with communication.Autonomous Agents and Multi-Agent Systems, 38(1):4, 2024

    Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent deep reinforcement learning with communication.Autonomous Agents and Multi-Agent Systems, 38(1):4, 2024

    2024

  97. [107]

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. InForty-first International Conference on Machine Learning, 2024. URL https://openreview.net/ forum?id=bWZKvF0g7G

    2024

  98. [108]

    Poisonedrag: Knowledge cor- ruption attacks to retrieval-augmented generation of large language models.arXiv preprint arXiv:2402.07867, 2024

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge cor- ruption attacks to retrieval-augmented generation of large language models.arXiv preprint arXiv:2402.07867, 2024. 16 A Related Work Large Language Model Safety.While demonstrating remarkable reas...

    2024 arXiv

  99. [2024]

    Accessed: 2025-05- 19

    URL https://www.lesswrong.com/posts/bce63kvsAMcwxPipX/ highlights-from-lex-fridman-s-interview-of-yann-lecun . Accessed: 2025-05- 19

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.