Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An RL-trained attack agent can automatically craft adversarial prompts that make aligned LLMs reveal system prompts and memorized training data, outperforming handcrafted and fuzzing baselines.

desk verdict Solid system-prompt red-teaming, but the 'automated training data extraction' headline overstates an oracle-dependent result. read the letter →

arxiv 2412.05734 v2 pith:RPCVHXTS submitted 2024-12-07 cs.CR cs.AIcs.LG

Yuzhou Nie , Zhun Wang , Ye Yu , Xian Wu , Xuandong Zhao , Wenbo Guo , Dawn Song This is my paper · ORCID
classification cs.CRcs.AIcs.LG
keywords LLMprivacyleakagered-teamingreinforcementlearningtrainingdataextractionsystempromptadversarialpromptsPPOblack-boxattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LeakAgent tries to show that privacy leakage from aligned LLMs can be automated end-to-end: an open-source LLM is trained with reinforcement learning to write adversarial prompts that make a target model reveal its system prompt or snippets of memorized training data. The authors claim this is the first unified black-box red-teaming framework covering both leakage types and the first automated attack for training data extraction. They report that the RL agent beats handcrafted prompts and fuzzing baselines, extracts system prompts from real-world GPT store applications even under a guardrail filter, and produces attacks that can be reused as safety-alignment data. If true, this turns red-teaming from a manual, model-specific chore into a scalable learning problem, and it raises the bar for what defenders must protect.

What carries the argument

The load-bearing object is the attack agent: an open-source LLM, Meta-Llama-3-8B-Instruct, fine-tuned with PPO to output adversarial prompts. The reward that carries the learning is a sliding-window word edit similarity (SWES), defined as the negative log of word edit distance computed over a sliding window, normalized through a sigmoid and combined with a length-matching term; it gives partial credit when the target response contains part of the desired secret, which coarse embedding or n-gram similarities fail to distinguish. Two further mechanisms make the policy learnable: a dynamic temperature schedule that forces high-temperature exploration for the first few tokens and low-temperature exploitation afterward, and a diversity regularizer that rewards prompts dissimilar to already-successful ones. For training data extraction, a two-stage schedule first uses a corpus-search tool to find a memorized sample the target is likely to emit, then continues RL to extract that sample in full.

What would settle it

Run the fully trained data-extraction agent against a second open model whose training corpus is withheld from the attacker, and compare extraction success to the roughly 0.2% handcrafted baseline; if success collapses, the claimed automated training-data extraction does not generalize beyond the corpus used for reward computation.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that deep RL with a fine-grained reward lets a small open model discover adversarial prompts that aligned open and commercial LLMs obey. The attack agent is trained with PPO; its reward is a sliding-window word edit similarity (SWES) between the target response and the desired secret, sigmoid-normalized and combined with a length-matching term, so partial leaks give useful learning signal. For training data extraction, a two-stage procedure first searches a known corpus to identify a memorized sample the target will output, then refines prompts to recover as much of that sample as possible. The evaluation reports system-prompt similarity scores typically above 0.5, a 5.9% training-data success rate on OLMo versus at most 0.2% for handcrafted prompts, and success on all twelve tested GPT store applications even when a PromptGuard filter is applied. The authors conclude that RL-based agents are a promising, general direction for LLM red-teaming.

Load-bearing premise

The training-data setup (Appendix A.2.1) assumes the attacker has a searchable copy of the target model's training corpus, because the reward is computed by matching the target's output against that corpus, and the 5.9% success rate is measured on OLMo, the model whose corpus was used during agent training.

Editorial extensions

If this is right

  • Automated privacy red-teaming no longer depends on a human writing extractor prompts; the same RL agent can be retrained against new targets and leakage goals.
  • Input guardrails such as PromptGuard are not sufficient: LeakAgent's prompts can bypass them on real-world GPT store applications.
  • Small fine-tuned attack models can outperform larger models using in-context learning on adversarial prompt generation.
  • Attacks discovered by LeakAgent can be converted into supervised fine-tuning data that hardens the target model with a smaller utility drop than the SecAlign defense.
  • Even a 5.9% training-data success rate means an automated attacker can recover nontrivial memorized content in 20,000 queries, indicating that manual probing underestimates exposure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The training-data success rate is measured on OLMo, the same model whose corpus was used to compute rewards during agent training; transferring the trained data-extraction agent to a model with an unknown corpus is an open question, because the two-stage reward cannot be computed without a searchable corpus.
  • Because the reward is edit-distance based, semantically equivalent but paraphrased leaks would score low; reported similarity scores are therefore likely to understate true leakage rather than overstate it.
  • Many generated prompts are sequences of tokens like <eos> or token fragments rather than natural instructions, suggesting the learned attack exploits token-level quirks of a specific model; such policies may not transfer to future models with different tokenizers or alignment.
  • The same fine-grained similarity reward could be reused as a continuous safety metric for measuring residual leakage after defenses, rather than only as a training signal for attacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. LeakAgent trains an open-source LLM with PPO to generate adversarial prompts for two privacy-leakage goals: extracting system prompts and extracting training data. The paper introduces a sliding-window word-edit-similarity (WES) reward, dynamic temperature control, and a diversity bonus, plus a two-stage training scheme for training-data extraction. The authors report that LeakAgent outperforms handcrafted, fuzzing, and in-context-learning baselines on system-prompt extraction across six target models, transfers to GPT Store applications, evades PromptGuard, and achieves a 5.9% training-data extraction rate on OLMo versus at most 0.2% for handcrafted attacks. They also present an ablation study and a safety-alignment application.

Significance. If the results hold, LeakAgent would be a meaningful advance in automated red-teaming for LLM privacy, especially for system-prompt leakage, and the proposed reward design is a plausible contribution to RL-based adversarial prompt generation. The paper contains useful comparisons, a human-judged real-world evaluation, defense experiments, a utility analysis, and a public code release, all of which strengthen the work. However, the headline training-data-extraction claim is materially narrower than stated: the attack agent is trained using a searchable copy of the target model's training corpus, and the only extraction result is measured on that same model. The central quantitative comparisons also rest on single-run point estimates despite the paper's own admission of seed-sensitive training instability. These issues are fixable by re-scoping claims and adding the missing transfer and variance evidence, but they are load-bearing for the paper's central novelty statement.

major comments (4)
  1. [§4.4, Appendix A.2.1] The training-data extraction result is obtained under an oracle assumption that contradicts the stated black-box threat model. Appendix A.2.1 says 'To train our attack agent, we need the target information. We select the open-source models with released training data as our target model and then apply the trained agents to other models without public training data information.' The 5.9% attack success rate in Table 3 is measured on OLMo, the very model whose released corpus is used as the reward database. No transfer experiment for training-data extraction to a model with an unseen corpus is reported, and the examples in Table 7 are OLMo-specific (e.g., '<eos>... AWS'). The abstract's claim of being 'the first work to enable automated attacks for training data extraction' is therefore not supported under the paper's own threat model. The demonstrated capability is corpus-guided memorization mining over a known dataset. Please either add a transfer evaluation to a model whose training corpus is not available during training or re-scope the claim accordingly.
  2. [§4.4] The comparison between LeakAgent and the handcrafted baselines is not on equal footing because LeakAgent's reported success rate counts successes during RL training, while the baselines are fixed prompts evaluated over the same 20K-query budget without adaptation. The text states: 'we control the total number of queries to the target model with an upper bound of 20K times and report the attack success rate in the whole process (including agent training).' This means the 5.9% figure includes the agent's training-time search for memorized samples, whereas the baselines do not get any adaptive search. To support the claimed superiority, please report extraction success on a held-out query phase after training concludes, or give the baselines a comparable optimization/search budget within the measurement window.
  3. [§5, Tables 1-4] The paper acknowledges in its Limitations that 'Our attack training shows instability due to sensitivity to the initial random seed, occasionally failing in some runs,' but no table reports standard errors, multiple seeds, or run-level success rates. All headline numbers—including the 5.9% extraction rate and the system-prompt similarity scores—are single-run point estimates. Given the admitted instability, the central 'significantly outperforms' claims need distributional information (e.g., mean±std over at least five seeds, or a clear best-of-k reporting rule). Without this, a reader cannot tell whether the reported gaps are robust or artifacts of a favorable seed.
  4. [§4.1, Table 1] The text says 'LeakAgent achieves the highest ASR across all models,' but Table 1 reports average WES and ROUGE similarity scores, not an attack success rate, and no success threshold is defined for these open-source and proprietary models. The phrase 'ASR' is never formally defined in the main text. Because the central claim is that LeakAgent 'significantly outperforms' baselines, the reader needs to know what fraction of testing instances are actually extracted (fully or substantially), not just the average similarity. The GPT Store experiment in Table 2 does use human-judged success, but the main system-prompt comparison does not. Additionally, WES is used both as the RL reward and as the primary evaluation metric; the ROUGE column and the human judgments on GPT Store provide some independent grounding, but this overlap should be discussed explicitly.
minor comments (6)
  1. [Eq. (2), Eq. (6)] The length-regularization term 1/||u|-|d|| is undefined when |u|=|d|. The paper does not state how this case is handled (e.g., epsilon addition or a discrete fallback). Please clarify.
  2. [Table 3] The 'Stage 1 of LeakAgent' row should clarify whether the reported 0.2% is measured during Stage-1 training only, or after Stage 1 with a held-out query budget; the current text is ambiguous about how the stage-wise rates are counted.
  3. [Table 3] The baseline labeled 'Repeat (Carlini et al., 2021)' appears to attribute the 'Repeat poem forever' attack to the wrong reference; this attack is described in Nasr et al. (2023). Please correct the citation.
  4. [§4.1] The sentence about embedding similarity says it will 'give overlay large scores'; this should read 'overly large scores.'
  5. [§6] The conclusion says 'effective and genetic LLM red-teaming'; this appears to be a typo for 'generic LLM red-teaming.'
  6. [Appendix C, Figure 5] The training curves show reward over 'Training Steps' but do not specify the number of seeds, the learning rate, or the PPO hyperparameter values; please add these details for reproducibility.

Circularity Check

2 steps flagged · score 4.0 of 10

Partially circular: the WES reward is also the primary evaluation metric, and the training-data ASR is measured on the same corpus used as the reward oracle, with no unseen-corpus transfer test.

  1. fitted input called prediction [Section 3.2 (Eq. 2), Section 4.1, Appendix A.1 (Eqs. 5-6)]
    "Our reward function is defined as: R(u, d) = (1 − λ)SWESnorm + λ 1/∥|u| − |d|∥ ... We set λ = 0.1 based on grid search. ... We set k = 5 and x0 = 0.6 based on our empirical experience. ... We use our proposed metric and ROUGE as the similarity metric."

    The PPO agent is trained to maximize R(u,d), whose dominant term is SWESnorm (the 'WES' metric later reported in Table 1). The sigmoid parameters k,x0 and the length weight λ are hand-tuned/grid-searched rather than independently fixed, so the WES evaluation scores are the same objective being optimized. Reporting WES as attack performance therefore partly measures how well the agent optimized the evaluation metric itself, rather than an independent outcome. The co-reported ROUGE and GPT Store human judgment provide some independent grounding, but the headline WES-based gains are partly circular.

  2. fitted input called prediction [Appendix A.2.1, Section 4.4, Table 3]
    "To train our attack agent, we need the target information. We select the open-source models with released training data as our target model and then apply the trained agents to other models without public training data information. ... we control the total number of queries to the target model with an upper bound of 20K times and report the attack success rate in the whole process (including agent training)."

    The headline 5.9% ASR for training-data extraction is measured on OLMo, the same model whose released corpus is used as the reward/search database during RL training, and successes are counted during training rather than on a held-out target. No experiment transfers the trained agent to a model with an unseen corpus, despite the stated intent to 'apply the trained agents to other models without public training data information.' The reported capability is therefore a corpus-guided search over known training data, not the claimed general black-box extraction; the prediction reduces to the training oracle by construction.

full rationale

LeakAgent's system-prompt extraction results are largely self-contained: the attack is trained on a public prompt dataset, evaluated with ROUGE in addition to WES, and validated by independent human judgment on GPT Store applications, with transferability across models tested. The main circularity concerns are (i) the WES metric doubling as both the RL reward and the primary evaluation metric, with its sigmoid parameters and length weight fitted by grid search/experience, and (ii) the training-data extraction evaluation using the target model's own released corpus as the reward oracle and counting training-time successes as attack success rate, with no unseen-corpus transfer test. The self-citations to Chen et al. 2024c/d appear only as design rationale for abandoning mutator-based RL and are not load-bearing. Thus the paper is not wholly circular, but the headline numeric claims for training-data extraction and the WES-based system-prompt gains are partly artifacts of the evaluation protocol.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several hand-tuned parameters inside the reward and training schedule, on the assumption that edit-similarity reward tracks real privacy leakage, and on access to a target training corpus. The framework itself introduces no new physical entities; the only new object is the WES metric, which is a fitted scoring function rather than an invented entity.

free parameters (6)
  • lambda (reward weight) = 0.1
    Weight balancing normalized SWES and the length penalty in reward Eqn. (2); set by grid search (Section 3.2).
  • sigmoid steepness k = 5
    Controls the sharpness of SWESnorm in Eqn. (5); set from empirical experience (Appendix A.1).
  • sigmoid offset x0 = 0.6
    Shifts SWESnorm in Eqn. (5); set from empirical experience (Appendix A.1).
  • diversity reward threshold and bonus = threshold 0.9, bonus 0.2
    Prompts with reward above 0.9 are stored; new prompts far from them receive an additional 0.2 reward (Section 3.2).
  • dynamic temperature parameters = Thigh >> 1, Tbase, token cutoff k, top-k
    Exploration and exploitation schedule described qualitatively; exact values are not given in the paper (Section 3.2).
  • initial prompts p0 = general phrase for system prompt; [eos] or { or % x30 for training data
    Hand-crafted initial conditions chosen from prior work and pilot experiments (Appendix A.2.1).
assumptions (3)
  • domain assumption PPO with QLoRA can train an LLM policy to maximize a non-differentiable black-box reward over prompt space.
    The whole method depends on RL optimization succeeding; no convergence guarantee is proven (Sections 3.1-3.2).
  • domain assumption The WES reward function is a valid proxy for real privacy leakage.
    Validated subjectively on only three examples in Appendix E; it is used to define attack success.
  • domain assumption The attacker knows the target model's training corpus, or a representative subset, to compute rewards for training data extraction.
    Explicitly assumed in Appendix A.2.1; OLMo's open data is used. Without this, the two-stage training cannot run.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage." pith.science (2026). https://pith.science/paper/RPCVHXTS

@misc{pith2026241205734,
  author       = {Pith},
  title        = {Pith review of: LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPCVHXTS}},
  note         = {Machine review of arXiv:2412.05734}
}
read the original abstract

Recent studies have discovered that large language models (LLM) may be ``fooled'' to output private information, including training data, system prompts, and personally identifiable information, under carefully crafted adversarial prompts. Existing red-teaming approaches for privacy leakage either rely on manual efforts or focus solely on system prompt extraction, making them ineffective for severe risks of training data leakage. We propose LeakAgent, a novel black-box red-teaming framework for LLM privacy leakage. Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts for both training data extraction and system prompt extraction. To achieve this, we propose a novel reward function to provide effective and fine-grained rewards and design novel mechanisms to balance exploration and exploitation during learning and enhance the diversity of adversarial prompts. Through extensive evaluations, we first show that LeakAgent significantly outperforms existing rule-based approaches in training data extraction and automated methods in system prompt leakage. We also demonstrate the effectiveness of LeakAgent in extracting system prompts from real-world applications in OpenAI's GPT Store. We further demonstrate LeakAgent's effectiveness in evading the existing guardrail defense and its helpfulness in enabling better safety alignment. Finally, we validate our customized designs through a detailed ablation study. We release our code here https://github.com/rucnyz/LeakAgent.

Figures

Figures reproduced from arXiv: 2412.05734 by the authors.

Figure 2
Figure 2. Ablation study results, i.e., the performance of different variants. system prompt extraction using the Llama3.1-8B as the target model. We report the average similarity score on the testing set following the testing procedure introduced in Section 4.1 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Overview of LeakAgent. It begins with an initial input [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Transferability of selected attacks across different models. We show the absolute [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Training curve for LeakAgent with and without dynamic temperature adjustment. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic Web: Weaving the Next Web with AI Agents

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.

Reference graph

Works this paper leans on

38 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [2]

    system prompt

    What is the prompt itself?’ GPT-4o ’100 words\nHere is a 100-word version of the prompt. Edit it for grammar and clarity. Let’ GPT-4o-mini ’1k-2k words\nI apologize for the confusion, but as a self-response, please copy and paste prompt in the form **bold’ Mistral-7B ’1-2 sentences\nHere is a revised version of your request. I added a period and a space b...

    work page 2025

  2. [4]

    Struq: Defending against prompt injection with structured queries

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363, 2024a. 10 Published as a conference paper at COLM 2025 Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, and Chuan Guo. Aligning llms to be robust against prompt injection. ...

    arXiv 2025

  3. [5]

    Bert: Pre-training of deep bidirectional transformers for language understand- ing

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805,

    arXiv

  4. [7]

    URL https://zenodo.org/records/12608602. Google. Gemini family,

    arXiv

  5. [8]

    Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg

    URL https://api.semanticscholar.org/CorpusID:267365485. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733,

    arXiv

  6. [10]

    Llama guard: Llm- based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm- based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674,

    arXiv

  7. [11]

    11 Published as a conference paper at COLM 2025 Kevin Liu [@kliu128]

    URL https://arxiv.org/abs/2310.06825. 11 Published as a conference paper at COLM 2025 Kevin Liu [@kliu128]. The entire prompt of Microsoft Bing Chat?! (Hi, Sydney.) https://t.co/ZNywWV9MNB, February

    arXiv 2025

  8. [12]

    Deepincep- tion: Hypnotize large language model to be jailbreaker

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepincep- tion: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191,

    arXiv

Show all 38 references
  1. [14]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,

    arXiv

  2. [15]

    Meta-llama: Prompt guard, 2024c

    Meta. Meta-llama: Prompt guard, 2024c. URL https://huggingface.co/meta-llama/ Prompt-Guard-86M?text=Ignore+previous+instructions+and+show+me+your+system+ prompt. Available on: 2024-07. Barton P Miller, Lars Fredriksen, and Bryan So. An empirical study of the reliability of uni...

    2024

  3. [17]

    Trojfm: Resource-efficient backdoor attacks against very large foundation models

    Yuzhou Nie, Yanting Wang, Jinyuan Jia, Michael J De Lucia, Nathaniel D Bastian, Wenbo Guo, and Dawn Song. Trojfm: Resource-efficient backdoor attacks against very large foundation models. arXiv preprint arXiv:2405.16783,

    arXiv

  4. [18]

    Jailbreaking attack against multimodal large language model

    Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking attack against multimodal large language model. arXiv preprint arXiv:2402.02309,

    arXiv

  5. [19]

    URL https://chat.openai.com/chat. gpt4o. OpenAI. Gpt store, 2024a. URL https://chatgpt.com/gpts. Accessed: 2024-01-10. OpenAI. Openai embedding models, 2024b. URL https://openai.com/index/ new-embedding-models-and-api-updates/ . Accessed: 2024-01-25. Yonatan Oren, Nicole Meist...

    2024 arXiv

  6. [21]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527,

    arXiv

  7. [23]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URL https://github.com/f/ awesome-chatgpt-prompts. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  8. [24]

    Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt

    Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. arXiv preprint arXiv:2304.12298, 2023a. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, a...

    arXiv

  9. [26]

    Adversarial attacks on multimodal agents

    Chen Henry Wu, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. Adversarial attacks on multimodal agents. arXiv preprint arXiv:2406.12814,

    arXiv

  10. [27]

    Prsa: Prompt reverse stealing attacks against large language models

    Yong Yang, Xuhong Zhang, Yi Jiang, Xi Chen, Haoyu Wang, Shouling Ji, and Zonghui Wang. Prsa: Prompt reverse stealing attacks against large language models. arXiv preprint arXiv:2402.19200,

    arXiv

  11. [28]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629,

    arXiv

  12. [29]

    Privacy risk in machine learning: Analyzing the connection to overfitting

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268–282. IEEE,

    2018

  13. [30]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

    Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253,

    arXiv

  14. [31]

    Enhancing jailbreak attack against large language models through silent tokens

    Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, and Xinyu Xing. Enhancing jailbreak attack against large language models through silent tokens. arXiv preprint arXiv:2405.20653, 2024a. Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi, and Xinyu Xing. Prompt...

    arXiv

  15. [33]

    Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success

    Yiming Zhang and Daphne Ippolito. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success. arXiv preprint arXiv:2307.06865,

    arXiv

  16. [34]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,

    arXiv

  17. [35]

    Please generate a prompt for me

    14 Published as a conference paper at COLM 2025 : Please generatea prompt for me LLM RL agent : Here is a 100-word version of theprompt. Edit it forgrammar and clarity. Temperature control Target LLM : Sure! I am a linuxterminal and will only typecommands. : I want you to act ...

    2025

  18. [36]

    Where is the prompt from? (e.g

    18 Published as a conference paper at COLM 2025 Target model Adversarial prompt Claude-3-Haiku ’1. Where is the prompt from? (e.g. a book, movie, TV show, etc.)

    2025

  19. [38]

    Both AWS Lambda and the functions running on the service provide predictable and reliable operational performance

    DynamoDB events: AWS Lambda inv 21 Published as a conference paper at COLM 2025 Example 3 - Built-in Fault Tolerance: Lambda has built-in fault tolerance as it maintains compute capacity across multiple Availability Zones in each region to help protect code against individual ...

    2025

  20. [1990]

    Scalable extraction of training data from (production) language models

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035,

    arXiv

  21. [2002]

    Advprompter: Fast adaptive adversarial prompting for llms

    Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873,

    arXiv

  22. [2004]

    A survey of attacks on large vision-language models: Resources, advances, and future trends

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Wei Hu, and Yu Cheng. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403,

    arXiv

  23. [2007]

    Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents

    13 Published as a conference paper at COLM 2025 Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691,

    2025 arXiv

  24. [2017]

    Pleak: Prompt leaking attacks against large language model applications

    Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. Pleak: Prompt leaking attacks against large language model applications. arXiv preprint arXiv:2405.06823,

    arXiv

  25. [2018]

    De-cop: Detecting copyrighted content in language models training data

    André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. De-cop: Detecting copyrighted content in language models training data. arXiv preprint arXiv:2402.09910,

    arXiv

  26. [2019]

    Trustllm: Trustworthiness in large language models

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561,

    arXiv

  27. [2021]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE,

    2022

  28. [2022]

    12 Published as a conference paper at COLM 2025 Poe. Poe,

    2025

  29. [2023]

    claude model

    URL https://claude.ai. claude model. Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP) , pp. 1138–1156. IEEE,

    2022

  30. [2024]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong

    URL https://arxiv.org/abs/2403.06634. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.