{"id":"14d0cbc6-58aa-4850-8136-cde0c7cf2c7f","arxiv_id":"2412.05734","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An RL-trained prompt generator outperforms rule-based and fuzzing baselines at extracting system prompts and training data from black-box LLMs.","lead":"LeakAgent trains an open-source LLM with reinforcement learning to generate prompts that pull hidden system prompts and training data out of target LLMs. It beats handcrafted and fuzzing attacks on several models and real GPT Store apps, and its prompts survive a guardrail filter.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-data extraction claim relies on an oracle training corpus; the 5.9% ASR is evaluated only on OLMo using that corpus, and no transfer experiment shows extraction on a model with an unseen corpus.","rationale":"The reader's weakest assumption identifies exactly the load-bearing dependency I find. The abstract and introduction claim that LeakAgent is the first automated framework for training-data extraction, but the method's reward requires the target's training corpus. For the only training-extraction experiment, OLMo was chosen precisely because its corpus is public, the agent was trained on that corpus, and the reported 5.9% ASR is measured over the entire RL process, including Stage-1 partial-match searches against the corpus database. This confounds 'attack capability' with 'oracle-guided training objective.' The claimed transferability to models without public training data is asserted but not tested for training extraction; the transfer experiments in Appendix B cover system-prompt extraction only. I therefore agree with the reader's conditional verdict: the system-prompt extraction results and ablation study are credible evidence for that task, and the paper's code release and COLM publication lend support, but the headline training-data-extraction claim needs either a transfer result to a corpus-unseen model or a revised scope statement. I would not reject the paper: the concern is about the strength and generality of a central claim, not about an internal contradiction or fabricated result.","tokens_in":20530,"tokens_out":5299,"duration_ms":53363,"concrete_test":"Freeze the LeakAgent checkpoint trained for OLMo training-data extraction, then run the same 20K-query protocol against a second open-weight model whose released pretraining corpus was never used during agent training (e.g., Pythia-1B with The Pile), scoring exact/partial matches against that model's own corpus. If the ASR is comparable to the 5.9% reported on OLMo, the attack transfers to unseen corpora and the concern is resolved; if it collapses to the ~0.1-0.2% handcrafted baseline, the reported result is an artifact of corpus-specific training and the 'first automated training data extraction' claim should be narrowed to corpus-known targets. In the same run, also report ASR excluding RL training-time queries, i.e., only fixed-agent queries after training, to separate oracle-guided search during training from post-training attack capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty claim—'first work to enable automated attacks for training data extraction'—rests on a reward/training oracle that requires a searchable copy of the target's exact training corpus. Appendix A.2.1 says this explicitly: 'To train our attack agent, we need the target information. We select the open-source models with released training data as our target model and then apply the trained agents to other models without public training data information.' No transfer evaluation for training-data extraction is reported: Table 3 is the only training-extraction result, measured on OLMo, the model whose corpus was used to train the agent, and the ASR counts successes during RL training because the query budget includes training. For a target with an unknown corpus, the attacker cannot run Stage-1 corpus search or Stage-2 fine-grained reward; the only demonstrated outputs are OLMo-specific prompts such as '<eos>... AWS' (Table 7). Thus the headline capability is not demonstrated under the paper's own black-box threat model. The reported numbers may be correct, but they support a weaker claim: corpus-guided memorization mining over a known dataset, not automated extraction from an unknown-corpus model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LeakAgent trains an open-source LLM with PPO to generate adversarial prompts for two privacy-leakage goals: extracting system prompts and extracting training data. The paper introduces a sliding-window word-edit-similarity (WES) reward, dynamic temperature control, and a diversity bonus, plus a two-stage training scheme for training-data extraction. The authors report that LeakAgent outperforms handcrafted, fuzzing, and in-context-learning baselines on system-prompt extraction across six target models, transfers to GPT Store applications, evades PromptGuard, and achieves a 5.9% training-data extraction rate on OLMo versus at most 0.2% for handcrafted attacks. They also present an ablation study and a safety-alignment application.","tokens_in":20715,"tokens_out":4800,"duration_ms":47962,"significance":"If the results hold, LeakAgent would be a meaningful advance in automated red-teaming for LLM privacy, especially for system-prompt leakage, and the proposed reward design is a plausible contribution to RL-based adversarial prompt generation. The paper contains useful comparisons, a human-judged real-world evaluation, defense experiments, a utility analysis, and a public code release, all of which strengthen the work. However, the headline training-data-extraction claim is materially narrower than stated: the attack agent is trained using a searchable copy of the target model's training corpus, and the only extraction result is measured on that same model. The central quantitative comparisons also rest on single-run point estimates despite the paper's own admission of seed-sensitive training instability. These issues are fixable by re-scoping claims and adding the missing transfer and variance evidence, but they are load-bearing for the paper's central novelty statement.","major_comments":[{"comment":"The training-data extraction result is obtained under an oracle assumption that contradicts the stated black-box threat model. Appendix A.2.1 says 'To train our attack agent, we need the target information. We select the open-source models with released training data as our target model and then apply the trained agents to other models without public training data information.' The 5.9% attack success rate in Table 3 is measured on OLMo, the very model whose released corpus is used as the reward database. No transfer experiment for training-data extraction to a model with an unseen corpus is reported, and the examples in Table 7 are OLMo-specific (e.g., '<eos>... AWS'). The abstract's claim of being 'the first work to enable automated attacks for training data extraction' is therefore not supported under the paper's own threat model. The demonstrated capability is corpus-guided memorization mining over a known dataset. Please either add a transfer evaluation to a model whose training corpus is not available during training or re-scope the claim accordingly.","section":"§4.4, Appendix A.2.1"},{"comment":"The comparison between LeakAgent and the handcrafted baselines is not on equal footing because LeakAgent's reported success rate counts successes during RL training, while the baselines are fixed prompts evaluated over the same 20K-query budget without adaptation. The text states: 'we control the total number of queries to the target model with an upper bound of 20K times and report the attack success rate in the whole process (including agent training).' This means the 5.9% figure includes the agent's training-time search for memorized samples, whereas the baselines do not get any adaptive search. To support the claimed superiority, please report extraction success on a held-out query phase after training concludes, or give the baselines a comparable optimization/search budget within the measurement window.","section":"§4.4"},{"comment":"The paper acknowledges in its Limitations that 'Our attack training shows instability due to sensitivity to the initial random seed, occasionally failing in some runs,' but no table reports standard errors, multiple seeds, or run-level success rates. All headline numbers—including the 5.9% extraction rate and the system-prompt similarity scores—are single-run point estimates. Given the admitted instability, the central 'significantly outperforms' claims need distributional information (e.g., mean±std over at least five seeds, or a clear best-of-k reporting rule). Without this, a reader cannot tell whether the reported gaps are robust or artifacts of a favorable seed.","section":"§5, Tables 1-4"},{"comment":"The text says 'LeakAgent achieves the highest ASR across all models,' but Table 1 reports average WES and ROUGE similarity scores, not an attack success rate, and no success threshold is defined for these open-source and proprietary models. The phrase 'ASR' is never formally defined in the main text. Because the central claim is that LeakAgent 'significantly outperforms' baselines, the reader needs to know what fraction of testing instances are actually extracted (fully or substantially), not just the average similarity. The GPT Store experiment in Table 2 does use human-judged success, but the main system-prompt comparison does not. Additionally, WES is used both as the RL reward and as the primary evaluation metric; the ROUGE column and the human judgments on GPT Store provide some independent grounding, but this overlap should be discussed explicitly.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"The length-regularization term 1/||u|-|d|| is undefined when |u|=|d|. The paper does not state how this case is handled (e.g., epsilon addition or a discrete fallback). Please clarify.","section":"Eq. (2), Eq. (6)"},{"comment":"The 'Stage 1 of LeakAgent' row should clarify whether the reported 0.2% is measured during Stage-1 training only, or after Stage 1 with a held-out query budget; the current text is ambiguous about how the stage-wise rates are counted.","section":"Table 3"},{"comment":"The baseline labeled 'Repeat (Carlini et al., 2021)' appears to attribute the 'Repeat poem forever' attack to the wrong reference; this attack is described in Nasr et al. (2023). Please correct the citation.","section":"Table 3"},{"comment":"The sentence about embedding similarity says it will 'give overlay large scores'; this should read 'overly large scores.'","section":"§4.1"},{"comment":"The conclusion says 'effective and genetic LLM red-teaming'; this appears to be a typo for 'generic LLM red-teaming.'","section":"§6"},{"comment":"The training curves show reward over 'Training Steps' but do not specify the number of seeds, the learning rate, or the PPO hyperparameter values; please add these details for reproducibility.","section":"Appendix C, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is listed as published at COLM 2025, so this report assesses the arXiv v2 version. The most consequential issue is that the training-data-extraction claim outruns the evidence: the method needs an oracle copy of the target's training corpus and is only evaluated on that same model. This is fixable by re-scoping the contribution and adding a transfer experiment to a model with an unseen corpus, but as written it is a load-bearing overclaim. The lack of seed variance, despite admitted instability, is also a serious reproducibility concern for the quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent RL red-teaming paper whose system-prompt extraction results are solid and useful. The training-data extraction headline is broader than what the experiments actually support, though the paper itself discloses the oracle requirement in Appendix A.2.1.\n\nWhat's new: applying PPO with a dense edit-similarity reward to generate adversarial prompts for privacy leakage, plus two-stage training for extraction and mechanisms for temperature control and diversity. The system prompt extraction evaluation is the strongest part: LeakAgent beats PromptFuzz, PLeak, handcrafted prompts, and ReAct across Llama, Mistral, GPT-4o, and Claude, shows transferability across models, gets human-verified extractions from GPT Store apps, and the defense/alignment experiments are a nice bonus. That is real, useful work with released code.\n\nSoft spots, in rough order of severity. First, the 'first automated training data extraction' claim does not hold under the stated black-box threat model. Training the extraction agent requires a searchable copy of the target's exact training corpus to compute rewards, and the 5.9% success rate is measured on OLMo, the model whose corpus was used during training and reward calculation. No transfer experiment shows extraction on a model with an unseen corpus. The stress-test note is right: the demonstrated capability is corpus-guided memorization mining, not automated extraction from an arbitrary black-box model. Second, there are no error bars or multiple-seed results anywhere, even though the limitations section acknowledges training instability. That is a real gap but not a fatal one. Third, WES is used both as the RL reward and as the headline evaluation metric, and its sigmoid parameters (k=5, x0=0.6, lambda=0.1) are fit on a handful of examples. Independent ROUGE results and human judgment on GPT Store apps partially mitigate this; it's a moderate concern, not a dealbreaker.\n\nThe paper is worth engaging. The system-prompt attack is a genuinely useful tool, and the extraction overclaim is a correction, not a rejection. It deserves serious peer review: I would send it out and ask for transfer experiments on training-data extraction and for variance reporting. I'd cite it for system prompt extraction, with a caveat on the training data claim.","headline":"Solid system-prompt red-teaming, but the 'automated training data extraction' headline overstates an oracle-dependent result.","tokens_in":21344,"tokens_out":2908,"would_cite":true,"duration_ms":28828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An RL-trained attack agent can automatically craft adversarial prompts that make aligned LLMs reveal system prompts and memorized training data, outperforming handcrafted and fuzzing baselines.","keywords":["LLM privacy leakage","red-teaming","reinforcement learning","training data extraction","system prompt leakage","adversarial prompts","PPO","black-box attack"],"falsifier":"Run the fully trained data-extraction agent against a second open model whose training corpus is withheld from the attacker, and compare extraction success to the roughly 0.2% handcrafted baseline; if success collapses, the claimed automated training-data extraction does not generalize beyond the corpus used for reward computation.","tokens_in":20262,"feed_emoji":"🕵️","tokens_out":8167,"duration_ms":68453,"temperature":0.7,"pith_summary":"LeakAgent tries to show that privacy leakage from aligned LLMs can be automated end-to-end: an open-source LLM is trained with reinforcement learning to write adversarial prompts that make a target model reveal its system prompt or snippets of memorized training data. The authors claim this is the first unified black-box red-teaming framework covering both leakage types and the first automated attack for training data extraction. They report that the RL agent beats handcrafted prompts and fuzzing baselines, extracts system prompts from real-world GPT store applications even under a guardrail filter, and produces attacks that can be reused as safety-alignment data. If true, this turns red-teaming from a manual, model-specific chore into a scalable learning problem, and it raises the bar for what defenders must protect.","feed_headline":"RL-trained agent beats manual prompts at extracting LLM secrets","feed_subtitle":"LeakAgent auto-generates adversarial prompts for system prompts and training data, spurring better safety alignment.","key_machinery":"The load-bearing object is the attack agent: an open-source LLM, Meta-Llama-3-8B-Instruct, fine-tuned with PPO to output adversarial prompts. The reward that carries the learning is a sliding-window word edit similarity (SWES), defined as the negative log of word edit distance computed over a sliding window, normalized through a sigmoid and combined with a length-matching term; it gives partial credit when the target response contains part of the desired secret, which coarse embedding or n-gram similarities fail to distinguish. Two further mechanisms make the policy learnable: a dynamic temperature schedule that forces high-temperature exploration for the first few tokens and low-temperature exploitation afterward, and a diversity regularizer that rewards prompts dissimilar to already-successful ones. For training data extraction, a two-stage schedule first uses a corpus-search tool to find a memorized sample the target is likely to emit, then continues RL to extract that sample in full.","core_discovery":"On its own terms, the paper claims that deep RL with a fine-grained reward lets a small open model discover adversarial prompts that aligned open and commercial LLMs obey. The attack agent is trained with PPO; its reward is a sliding-window word edit similarity (SWES) between the target response and the desired secret, sigmoid-normalized and combined with a length-matching term, so partial leaks give useful learning signal. For training data extraction, a two-stage procedure first searches a known corpus to identify a memorized sample the target will output, then refines prompts to recover as much of that sample as possible. The evaluation reports system-prompt similarity scores typically above 0.5, a 5.9% training-data success rate on OLMo versus at most 0.2% for handcrafted prompts, and success on all twelve tested GPT store applications even when a PromptGuard filter is applied. The authors conclude that RL-based agents are a promising, general direction for LLM red-teaming.","pith_inferences":["The training-data success rate is measured on OLMo, the same model whose corpus was used to compute rewards during agent training; transferring the trained data-extraction agent to a model with an unknown corpus is an open question, because the two-stage reward cannot be computed without a searchable corpus.","Because the reward is edit-distance based, semantically equivalent but paraphrased leaks would score low; reported similarity scores are therefore likely to understate true leakage rather than overstate it.","Many generated prompts are sequences of tokens like <eos> or token fragments rather than natural instructions, suggesting the learned attack exploits token-level quirks of a specific model; such policies may not transfer to future models with different tokenizers or alignment.","The same fine-grained similarity reward could be reused as a continuous safety metric for measuring residual leakage after defenses, rather than only as a training signal for attacks."],"forward_implications":["Automated privacy red-teaming no longer depends on a human writing extractor prompts; the same RL agent can be retrained against new targets and leakage goals.","Input guardrails such as PromptGuard are not sufficient: LeakAgent's prompts can bypass them on real-world GPT store applications.","Small fine-tuned attack models can outperform larger models using in-context learning on adversarial prompt generation.","Attacks discovered by LeakAgent can be converted into supervised fine-tuning data that hardens the target model with a smaller utility drop than the SecAlign defense.","Even a 5.9% training-data success rate means an automated attacker can recover nontrivial memorized content in 20,000 queries, indicating that manual probing underestimates exposure."],"supporting_citations":[{"why":"Establishes training data extraction from LLMs and supplies the memorization threat model the paper targets.","marker":"Carlini et al., 2021"},{"why":"Shows scalable extraction from production models and provides the 'Repeat poem forever' handcrafted prompt baseline.","marker":"Nasr et al., 2023"},{"why":"Provides the '<eos>' token attack and one of the initial prompt patterns used by LeakAgent for training data extraction.","marker":"Yu et al., 2024a"},{"why":"PromptFuzz is the fuzzing baseline LeakAgent compares against for system prompt extraction.","marker":"Yu et al., 2024b"},{"why":"PLeak is the white-box system-prompt-leakage baseline whose low performance on newer models motivates the black-box RL approach.","marker":"Hui et al., 2024"},{"why":"PPO is the reinforcement learning algorithm used to train the attack agent.","marker":"Schulman et al., 2017"},{"why":"OLMo provides the open-training-data target model used to evaluate training data extraction.","marker":"Groeneveld et al., 2024"},{"why":"Supplies the handcrafted system-prompt extraction prompts used as the HandCraft baseline.","marker":"Zhang & Ippolito, 2023"},{"why":"SecAlign is the defense baseline that LeakAgent-generated data is compared against for safety alignment.","marker":"Chen et al., 2024b"}],"fun_headline_variants":["RL red-teaming agent extracts LLM secrets better than humans","AI trains AI to leak: LeakAgent beats manual prompt hacking","LeakAgent: RL agent cracks system prompts and training data","Deep RL red-teamer outperforms manual prompt attacks","LLM secrets vulnerable: RL agent learns to extract them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training-data setup (Appendix A.2.1) assumes the attacker has a searchable copy of the target model's training corpus, because the reward is computed by matching the target's output against that corpus, and the 5.9% success rate is measured on OLMo, the model whose corpus was used during agent training.","fun_headline_variants_meta":{"raw":{"variants":["RL red-teaming agent extracts LLM secrets better than humans","AI trains AI to leak: LeakAgent beats manual prompt hacking","LeakAgent: RL agent cracks system prompts and training data","Deep RL red-teamer outperforms manual prompt attacks","LLM secrets vulnerable: RL agent learns to extract them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3285,"prompt_tokens":988,"completion_tokens":2297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2213}},"tokens_in":604,"tokens_out":2297,"duration_ms":15956,"temperature":1.0,"reasoning_tokens":2213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:25:00.374297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fully trained data-extraction agent against a second open model whose training corpus is withheld from the attacker, and compare extraction success to the roughly 0.2% handcrafted baseline; if success collapses, the claimed automated training-data extraction does not generalize beyond the corpus used for reward computation.","supporting_citations":[],"review_version":1}