Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

VERA: Variational Inference Framework for Jailbreaking Large Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VERA treats jailbreak prompting as variational inference, training a small attacker LLM to learn the distribution of adversarial prompts and sample diverse jailbreaks without per-prompt optimization.

desk verdict A competent REINFORCE-based jailbreak generator whose reported success rates are undermined by a same-family judge/evaluator circularity; worth reviewing but not as-is. read the letter →

arxiv 2506.22666 v3 pith:7XU67SPW submitted 2025-06-27 cs.CR cs.CLcs.LGstat.ML

Anamika Lochab , Lu Yan , Patrick Pynadath , Xiangyu Zhang , Ruqi Zhang This is my paper
classification cs.CRcs.CLcs.LGstat.ML
keywords jailbreakingvariationalinferenceblack-boxattackLLMsafetyredteamingadversarialpromptsREINFORCEHarmBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that finding jailbreak prompts is not a one-off search problem but a statistical inference problem: for a given harmful behavior, there is a distribution of prompts that would make the target model comply, and a small attacker language model can be trained to approximate that distribution. The proposed method, VERA, treats the attacker as a variational approximation to the posterior over adversarial prompts, optimizes an evidence lower bound with a judge model standing in for the target model's unknown harmful-response likelihood, and updates the attacker with the REINFORCE gradient estimator. Once trained, the attacker generates diverse, fluent jailbreak prompts for that behavior in a single forward pass, with no per-prompt genetic search and no dependence on hand-written jailbreak templates. The paper reports that this distributional approach matches or beats existing black-box and white-box attacks on the HarmBench benchmark, transfers across models, and survives perplexity and circuit-breaker defenses, which matters because it suggests red teams can map the breadth of a model's failures rather than only isolated exploits.

What carries the argument

The load-bearing object is the variational ELBO written as $\mathbb{E}_{q_\theta(x)}[\log J(x,\hat{y}) + \log P(x) - \log q_\theta(x)]$, with the judge $J$ standing in for the unobservable probability that the target model emits harmful content, the prior $P(x)$ fixing the attacker's initial linguistic behavior, and the entropy term $-\log q_\theta(x)$ enforcing diversity. The attacker $q_\theta$ is a small pretrained LLM with LoRA adaptors, so gradient updates touch only a small parameter set, and optimization uses the REINFORCE policy-gradient estimator over batches of sampled prompts. The judge's softmax confidence on the harmful class is the training reward; this is what converts a black-box API into a differentiable-in-expectation objective, and it is the mechanism that lets a single trained attacker emit many distinct, fluent jailbreak prompts without search.

What would settle it

Sample a set of VERA's successful prompts, send them to the target models, and have independent human annotators score the responses for actual harmfulness; if human labels disagree substantially with the judge scores used in training, the reported attack success rates are artifacts of the judge rather than evidence of real jailbreaks.

Watch

Extended reading notes

Core claim

The central claim is that the problem of jailbreaking a black-box LLM can be recast as variational inference over the posterior $P_{LM}(x|y^*)$ of prompts $x$ that elicit a harmful response $y^*$ from the target $P_{LM}$. An attacker LLM $q_\theta(x)$, a small pretrained model with LoRA parameters, is trained to minimize $D_{KL}(q_\theta(x)\|P_{LM}(x|y^*))$, equivalently to maximize the ELBO $\mathbb{E}_{q_\theta}[\log P_{LM}(y^*|x) + \log P(x) - \log q_\theta(x)]$, where the prior $P(x)$ is the attacker model before adaptation and the log-likelihood term is approximated by a judge score $J(x,\hat{y})$ on the target's actual response. The entropy term in the ELBO keeps the learned prompt distribution diverse, and the REINFORCE estimator makes the objective trainable with only API access. The paper argues, with experiments across eight target models, that this single formulation delivers prompt diversity, independence from manual templates, amortized generation cost, transferability across models, and measurable robustness to existing defenses.

Load-bearing premise

The framework rests entirely on trusting the scorer: if the judge's harmfulness score is not a faithful estimate of whether the target model actually produces harmful content, the attacker will learn prompts that fool the judge rather than genuinely unsafe prompts.

Editorial extensions

If this is right

  • For a fixed harmful behavior, one VERA training run replaces many per-prompt optimization loops: after training, generating a new attack is one forward pass through a small model and can be parallelized on GPUs.
  • Because the trained attacker samples from a distribution rather than mutating a template, red teams get a coverage map of a model's failure modes (low self-BLEU, low similarity to the seed prompt), not a single exploit.
  • VERA's prompts transfer across target models, with e.g. GPT-3.5-trained prompts reaching 78.9% ASR on Vicuna-7B, so one attacker can stress-test a family of models.
  • The method keeps working against perplexity filtering and circuit-breaker defenses where GCG and AutoDAN are heavily degraded or nullified, implying that defenses blocking known templates or exact representations are insufficient against distributional attacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same amortized-inference recipe is domain-general: swap the harmfulness judge for any attribute classifier (untruthfulness, bias, toxicity toward a specific group) and the same ELBO would learn the distribution of prompts eliciting that attribute from a black-box model.
  • Because the training reward and the reported metric both come from the same benchmark family, the cleanest independent check is a human-annotation audit: if VERA's top prompts are judged harmless by raters who don't know the classifier, the judge is being gamed rather than the model genuinely unsafe.
  • The per-behavior training cost (acknowledged in the paper's limitations) suggests a natural next step the paper does not take: condition the attacker on the behavior description and train once across many behaviors, approximating a posterior over prompts for any query at test time.
  • A defense-oriented reading is that future alignment work should treat the distribution of effective prompts, not individual prompt families, as the threat surface; perplexity filters and circuit breakers that target known attack patterns will face an adaptive attacker that samples novel phrasings from a learned region of prompt space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes VERA, a black-box jailbreak method that trains a small attacker LLM (with LoRA adapters) to approximate a posterior distribution over adversarial prompts that elicit harmful responses from a target LLM. The variational objective is an ELBO, Eq. (4), in which the target-LLM likelihood is replaced by a judge score J(x, yhat), Eq. (5), and the ELBO is optimized with a REINFORCE gradient estimator, Eq. (8). The authors claim that once trained, VERA samples diverse, fluent, template-free jailbreak prompts without per-query optimization, and they report state-of-the-art ASR on HarmBench across open- and closed-source models, transferability to other models, and robustness to several defenses. The paper also includes ablations on the attacker backbone, KL coefficient, judge model, and a Best-of-N baseline, plus a limitations section that acknowledges per-behavior retraining, black-box query cost, and sparse-reward gradient flatness.

Significance. If the central claims hold, VERA would be a useful red-teaming tool: amortized attack generation, diversity of prompts, and independence from manually curated templates are all valuable practical properties. The mathematical core is standard and correctly stated: the ELBO derivation, the KL regularizer, and the REINFORCE estimator are presented cleanly, and the paper explicitly connects the framework to RLHF in Table 4, which is helpful for situating the method. The authors should be credited for including several ablations, for being transparent about the lack of error bars in the checklist, and for listing limitations in Section F. However, the empirical evidence is not yet convincing for the paper's headline claims. The most serious issue is evaluator circularity: the reward used for training is a HarmBench-family judge, and the reported ASR is also measured with a HarmBench-family classifier, so the reported success rates may reflect overfitting to that classifier family rather than genuine target-model harmful behavior.

major comments (5)
  1. [Section 4.1, Appendix A.1, Algorithm 1, Eq. (5)] The reported ASR is circular with respect to the training reward. The judge used to approximate P_LM(y*|x) in Eq. (5) is the HarmBench Validation Classifier (Appendix A.1), and evaluation success in Section 4.1 is determined by the HarmBench protocol's fine-tuned LLaMA2-13B classifier. The paper never establishes that these two classifiers are independent or jointly calibrated to genuine harmfulness. Because Algorithm 1 early-stops on the first prompt whose judge score exceeds tau (line 13), the optimization can select prompts that exploit idiosyncrasies of this classifier family even when the target model's output is not genuinely harmful. This is load-bearing because the abstract and introduction claim that VERA elicits harmful responses, not classifier scores. Please add an evaluation with an independent judge on a sample of VERA outputs—for example, StrongREJECT, a different LLM-based judge, or human annotation—and report agreement rates. The ablation in Table 8 replaces only the training judge while still evaluating under the HarmBench-family classifier, so it does not break this circularity.
  2. [Related Work, Table 1, Table 4] The closest RL-based baselines are missing from the main comparison. The paper's own Table 4 maps VERA structurally to RLHF with a policy, KL regularizer, and REINFORCE update, and the Related Work cites RL-jack [7], DRL-guided search [6], and the RL-trained attacker of Wang et al. [40]. Yet Table 1 includes none of these methods. Because the central contribution is the optimization objective, not the model class, omitting methods that use the same learning paradigm leaves the main empirical claim unsupported against the most relevant competitors. Please add at least one RL/REINFORCE-based baseline under the same judge and evaluation protocol, or explain why such a comparison is not feasible.
  3. [Section 4.1, Checklist Item 7, Table 1, Figure 3] No error bars, confidence intervals, or multiple-seed results are reported for the main tables or the time-budget comparisons. The checklist explicitly states that error bars are not reported because they would be too expensive. This is insufficient for claims of state-of-the-art performance when the margins are small—for example, VERA at 70.0% versus AutoDAN at 66.0% on Vicuna-7B in Table 1—and for the diversity and regularization benefits central to the variational claim. Please report results over at least three seeds, or provide bootstrapped confidence intervals or a paired statistical test for the ASR comparisons and for Figures 3b and 3c.
  4. [Section 5, Table 1, Table 2, Appendix B] Several reported numbers are mutually inconsistent or not connected to a clearly specified experimental setting. The Conclusion states that VERA achieves ASR of up to 78.6%, but Table 1's maximum VERA ASR is 72.0%, and the 78.9% value in Table 2 is a transfer result on Vicuna-7B from GPT-3.5, not a direct attack result. Appendix B reports ASR values of 94.0% for VERA with the default KL coefficient, while Section 3.3 says the comparisons there use a 50-behavior subset of HarmBench; the main results in Table 1 do not state whether they use the full 400 behaviors or the same subset. Please reconcile these numbers and state explicitly for each table which target behavior set, which target model, and which evaluation metric is used.
  5. [Section 3.1, Appendix B, Eq. (5)] The approximation P_LM(y*|x) ≈ J(x, yhat) requires the judge score to be a meaningful probability of harmful content, but the paper does not provide calibration evidence. The low variance of judge scores across 10 generations (0.107) reported in Appendix B addresses sampling noise, not calibration or gradient-estimator variance, and the objective can be driven by a single sampled response yhat. Please add an analysis of judge calibration on a held-out set or a control experiment with an untrained or random prompt distribution, and report the variance of the REINFORCE gradient estimator or its effect on final ASR.
minor comments (4)
  1. [Throughout] There are many broken cross-references and typos: 'Section ??' in Section 4.1, 'Table ??' in Appendix B, 'expxlain' and 'understandding' in Section H, and other proofreading issues. Please fix these before the final version.
  2. [Abstract and Checklist Item 5] The abstract states 'The code is available at VERA', but Checklist Item 5 answers [No] and says the paper does not currently provide public access to code. Please clarify the actual code release status and, if code is not public, remove or qualify the abstract statement.
  3. [Section 3.2, Algorithm 1] The notation for the gradient update is unclear: line 16 writes 'compute REINFORCE estimator using (8)' but the expression in Eq. (8) already includes the 1/N average; the algorithm should make the batch size B and the relationship between N and B explicit.
  4. [Section 4.3, Table 2] The transferability table uses an em dash in the first column and rows that may be read as missing data; please use a clearer placeholder such as '—' and explain whether diagonal entries are omitted by design, and state whether the transferred prompts are the full output distribution or only the single best prompt per behavior.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported ASR is coupled to the training reward: both use the HarmBench classifier family, so the headline success numbers partly measure fit to the evaluator.

  1. fitted input called prediction [Section 3.1 Eq. (5); Algorithm 1; Appendix A.1; Section 4.1]
    "PLM (y*|x)≈J(x,yˆ) (Eq. 5). Judge Model: We use the HarmBench Validation Classifier as the Judge model in our setup. Success determination follows the HarmBench protocol, utilizing a fine-tuned LLaMA2-13B classifier."

    The training reward in Eq. (4) is log J(x,yhat) through the approximation in Eq. (5), and Algorithm 1 stops and returns the first prompt with J >= tau. Appendix A.1 makes J the HarmBench Validation Classifier, while Section 4.1 defines ASR success via the HarmBench protocol's fine-tuned LLaMA2-13B classifier. The paper does not establish that these classifiers are distinct or independent; both are HarmBench-family classifiers. Thus optimizing Eq.

full rationale

The formal derivation from the KL objective (Eqs. 2-4) to the REINFORCE estimator (Eqs. 6-8) is internally consistent and is not circular by itself: optimizing q_theta to maximize a judge-rewarded log-probability with a KL regularizer is a legitimate training objective. The circularity enters at the measurement layer. Eq. (5) substitutes J(x,yhat) for P_LM(y*|x); Appendix A.1 instantiates J as the HarmBench Validation Classifier; Section 4.1 defines ASR with the HarmBench protocol's LLaMA2-13B classifier. Because the paper does not distinguish the training judge from the evaluation classifier, and both are HarmBench-family artifacts, the central ASR tables are not an independent test of the claim that the prompts elicit genuinely harmful target-model behavior. This is partial rather than total circularity: the AdvBench result (Table 9) uses a keyword judge and is genuinely external, and the transferability results provide some independent signal, so the method is not vacuous. However, for the main HarmBench comparisons, the training reward and the evaluation metric are coupled, warranting a score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No physical or ontological entities are invented; the main load-bearing assumptions are the judge-as-likelihood proxy and the expressivity of the LoRA variational family. The overlap between the training judge and the evaluation classifier is the key circularity concern.

free parameters (6)
  • KL coefficient = 0.8
    Chosen via ablation on a 50-behavior Vicuna-7B subset to maximize ASR (Appendix B, Table 7).
  • Learning rate = 1e-3
    Fixed hyperparameter reported in Appendix A.1.
  • Batch size = 32
    Fixed hyperparameter reported in Appendix A.1.
  • Max epochs per behavior = 10
    Training budget per harmful behavior, Appendix A.1.
  • Early-stopping judge threshold = not reported
    Algorithm 1 uses threshold tau on judge score to stop; the value is never given.
  • LoRA rank = not reported
    LoRA configuration is not specified anywhere in the paper, despite being central to the attacker parameterization.
assumptions (4)
  • domain assumption Judge score J(x, y) approximates P_LM(y*|x) (Eq. 5)
    The entire optimization uses the judge as a likelihood proxy; if the judge is biased, the learned prompts may not elicit genuinely harmful content.
  • domain assumption Prior P(x) is the initial attacker LLM without LoRA
    Fluency and plausibility are enforced by staying close to pretrained Vicuna-7B; this choice shapes all generated prompts.
  • domain assumption Single-sample judge scores provide a sufficient reward signal
    One target response per prompt is scored; the paper reports low variance across 10 samples in Appendix B but does not use variance in training.
  • domain assumption LoRA on a 7B model can approximate the posterior over adversarial prompts
    Expressivity of the variational family is assumed; no convergence or coverage guarantee is given, and the checklist says the paper makes no theoretical claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VERA: Variational Inference Framework for Jailbreaking Large Language Models." pith.science (2026). https://pith.science/paper/7XU67SPW

@misc{pith2026250622666,
  author       = {Pith},
  title        = {Pith review of: VERA: Variational Inference Framework for Jailbreaking Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7XU67SPW}},
  note         = {Machine review of arXiv:2506.22666}
}
read the original abstract

The rise of API-only access to state-of-the-art LLMs highlights the need for effective black-box jailbreak methods to identify model vulnerabilities in real-world settings. Without a principled objective for gradient-based optimization, most existing approaches rely on genetic algorithms, which are limited by their initialization and dependence on manually curated prompt pools. Furthermore, these methods require individual optimization for each prompt, failing to provide a comprehensive characterization of model vulnerabilities. To address this gap, we introduce VERA: Variational infErence fRamework for jAilbreaking. VERA casts black-box jailbreak prompting as a variational inference problem, training a small attacker LLM to approximate the target LLM's posterior over adversarial prompts. Once trained, the attacker can generate diverse, fluent jailbreak prompts for a target query without re-optimization. Experimental results show that VERA achieves strong performance across a range of target LLMs, highlighting the value of probabilistic inference for adversarial prompt generation.

Figures

Figures reproduced from arXiv: 2506.22666 by the authors.

Figure 1
Figure 1. Comparison between traditional jailbreak pipelines and VERA. Traditional methods (e.g., [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of VERA training process. Given a target behavior – e.g [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Key properties of VERA-generated adversarial prompts compared against GPTFuzzer and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Prompt used to condition the attacker LLM. This prompt instructs the attacker to generate [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Representative adversarial prompts generated by VERA that successfully elicit harmful [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models

    cs.CR 2025-10 conditional novelty 6.0 of 10

    VERA-V learns a distribution of text-image jailbreak prompts via variational inference, achieving higher attack success rates and lower toxicity detection than prior multimodal red-teaming methods.

Reference graph

Works this paper leans on

51 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [7]

    Rl-jack: Reinforcement learning-powered black-box jailbreaking attack against llms.arXiv preprint arXiv:2406.08725, 2024

    Xuan Chen, Yuzhou Nie, Lu Yan, Yunshu Mao, Wenbo Guo, and Xiangyu Zhang. Rl-jack: Reinforcement learning-powered black-box jailbreaking attack against llms.arXiv preprint arXiv:2406.08725, 2024

    arXiv 2024

  2. [6]

    When LLM meets DRL: Advancing jailbreaking efficiency via DRL-guided search

    Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. When LLM meets DRL: Advancing jailbreaking efficiency via DRL-guided search. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id= FfFcDNDNol

    work page 2024

  3. [40]

    Reinforcement learning- driven LLM agent for automated attacks on LLMs

    Xiangwen Wang, Jie Peng, Kaidi Xu, Huaxiu Yao, and Tianlong Chen. Reinforcement learning- driven LLM agent for automated attacks on LLMs. In Ivan Habernal, Sepideh Ghanavati, Abhi- lasha Ravichander, Vijayanta Jain, Patricia Thaine, Timour Igamberdiev, Niloofar Mireshghallah, and Oluwaseyi Feyisetan, editors,Proceedings of the Fifth Workshop on Privacy in...

    work page 2024

  4. [1]

    Detecting language model attacks with perplexity, 2023

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023. URLhttps://arxiv.org/abs/2308.14132. 10

    arXiv 2023

  5. [2]

    EBGCG: Effective white-box jailbreak attack against large language model

    Anonymous. EBGCG: Effective white-box jailbreak attack against large language model. InSubmitted to ACL Rolling Review - June 2024, 2024. URL https://openreview.net/ forum?id=EKlispzX65. under review

    work page 2024

  6. [3]

    Defending against alignment-breaking attacks via robustly aligned LLM

    Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment-breaking attacks via robustly aligned LLM. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (V olume 1: Long Papers), pages 10542–10560, Bangkok, Thailand, August 2024. Associatio...

    doi:10.18653/v1/2024.acl-long.568 2024

  7. [4]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2023

    2023

  8. [5]

    When llm meets drl: Advancing jailbreaking efficiency via drl-guided search.arXiv preprint arXiv:2406.08705, 2024

    Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search.arXiv preprint arXiv:2406.08705, 2024

    arXiv 2024

Show all 51 references
  1. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/...

    2023

  2. [9]

    Gradient-based ad- versarial attacks against text transformers

    Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based ad- versarial attacks against text transformers. InProceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pages 5747–5757, November 2021. URL https://aclanthology...

    2021

  3. [10]

    Gradient-based adversarial attacks against text transformers.arXiv preprint arXiv:2104.13733, 2021

    Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers.arXiv preprint arXiv:2104.13733, 2021

    2021 arXiv

  4. [11]

    Cold-attack: Jailbreaking llms with stealthiness and controllability, 2024

    Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability, 2024

    2024

  5. [12]

    Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    2022

  6. [14]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  7. [16]

    Baseline defenses for adversarial attacks against aligned language models, 2023

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. URL https://arxiv. org/abs/2309.00614. 11

    2023 arXiv

  8. [17]

    Improved techniques for optimization-based jailbreaking on large language models,

    Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved techniques for optimization-based jailbreaking on large language models,

  9. [18]

    Open sesame! universal black box jailbreaking of large language models.arXiv preprint arXiv:2309.01446, 2023

    Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models.arXiv preprint arXiv:2309.01446, 2023

    2023 arXiv

  10. [19]

    Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms.arXiv preprint arXiv:2402.14872, 2024

    Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms.arXiv preprint arXiv:2402.14872, 2024

    2024 arXiv

  11. [20]

    Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms.arXiv preprint arXiv:2410.05295, 2024

    Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy V orobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms.arXiv preprint arXiv:2410.05295, 2024

    arXiv

  12. [21]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=7Jwpw4qKkb

    2024

  13. [22]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    2024 arXiv

  14. [23]

    Tree of attacks: Jailbreaking black-box llms automatically, 2024

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically, 2024

    2024

  15. [24]

    Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045, 2023

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045, 2023

    2023 arXiv

  16. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024 arXiv

  17. [26]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  18. [27]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  19. [28]

    Jailbreaking llms: A comprehensive guide (with examples)

    Promptfoo. Jailbreaking llms: A comprehensive guide (with examples). 2025. URL https: //www.promptfoo.dev/blog/how-to-jailbreak-llms/

    2025

  20. [29]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks.Trans. Mach. Learn. Res., 2025, 2023. URL https://api.semanticscholar.org/CorpusID:263671542

    2025

  21. [30]

    Scalable and transferable black-box jailbreaks for language models via persona modulation.arXiv preprint arXiv:2311.03348, 2023

    Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation.arXiv preprint arXiv:2311.03348, 2023

    2023 arXiv

  22. [31]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2023

    2023

  23. [32]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–...

    2024

  24. [33]

    Auto- prompt: Eliciting knowledge from language models with automatically generated prompts

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020

    2010 arXiv

  25. [34]

    A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    2024

  26. [35]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  27. [36]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 13

    2023 arXiv

  28. [37]

    Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944, 2023

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro V on Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944, 2023

    arXiv

  29. [38]

    Universal adversarial triggers for attacking and analyzing NLP

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

    2019

  30. [39]

    Universal adversarial triggers for attacking and analyzing nlp.arXiv preprint arXiv:1908.07125, 2019

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp.arXiv preprint arXiv:1908.07125, 2019

    1908 arXiv

  31. [41]

    Jailbroken: How does llm safety training fail?Advances in Neural Information Processing Systems, 36:80079–80110, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?Advances in Neural Information Processing Systems, 36:80079–80110, 2023

    2023

  32. [42]

    Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36:51008–51025, 2023

    2023

  33. [43]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine learning, 8:229–256, 1992

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning.Machine learning, 8:229–256, 1992

    1992

  34. [44]

    Baichuan 2: Open large-scale language models,

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fa...

  35. [45]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts, 2023

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts, 2023

    2023

  36. [46]

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher, 2024

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher, 2024

    2024

  37. [47]

    How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024

  38. [48]

    Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023

    Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023

    2023

  39. [49]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  40. [50]

    A hacker once used a method to

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024...

    2024

  41. [53]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  42. [2024]

    URLhttps://arxiv.org/abs/2405.21018

    arXiv

  43. [2025]

    URLhttps://arxiv.org/abs/2309.10305

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.