Adversarial Preference Learning for Robust LLM Alignment

Bo Tang; Chaochao Lu; Chao Yang; Chen Chen; Chenyang Xi; Feiyu Xiong; Jie Hu; Jingfeng Zhang; Junyi Zhu; Keming Mao

REVIEW 3 major objections 6 minor 48 references

Adversarial Preference Learning for Robust LLM Alignment

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adversarial preference learning makes LLMs far more resistant to jailbreaks in the paper's tests, with a small utility cost.

desk verdict APL offers a clever self-contained adversarial alignment idea, but the refusal-flavored safety gains and test-set tuning make the headline claim premature. read the letter →

arxiv 2505.24369 v1 pith:3KNNUGDB submitted 2025-05-30 cs.LG cs.AI

Yuanfu Wang , Pengyu Wang , Chenyang Xi , Bo Tang , Junyi Zhu , Wenqiang Wei , Chen Chen , Chao Yang

show 8 more authors

Jingfeng Zhang Chaochao Lu Yijun Niu Keming Mao Zhiyu Li Feiyu Xiong Jie Hu Mingchuan Yang

This is my paper

classification cs.LGcs.AI MSC 68T50

keywords adversarialtrainingpreferencelearningLLMsafetyjailbreakrobustnessdirectoptimizationredteamingiterative

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that language models can be made substantially more resistant to adversarial jailbreak attacks without relying on external classifiers, expensive human annotation, or fixed attack templates. The proposed method, Adversarial Preference Learning, turns the alignment problem into an iterative game: a generative attacker rewrites prompts to exploit the defender's weaknesses, and the defender is trained on the most effective attacks. On Mistral-7B, harmful outputs drop from 5.88% to 0.43% and attack success rates fall by up to 65%, while an MT-Bench score of 6.59 stays close to the baseline's 6.78.

What carries the argument

The core mechanism is an iterative minimax training loop between a generative attacker and a defender, both trained with Direct Preference Optimization. The attack-effectiveness reward is the defender's own log preference ratio between the dispreferred and preferred response, minus a reference-model baseline; this intrinsic signal replaces external classifiers. The attacker and defender co-evolve over 30 iterations, with the most and least effective adversarial prompts selected each round to train attacker and defender respectively.

What would settle it

Run the APL training loop on Mistral-7B and then evaluate the trained defender on a suite of benign prompts that are phrased in unusual or adversarial-sounding ways; if the false-refusal rate rises sharply while the attack-success rate falls, that would show the reward signal is being gamed by over-refusal rather than by improved discrimination. A cleaner test would be to pit the trained defender against a held-out, never-seen attack family and check whether the reduction in attack success rate persists out of distribution.

Watch

Extended reading notes

Core claim

The paper's central claim is that an iterative adversarial training loop using the model's own preference probabilities as the attack-effectiveness signal can significantly improve safety against diverse attacks while keeping utility competitive. Concretely, the defender's harmlessness win rate over the base model reaches 83.33%, harmful outputs drop from 5.88% to 0.43%, and attack success rates on HarmBench fall by 21% to 65% across Direct Request, Zero-Shot, Few-Shot, and GCG attacks on Mistral-7B-Instruct-v0.3. The method also improves the already-aligned Meta-Llama-3-8B-Instruct.

Load-bearing premise

The method assumes that the defender's own preference probability is a trustworthy measure of how harmful an attack is, so that optimizing against it improves genuine safety rather than merely teaching the model to refuse anything unusual.

Editorial extensions

If this is right

If the gains hold, adversarial robustness could be improved without training a separate reward model or safety classifier, reducing annotation cost.
An iterative attacker that adapts to the defender's current weaknesses may cover more of the adversarial input space than static attack libraries.
The method's usefulness on an already-aligned model suggests it could serve as a post-alignment hardening step before deployment.
The framework's closed-loop design points toward fully automated red-teaming and safety-tuning pipelines, provided the intrinsic reward signal remains trustworthy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported robustness gains may partly reflect increased over-refusal rather than genuine harmlessness; the paper's own XS-Test false-refusal rate rises to 40.89%, so an external judge could drive the internal reward up by simply refusing more broadly.
A natural testable extension is to replace the intrinsic preference ratio with a calibrated harmlessness classifier and compare whether the iterative game still helps; that would reveal how much of the gain is due to adversarial co-evolution versus the reward signal.
The framework could in principle be transferred to other preference-based alignment objectives such as SimPO or IPO, since only the reward signal in Eq. 5 and the DPO update need swapping.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on APL. The core idea—using the defender's own preference probabilities as the attack reward, then co-training attacker and defender with DPO—is clean and genuinely useful. It removes the need for a separate reward model or classifier, and the ablation against APL(RM) suggests the intrinsic signal carries real information. The paper is honest: the limitations section calls out increased false refusals and sensitivity to beta and alpha. I'd give credit for that.

The problems are in the evaluation, not the method. The headline 83.33% win rate comes from 60 prompts with no error bars or significance tests. The authors picked the best hyperparameters (beta_att=0.03, beta_def=0.01, alpha=0.2) directly on the test metric, so the result is optimistic. More importantly, XS-Test false refusal on Mistral doubles from 20% to 40.89%. That's not a side effect; it's the main event. Eq. 5's reward measures preference-probability imbalance, and the attacker can in principle win by making the model unsure, not necessarily by eliciting harmful content. The defender then suppresses that uncertainty, which is exactly the broad refusal behavior you'd expect. Since HarmBench ASR and LLaMA-Guard are refusal-oriented judges, part of the "safety" gain is likely just refusals.

None of this is an internal contradiction—the equations are coherent and the pipeline is reproducible in principle. But the central claim, "significantly enhances model safety while maintaining competitive utility," is not established. On utility, MT-Bench of 6.59 vs 6.78 is fine, but the XS-Test increase is a direct utility cost the paper underweights. I'd also like to see comparisons against MART or SEAS, which it cites but doesn't benchmark.

Who should read it: people working on red-teaming and alignment with limited annotation budgets. It deserves a serious referee—the idea is worth engaging with, and the negative result on refusal is useful. For a top venue, though, the authors need held-out hyperparameter selection, error bars, and a decomposition of safety gains into genuine robustness versus refusal. As a desk decision, I'd send it to review and let referees push on that. The current version wouldn't be my accept, but it's a solid conditional.

Referee Report

3 major / 6 minor

Summary. The paper proposes Adversarial Preference Learning (APL), an iterative framework in which a generative attacker rewrites prompts and a defender is trained via DPO on the most effective adversarial prompt, with attack effectiveness measured by the defender's log-probability ratio of dispreferred to preferred responses relative to a reference model. The authors report experiments on Mistral-7B-Instruct-v0.3 and Llama-3-8B-Instruct showing reduced HarmBench attack success rates, higher harmlessness win rates, lower unsafe scores, and roughly maintained MT-Bench/LC-WinRate. The central claim (Section 4.3) is that APL 'significantly enhances model safety by improving adversarial robustness while maintaining competitive utility.'

Significance. If valid, APL would be an appealing self-contained alignment method: no external reward model is needed during training, the attacker and defender co-evolve automatically, and the framework is simple enough to build on DPO. The ablation study (Figure 3, Table 2) and cross-model results (Table 1) are useful. However, the significance is conditional on the validity of the intrinsic reward as a harmfulness proxy and on the stability of the small-sample evaluations; the paper's own XS-Test results raise a concrete risk that the method is partly learning broad refusal rather than targeted robustness. The paper does not include code or machine-checked proofs, so reproducibility rests on the appendix details.

major comments (3)

[§3.3, Eq. (5); Table 1] The attack-effectiveness reward R(x'; y_pre, y_dis) is defined as the defender's log-probability ratio between the dispreferred and preferred completions (minus a reference term), which measures preference-probability imbalance, not harmfulness. An attacker maximizing R can succeed by producing prompts that merely increase the defender's uncertainty between y_pre and y_dis, and the defender's DPO update can lower R by refusing broadly rather than by becoming robust to the specific harmful content. Table 1 (Mistral XS-Test rising from 20.00% to 40.89%, the worst among all methods) is direct evidence of broad refusal, and since HarmBench ASR is judged by a refusal-oriented classifier while the GPT-4o win rate rewards harmless (often refusal-like) outputs, the reported robustness gains may partly reflect over-refusal. I ask for a disentangling analysis: refusal rates on HarmBench prompts, ASR computed conditional on non-refusal responses, and evidence that R tracks harmfulness (e.g., correlation with external harmfulness labels) rather than general uncertainty.
[§4.2, Table 1; Figure 3; Table 2] The headline harmlessness win rate is computed on only 60 prompts from HH-RLHF, and the paper reports no error bars, confidence intervals, or significance tests for any comparison in Table 1. Moreover, the main hyperparameters (β_att=0.03, β_def=0.01, α=0.2) appear to be selected using the same test-set metrics that support the central claim (Figure 3 and Table 2), which risks overfitting the evaluation. Please report variance across at least a few seeds, bootstrap confidence intervals for the win rate, and a clear separation between validation and test sets for hyperparameter selection, or at a minimum disclose the selection protocol.
[§3.4, Algorithm 1; §4.1] The defender is trained with DPO on (x'_w, y_pre, y_dis), reusing preference labels from the original prompt x without re-annotation for the adversarial rewrite x'_w. This assumes that y_pre/y_dis remain valid and semantically appropriate for x'_w, which is not obvious because x'_w is generated to be 'more harmful.' If y_pre is a refusal for a harmful original prompt, this training directly reinforces refusal on adversarial prompts, providing another mechanism for the XS-Test increase. The authors should validate the label transfer, for example by sampling and manually/LLM-checking whether y_pre is still the preferred response to x'_w, or by comparing against a variant that re-annotates responses for adversarial prompts.

minor comments (6)

[Abstract; §3.3] The term 'direct harmfulness metric' overstates what Eq. (5) computes; consider renaming it 'attack-effectiveness metric' to avoid conflating a likelihood ratio with harmfulness.
[Algorithm 1; Eq. (5)] Algorithm 1 uses α and β as update coefficients in lines 7–8, while α is the reward weight in Eq. (5) and β_att/β_def are DPO regularization coefficients; this notation clash should be fixed.
[§4.1; Algorithm 1] Section 4.1 says the attacker is initialized with an 'untrained base model,' but Algorithm 1 sets π_att ← π_ref; please clarify what 'untrained' means and how the prompt-based generation in Appendix D.1 is initialized.
[Table 1; §4.3] The XS-Test column is not discussed in the main text despite APL having the highest false-refusal rate on Mistral; please address this directly when claiming 'competitive utility.'
[Eq. (4); Eq. (6)] Equation (4) writes the inner problem as a minimization over x' ∈ A(x), but the algorithm samples K candidates and selects the argmax of R (Eq. 6); the text should clarify that Eq. (4) is a conceptual formulation rather than the implemented optimization.
[Table 2; §4.4] Table 2's caption says 'defined in 5' instead of 'Eq. (5)'; also the α values in the table should be referenced consistently with the main text.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity: the APL loop is a standard adversarial game and the headline claims are externally evaluated, though the intrinsic reward proxy and test-set hyperparameter selection are validity risks.

full rationale

APL's derivation chain is not circular at the equation level. Equation (5) defines the attacker's reward R as the defender's own preference-ratio imbalance, and the defender is then updated by DPO on the selected adversarial prompt x'_w. This is a standard minimax/adversarial-training construction: the intrinsic signal is an input to training, and the defender's improvement is not measured by R. The headline safety results are externally evaluated: HarmBench ASR uses the Llama-2-13b-cls classifier, the harmlessness win rate is judged by GPT-4o, and the Unsafe score is measured by LLaMA-Guard. These external judges break any chain in which the conclusion is equivalent to the training objective. There is no load-bearing self-citation chain and no imported uniqueness theorem; the cited baselines and benchmarks are standard external resources. Two validity concerns are present but are not circularity. First, the paper's own XS-Test results show false refusal rising from 20.00% to 40.89%, and the Limitations section concedes 'increased false refusal rates,' so part of the reported robustness gain may reflect broad refusal rather than targeted safety improvement; this is an external-validity risk. Second, Section 4.4 selects beta_att, beta_def, and alpha using the same harmless win-rate metric later reported as the headline 83.33%, so that figure is partly a selection artifact rather than an independent prediction; this is an evaluation-hygiene concern. Neither concern makes any derived quantity identical to its input by construction, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on three free hyperparameters chosen on the test metric, and on the assumption that a defender's own preference log-ratio is a trustworthy harmlessness signal. No new entities are introduced.

free parameters (3)

beta_att (attacker DPO regularization) = 0.03
Chosen by tuning in {0.01, 0.03, 0.1, 0.3, Fixed}; the best test-set win rate (83.33%) is reported at beta_att=0.03 in Section 4.4.
beta_def (defender DPO regularization) = 0.01
Tuned in {0.01, 0.03, 0.1, 0.3}; best win rate at beta_def=0.01 (Section 4.4).
alpha (reward weight in Eq. 5) = 0.2
Tuned over {0.0, 0.1, 0.2, 0.4, 0.6, 1.0}; highest win rate 83.33% at alpha=0.2 (Table 2).

assumptions (4)

standard math Bradley-Terry preference model underlies the DPO loss used to train both attacker and defender.
DPO derivation in Rafailov et al. (2024), used in Eq. 2, relies on the Bradley-Terry model of pairwise preferences.
domain assumption The HH-RLHF harmless dataset provides valid preferred/dispreferred response pairs.
The method trains on the human preference labels from Bai et al. (2022a); if these labels are noisy or biased, the intrinsic reward inherits the bias.
domain assumption The defender's preference probability ratio is a valid proxy for harmfulness of a prompt.
Eq. 5 defines attack effectiveness as a log-ratio of defender probabilities; the paper assumes optimizing this ratio improves genuine safety rather than only making the model refuse more.
domain assumption Training the defender with DPO on adversarial prompts reduces real-world harmful outputs rather than only satisfying the intrinsic reward.
The XS-Test increase (false refusal) suggests the defender partially games the reward, so the assumption that the metric is aligned with true harmlessness is not fully met.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adversarial Preference Learning for Robust LLM Alignment." pith.science (2026). https://pith.science/paper/3KNNUGDB

@misc{pith2026250524369,
  author       = {Pith},
  title        = {Pith review of: Adversarial Preference Learning for Robust LLM Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3KNNUGDB}},
  note         = {Machine review of arXiv:2505.24369}
}

read the original abstract

Modern language models often rely on Reinforcement Learning from Human Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to adversarial attacks due to three key limitations: (1) the inefficiency and high cost of human annotation, (2) the vast diversity of potential adversarial attacks, and (3) the risk of feedback bias and reward hacking. To address these challenges, we introduce Adversarial Preference Learning (APL), an iterative adversarial training method incorporating three key innovations. First, a direct harmfulness metric based on the model's intrinsic preference probabilities, eliminating reliance on external assessment. Second, a conditional generative attacker that synthesizes input-specific adversarial variations. Third, an iterative framework with automated closed-loop feedback, enabling continuous adaptation through vulnerability discovery and mitigation. Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly enhances robustness, achieving 83.33% harmlessness win rate over the base model (evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured by LLaMA-Guard), and lowering attack success rate by up to 65% according to HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against the base model.

Figures

Figures reproduced from arXiv: 2505.24369 by the authors.

**Figure 2.** Comparison of Attack Success Rate (ASR) over multiple training iterations. Our method (APL) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Ablation study on attacker βatt in {0.01, 0.03, 0.1, 0.3, Fixed} and defender βdef in {0.01, 0.03, 0.1, 0.3}, with the performance evaluated based on harmless win rate against Mistral-7B. vs 14.00%), FS (6.25% vs 17.50%), and GCG (13.75% vs 16.25%), with a higher Win Rate of 56.67%. For safety, APL achieves a lower Unsafe score (1.34% vs 1.95%) and a higher RM score (2.09 vs 2.02). These results further demonstrate … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Evolution of Harmlessness Win Rate, evalu [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Ablation study on attacker βatt in {0.01, 0.03, 0.1, 0.3, Fixed} and defender βdef in {0.01, 0.03, 0.1, 0.3}, with performance evaluated on length-controlled Win Rate (LC-WR) of helpfulness against Mistral-7B. ble of consistently improving the harmlessness win rate thr…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 6 canonical work pages

[1]

Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, and Prateek Jain. 2024. Does safety training of llms generalize to semantically related natural prompts? arXiv preprint arXiv:2412.03235

arXiv 2024
[2]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. https://arxiv.org/abs/2404.02151 Jailbreaking leading safety-aligned llms with simple adaptive attacks . Preprint, arXiv:2404.02151

arXiv 2024
[3]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022
[4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022
[5]

Rishabh Bhardwaj and Soujanya Poria. 2023. https://arxiv.org/abs/2308.09662 Red-teaming large language models using chain of utterances for safety-alignment . Preprint, arXiv:2308.09662

arXiv 2023
[6]

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442

arXiv 2023
[7]

Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. 2024. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030

arXiv 2024
[8]

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. 2023. Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505

arXiv 2023

Show all 48 references

[9]

Muxi Diao, Rumei Li, Shiyang Liu, Guogang Liao, Jingang Wang, Xunliang Cai, and Weiran Xu. 2024. Seas: Self-evolving adversarial safety optimization for large language models. arXiv preprint arXiv:2408.02632

2024 arXiv
[10]

Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. 2023. Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak. arXiv preprint arXiv:2312.04127

2023 arXiv
[11]

Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

2024 arXiv
[12]

Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. 2024. https://doi.org/10.18653/v1/2024.naacl-long.107 MART : Improving LLM safety with multi-round automatic red-teaming . In Proceedings of the 2024 Conference of the North A...

2024 doi
[13]

Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679

2024 arXiv
[14]

Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. 2024. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:2402.19464

2024 arXiv
[15]

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. https://arxiv.org/abs/2309.00614 Baseline defenses for adversarial attacks against aligned language models ....

2023 arXiv
[16]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[17]

Bojian Jiang, Yi Jing, Tianhao Shen, Tong Wu, Qing Yang, and Deyi Xiong. 2024. Automated progressive red teaming. arXiv preprint arXiv:2407.03876

2024 arXiv
[18]

Haibo Jin, Ruoxi Chen, Andy Zhou, Yang Zhang, and Haohan Wang. 2024. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models. arXiv preprint arXiv:2402.03299

2024
[19]

Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, et al. 2024. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:2405.18540

2024 arXiv
[20]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

2023
[21]

Zeyi Liao and Huan Sun. 2024. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921

2024 arXiv
[22]

Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. 2024. https://arxiv.org/abs/2410.05295 Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms . Preprint, arXiv:2410.05295

2024 arXiv
[23]

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451

2023 arXiv
[24]

AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

2024 arXiv
[25]

Ilya Loshchilov and Frank Hutter. 2019. https://arxiv.org/abs/1711.05101 Decoupled weight decay regularization . Preprint, arXiv:1711.05101

2019 arXiv
[26]

Aleksander M a dry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. stat, 1050(9)

2017
[27]

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249

2024 arXiv
[28]

Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.41 FLIRT : Feedback loop in-context red teaming . In Proceedings of the 2024 Conference on E...

2024 doi
[29]

Charles O'Neill, Jack Miller, Ioana Ciuca, Yuan-Sen Ting, and Thang Bui. 2023. https://arxiv.org/abs/2308.13768 Adversarial fine-tuning of language models: An iterative optimisation approach for the generation and detection of problematic content . Preprint, arXiv:2308.13768

2023 arXiv
[30]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2022 arXiv
[31]

Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2024. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873

2024 arXiv
[32]

Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286

2022 arXiv
[33]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[34]

Rubra AI . 2024. https://doi.org/10.57967/hf/2684 Mistral-7b-instruct-v0.3 (revision 94e63ce)

2024 doi
[35]

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. https://arxiv.org/abs/2308.01263 Xstest: A test suite for identifying exaggerated safety behaviours in large language models . Preprint, arXiv:2308.01263

2024 arXiv
[36]

Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. 2024. https://arxiv.org/abs/2402.16822 Rainbow teaming: Open-ended genera...

2024 arXiv
[37]

Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. 2024. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:...

2024 arXiv
[38]

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980

2020 arXiv
[39]

Nevan Wichers, Carson Denison, and Ahmad Beirami. 2024. Gradient-based language model red teaming. arXiv preprint arXiv:2401.16656

2024 arXiv
[40]

Sophie Xhonneux, Alessandro Sordoni, Stephan G \"u nnemann, Gauthier Gidel, and Leo Schwinn. 2024. Efficient adversarial training in llms with continuous attacks. arXiv preprint arXiv:2405.15589

2024 arXiv
[41]

Yuting Yang, Pei Huang, Juan Cao, Jintao Li, Yun Lin, and Feifei Ma. 2024. A prompt-based approach to adversarial example generation and robustness enhancement. Frontiers of Computer Science, 18(4):184318

2024
[42]

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463

2023 arXiv
[43]

Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373

2024 arXiv
[44]

Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, and Songlin Hu. 2024. Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction. arXiv preprint arXiv:2409.16783

2024 arXiv
[45]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

2023 arXiv
[46]

Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2024. Autodan: interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling

2024
[47]

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

2019 arXiv
[48]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. https://api.semanticscholar.org/CorpusID:260202961 Universal and transferable adversarial attacks on aligned language models . ArXiv, abs/2307.15043

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, and Prateek Jain. 2024. Does safety training of llms generalize to semantically related natural prompts? arXiv preprint arXiv:2412.03235

arXiv 2024

[2] [2]

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. https://arxiv.org/abs/2404.02151 Jailbreaking leading safety-aligned llms with simple adaptive attacks . Preprint, arXiv:2404.02151

arXiv 2024

[3] [3]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022

[4] [4]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022

[5] [5]

Rishabh Bhardwaj and Soujanya Poria. 2023. https://arxiv.org/abs/2308.09662 Red-teaming large language models using chain of utterances for safety-alignment . Preprint, arXiv:2308.09662

arXiv 2023

[6] [6]

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442

arXiv 2023

[7] [7]

Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. 2024. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030

arXiv 2024

[8] [8]

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. 2023. Attack prompt generation for red teaming and defending large language models. arXiv preprint arXiv:2310.12505

arXiv 2023