Pith. sign in

REVIEW 4 major objections 6 minor 58 references

Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LLM red-teaming should optimize for goal-driven diversity—coverage of a risk-category by attack-style behavior grid—and presents QDRT, which improves QD-Score and behavior coverage over prior automated red-teaming…

desk verdict QDRT is a genuinely new combination of QD and RL for red-teaming with large empirical gains on judge-defined coverage; the evaluation is partly circular and the arXiv abstract overclaims commercial-model tests, but the paper deserves serious peer review. read the letter →

arxiv 2506.07121 v2 pith:4OYB4FZ4 submitted 2025-06-08 cs.LG cs.AIcs.NE

Ren-Jian Wang , Ke Xue , Zeyu Qin , Ziniu Li , Sheng Tang , Hao-Tian Li , Shengcai Liu , Zhi Yu
show 2 more authors
Yuanpeng Tan Chao Qian
This is my paper
classification cs.LGcs.AIcs.NE
keywords red-teaminglargelanguagemodelsafetyquality-diversityoptimizationadversarialpromptgenerationbehavior-conditionedreinforcementlearningMAP-Elitesattackdiversityjailbreak
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Quality-Diversity Red-Teaming (QDRT) aims at a recurring trade-off in automated LLM red-teaming: diverse prompts are usually weak, while RL-trained attackers that are strong collapse into homogeneous prompts. The paper's solution is to define diversity over a structured behavior grid—14 risk categories crossed with 11 attack styles—and to train several specialized attacker LLMs with behavior-conditioned reinforcement learning, storing data in a MAP-Elites archive buffer. On Llama-3.1-8B, Gemma-2-9B, and Qwen2.5-7B, the paper reports that QDRT fills 97–99% of that grid with high-toxicity attacks and improves QD-Score by 22.13% and behavior coverage by 19.33% over state-of-the-art baselines, with attacks that also transfer to larger models. If correct, this turns automated red-teaming into a more systematic safety audit: the archive becomes a per-category, per-style map of where a model fails, rather than a collection of superficially dissimilar sentences.

What carries the argument

The central object is the behavior space $\mathcal{B} = \mathcal{S} \times \mathcal{C}$—14 risk categories crossed with 11 attack styles—which serves as the archive grid. The load-bearing identity is the behavior-conditioned reward $r(c,s,x)=\mathbb{E}_{y\sim p(\cdot|x)}[p_\phi(c|x,y)\,p_\psi(s|x)\,r(x,y)]$, which credits an attack only when the target response is toxic, the risk category matches the request, and the style matches. Two supporting mechanisms carry the algorithm: a deep MAP-Elites prioritized replay buffer that keeps a per-cell quality-ranked set of prompts, and an adaptive behavior-assignment procedure that periodically gives each attacker the attack styles it generates best. Together these let specialized attackers cover a grid that a single policy tends to collapse onto a few modes.

What would settle it

Have independent human annotators assign one of the 11 attack styles to a random sample of prompts from a QDRT archive and compare with the judge-model labels; if agreement is low or many prompts are judged style-ambiguous, the reported 97–99% behavior coverage is a label artifact rather than genuine diversity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that meaningful attack diversity in red-teaming is a behavior-space coverage problem, not a text-similarity problem. The authors define an attack's behavior as a pair (risk category, attack style), estimate the behavior-conditioned reward $r(c,s,x)=\mathbb{E}_{y\sim p(\cdot|x)}[p_\phi(c|x,y)\,p_\psi(s|x)\,r(x,y)]$, and use it both to train multiple attackers toward specific cells and to maintain a deep MAP-Elites replay buffer that keeps the highest-toxicity prompt in each cell. They report that QDRT simultaneously raises toxicity and coverage on every tested target, with 97–99% coverage in the augmented-data setting, and that the resulting prompts transfer to unseen larger models. The central claim is that goal-driven diversity removes the quality–diversity trade-off that limits single-attacker, embedding-similarity approaches.

Load-bearing premise

The whole coverage claim rests on the automatic judges—especially the closed-source attack-style judge the paper's own limitation passage flags as opaque—labeling risk category and style accurately enough that a filled grid cell really is that style of attack.

Editorial extensions

If this is right

  • Safety teams can audit a model against a fixed behavior grid, making coverage numbers directly comparable across models and over successive safety updates.
  • Because behavior-conditioned supervised fine-tuning already beats plain fine-tuning in the paper's tables, the structured diversity signal helps even before reinforcement learning is applied.
  • QDRT's gains hold across several open-source targets and transfer to larger unseen models, implying that a diverse archive is a reusable asset rather than a target-specific artifact.
  • The framework is explicitly compatible with arbitrary reinforcement learning algorithms, so future work can replace the GFlowNets training loop used here without reworking the behavior-space machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same reward-and-archive machinery would transfer to other behavior taxonomies—multi-turn attack strategies, privacy harms, or policy-specific risk lists—since nothing in the method depends on the particular 14-by-11 grid.
  • Editorial caution: coverage is measured with the same judge models that label the cells, so the 97–99% figures are best read as an upper bound on true strategic diversity until a held-out human-annotated label set confirms the styles.
  • Editorial extension: the paper's own stated limitation—reliance on a closed-source style judge with opaque logic—suggests a concrete stress test: fine-tune an open judge, measure label agreement on QDRT's archive, and see whether the reported coverage survives re-labeling.
  • Editorial inference: the transfer results raise a natural follow-up question of whether archive coverage degrades when the target model is safety-tuned after the archive is built, and whether QDRT's periodic behavior reassignment can adapt to that shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Quality-Diversity Red-Teaming (QDRT), a generator-level red-teaming framework that formulates adversarial prompt generation as a quality-diversity optimization problem over a structured behavior space of risk categories (14 classes, following Llama-Guard-3) and attack styles (11 classes, extending Rainbow Teaming). The method combines behavior-conditioned supervised fine-tuning, behavior-conditioned RL rewards, multiple specialized attacker models with an adaptive behavior-assignment strategy, and a deep MAP-Elites replay buffer. Experiments on six open-weight target models (GPT-2, Llama-3.2-3B, Gemma-2-2B, Gemma-2-9B, Llama-3.1-8B, Qwen2.5-7B) compare QDRT with REINFORCE, GFlowNets, and several ablations, reporting higher QD-Score and coverage, with further transfer experiments to larger models. The paper also releases code and documents hyper-parameters and instructions in the appendices.

Significance. If the quantitative claims hold, QDRT addresses a real weakness of prior automated red-teaming methods, which typically optimize toxicity without explicitly controlling attack style and risk category. The behavior-space formulation is a useful step beyond embedding-based or n-gram diversity measures, and the combination of MAP-Elites replay, behavior-conditioned rewards, and multiple attackers is technically coherent. The paper ships a code repository, detailed hyper-parameters, and full prompt templates, and it includes ablation studies, archive visualizations, archive profiles, and transfer experiments; these are concrete strengths. The main claims are nevertheless currently supported only by a small set of runs without dispersion measures in the headline table, and the abstract overstates the scope of the empirical evaluation by listing commercial models among the targets actually tested. The diversity metric also depends entirely on judge models that also supply the training reward, so the interpretation of 'coverage' as meaningful, human-recognizable diversity requires additional validation.

major comments (4)
  1. [Abstract and Section 4.3] The abstract states that QDRT is evaluated 'against a wide range of target LLMs, including ... commercial models GPT-4.1 and GPT-5-Chat.' The experimental sections, however, report only open-weight targets: GPT-2, Llama-3.2-3B, Gemma-2-2B, Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B. GPT-4.1 appears only as an attack-style judge (Section 4.3), and GPT-5-Chat appears nowhere. This is an unsupported empirical claim and should be corrected or the corresponding experiments added.
  2. [Table 1 and Section 4.3] The central quantitative claim of a 22.13% QD-Score and 19.33% behavior coverage improvement rests on Table 1, which reports means over three runs without standard deviations, confidence intervals, or significance tests. Figure 2 provides medians and quartile bands only for the basic-benchmark setting, not for the augmented-dataset results in Table 1. Without dispersion measures, the reader cannot judge whether the margins over GFlowNets are reproducible; please report per-run results, standard deviations, or a statistical test for the three complex-setting target models.
  3. [Introduction, contribution bullet, and Table 1] The claimed aggregate improvements do not match a direct computation from Table 1. Using QDRT versus GFlowNets on the three complex targets, the relative QD-Score improvements are approximately 13.8% (Llama-3.1-8B), 29.7% (Gemma-2-9B), and 33.2% (Qwen2.5-7B), averaging about 25.6%, not 22.13%; the corresponding coverage improvements average about 24.8%, not 19.33%. Please clarify how the stated percentages were computed, or update them to reflect the table.
  4. [Section 3.1, Eq. (2), Section 4.1, and Algorithm 2] The same judge models p_phi (risk category) and p_psi (attack style) are used for the training reward in Eq. (2) and for the evaluation metrics QD-Score and coverage in Section 4.1, and Algorithm 2 places prompts into archive cells using exactly these judges. Consequently, high coverage may partly reflect that the attacker learned to imitate the judge's label assignments rather than to produce diverse, human-recognizable attack strategies. This is a correctness-risk concern rather than an inherent flaw, but it is load-bearing for the paper's diversity claim. I ask for a concrete validation study: sample a set of generated prompts, have human annotators label risk category and attack style, and report agreement with p_phi and p_psi; or add an independent style classifier or a qualitative evaluation showing that distinct cells correspond to distinct observable attack strategies.
minor comments (6)
  1. [Algorithm 3] The loop condition in Algorithm 3 reads 'while S ≠ ∅', but S denotes the set of attack styles that is never modified; the termination condition should be 'while T ≠ ∅'. Please fix this typo.
  2. [Algorithm 2] Line 4 of Algorithm 2 has an extra closing parenthesis in the tuple '(x, r(x, y), p_phi(·|x, y), p_psi(·|x)))'. Please correct the syntax.
  3. [Section 3.1] The definition of p_psi(s|x) as 'the probability of the corresponding token of the attack style ID' is underspecified: it is unclear how a single token probability is obtained from a generative judge model and how the 11 style classes are mapped to token IDs. Please provide the exact decoding procedure.
  4. [Section 4.1] The evaluation metrics section does not specify the archive size or the number of cells used for QD-Score and coverage in the complex setting; the hyper-parameters table lists a replay buffer size of 5000, but the archive structure is not described. Please state the number of cells (14 × 11) and how QD-Score is normalized.
  5. [Appendix D.5] Tables 5 and 6 contain empty entries for several attack styles (e.g., Slang, Misspellings, Word Play in Table 5), suggesting that some styles initially have no seed prompts. The paper should state how these styles are handled in SFT and whether the final results depend on the initial data imbalance.
  6. [Section 2, Eq. (1)] The notation r(x, y) is used for the toxicity reward while Eq. (2) introduces r(c, s, x) for the behavior-conditioned score; the relationship and the expectation over y in Eq. (2) could be stated more explicitly to avoid confusion.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: behavior-coverage metric reuses the same judge models that appear in the training reward; effectiveness and transfer claims remain independent.

  1. self definitional [Eq. (2) (Section 3.2); Algorithm 2 (Appendix A.3); Section 4.1 Metrics]
    "we define the behavior-conditioned score r(c, s, x) as the probability that the attack x is successful with the given goal-driven behavior (c, s): r(c, s, x) = E_{y∼p(·|x)}[pϕ(c|x, y)pψ(s|x)r(x, y)] ... To evaluate the performance of the methods, we put the generated adversarial prompts into an MAP-Elites archive. ... (2) Coverage: The percentage of cells that have been covered by prompts in the archive. ... Algorithm 2: c←arg max c pϕ(c|x, y); s←arg max s pψ(s|x)."

    The same evaluators pϕ and pψ appear in the RL reward being optimized and in the cell-assignment rule that defines the archive and the coverage metric. A prompt is placed into the archive cell (argmax pϕ, argmax pψ), and coverage is simply the fraction of those cells that contain at least one generated prompt. Because QDRT samples every behavior (c,s) during training and rewards high pϕ(c|x,y)pψ(s|x)r(x,y), the trained attackers are directly optimized to produce prompts that these exact judge models label with the requested category and style. The reported 19.33% coverage improvement therefore measures, in part, how well the policy matches the same classifiers used as its reward; it is not an independent probe of attack diversity.

full rationale

The derivation chain is otherwise self-contained. The risk-category and style spaces are imported from external sources (Llama-Guard-3 and Rainbow Teaming), the toxicity signal r comes from Llama-Guard-3, and the transfer experiments (Appendix D.4) evaluate prompts on unseen target models, so those parts are not circular. The GEM citation [26] is by overlapping authors but is used as an implemented baseline and empirically compared, not as a load-bearing uniqueness theorem; it does not raise the circularity score. The main reservation is the shared-judge measurement: Eq. (2) optimizes pϕ and pψ, and Section 4.1's coverage metric counts cells defined by argmax of those same pϕ and pψ. This makes the coverage claim partly an in-sample objective rather than an independent diversity measure. The paper's own Section 5 limitation (reliance on closed-source GPT-4.1 for style evaluation) reinforces this concern. Because toxicity-based QD-Score, the archive profile, and transfer results provide independent evidence, the circularity is partial and the central effectiveness claim survives; score 4 rather than 6 or higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The main uncharged inputs are the behavior taxonomy, the judge models, and the hand-chosen hyperparameters; the judge-model outputs are particularly load-bearing because they appear simultaneously in the training reward, the replay-buffer archive, and the evaluation metrics.

free parameters (4)
  • Number of attacker models = 4
    Chosen by hand; balances compute, parallelism, and coverage. The paper does not report a sensitivity sweep over N.
  • Replay buffer capacity = 5000
    Chosen by hand and listed in Table 4; affects the balance between quality and diversity of stored attack data.
  • Steps between behavior reassignment = 400
    Chosen by hand; controls how often the adaptive style assignment is recomputed and affects exploration versus exploitation.
  • Training steps = 5000
    Fixed budget for all methods; the reported QD-Score and coverage are the terminal values at this step.
assumptions (5)
  • domain assumption The 14 risk categories from Llama-Guard-3 and the 11 attack styles from Rainbow Teaming form a meaningful and complete behavior space for red-teaming diversity.
    Invoked in Section 3.1 and used to define the archive grid; if this taxonomy misses important attack strategies, high coverage of this grid does not imply comprehensive red-teaming.
  • domain assumption The attack-style judge models (Llama-3.2-3B or GPT-4.1) assign correct styles with usable token probabilities p_psi(s|x).
    Used in Eq. (2) as reward and in Algorithm 2 and Section 4.1 as evaluation; no ground-truth or human validation of style labels is provided.
  • domain assumption Llama-Guard-3 toxicity scores r(x,y) are a valid proxy for safety violation severity.
    Used as the quality component in QD-Score and in Eq. (2); the paper does not validate against human harm judgments.
  • domain assumption MAP-Elites archive metrics QD-Score and coverage measure real diversity and attack quality.
    The metrics are computed by inserting prompts into cells assigned by the same judge models used during training, so they partly measure agreement with that judge.
  • standard math GFlowNets and the RL update behave as described in the cited prior work.
    QDRT relies on GFlowNets as the underlying RL algorithm and does not re-derive its convergence properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models." pith.science (2026). https://pith.science/paper/4OYB4FZ4

@misc{pith2026250607121,
  author       = {Pith},
  title        = {Pith review of: Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OYB4FZ4}},
  note         = {Machine review of arXiv:2506.07121}
}
read the original abstract

Ensuring the safety and robustness of large language models (LLMs) is a fundamental challenge and a critical prerequisite for the responsible deployment of artificial intelligence. Red-teaming, a systematic framework to identify adversarial prompts that elicit harmful responses from target LLMs, has emerged as a crucial safety evaluation paradigm. Within this framework, the diversity of adversarial prompts is critical for a comprehensive safety assessment. However, previous red-teaming approaches often pursue diversity through simplistic metrics such as word frequency or sentence embedding similarity, which may not capture meaningful variation in attack strategies. In addition, the common practice of training a single attacker model restricts coverage across all potential attack styles and risk categories. This paper introduces Quality-Diversity Red-Teaming (QDRT), a new framework designed to address these limitations. QDRT achieves goal-driven diversity through behavior-conditioned training and implements a behavioral replay buffer in an open-ended manner. In addition, it trains multiple specialized attackers capable of generating high-quality attacks across diverse styles and risk categories. Our empirical evaluation demonstrates that QDRT generates attacks that are both more diverse and more effective against a wide range of target LLMs, including open-source models \texttt{GPT-2}, \texttt{Llama-3}, \texttt{Gemma-2}, \texttt{Qwen2.5}, and commercial models \texttt{GPT-4.1} and \texttt{GPT-5-Chat}. This work advances the field of LLM safety by providing a systematic and effective approach to automated red-teaming, ultimately supporting the responsible deployment of LLMs. Our code is available at https://github.com/lamda-bbo/QDRT.

Figures

Figures reproduced from arXiv: 2506.07121 by the authors.

Figure 1
Figure 1. The example archives of raw benchmark data (left), augmented data (mid), and our [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different ablations of QDRT in terms of QD-Score and Coverage on three [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the prompt archive obtained by different methods on three target models. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance comparison of prompts generated in the training process in terms of QD-Score [PITH_FULL_IMAGE:figures/full_fig_p028_4.png]
Figure 5
Figure 5. Figure 5: QDRT exhibits the slowest decay in the percentage of prompts as the threshold increases, [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]
Figure 5
Figure 5. Figure 5: The archive profile of different methods on three target models with [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the prompt archive obtained by AdvBench, data augmentation, GEM, and [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Visualization of the prompt archive obtained by AdvBench, data augmentation, GEM, and [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the prompt archive obtained by AdvBench, data augmentation, GEM, and [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]
Figure 9
Figure 9. Figure 9: We collect the prompts generated for Gemma-2-9B by different methods and evaluate them with Gemma-2-27B, Qwen2.5-32B, and Llama-3.3-70B, and compare their QD-Scores and coverage. The x-value of each point is the coverage of the corresponding method, and the y-value of …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 32 canonical work pages

  1. [1]

    Bäck.Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms

    T. Bäck.Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press, 1996

    work page 1996

  2. [2]

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  3. [3]

    Bengio, M

    E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y . Bengio. Flow network based genera- tive models for non-iterative diverse candidate generation.Advances in Neural Information Processing Systems, 34:27381–27394, 2021

    work page 2021

  4. [4]

    Beutel, K

    A. Beutel, K. Xiao, J. Heidecke, and L. Weng. Diverse and effective red teaming with auto- generated rewards and multi-step reinforcement learning.arXiv preprint arXiv:2412.18693, 2024

    arXiv 2024

  5. [5]

    Bhatt, B

    V . Bhatt, B. Tjanaka, M. C. Fontaine, and S. Nikolaidis. Deep surrogate assisted generation of environments. InAdvances in Neural Information Processing Systems 35 (NeurIPS), New Orleans, LA, 2022

    work page 2022

  6. [6]

    Bianchi, M

    F. Bianchi, M. Suzgun, G. Attanasio, P. Rottger, D. Jurafsky, T. Hashimoto, and J. Zou. Safety- tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. InProceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 2024

    work page 2024

  7. [7]

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2023

    arXiv 2023

  8. [8]

    Chatzilygeroudis, A

    K. Chatzilygeroudis, A. Cully, V . Vassiliades, and J.-B. Mouret. Quality-diversity optimization: A novel branch of stochastic optimization. InBlack Box Optimization, Machine Learning, and No-Free Lunch Theorems, pages 109–135. Springer, 2021. 10

    work page 2021

Show all 58 references
  1. [9]

    Cully, J

    A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret. Robots that can adapt like animals.Nature, 521(7553):503–507, 2015

    2015

  2. [10]

    Cully and Y

    A. Cully and Y . Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245–259, 2018

    2018

  3. [11]

    Dathathri, A

    S. Dathathri, A. Madotto, J. Lan, J. H. ung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach to controlled text generation. InInternational Conference on Learning Representations, 2020

    2020

  4. [12]

    Dinan, S

    E. Dinan, S. Humeau, B. Chintagunta, and J. Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack.arXiv preprint arXiv:1908.06083, 2019

    1908 arXiv

  5. [13]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Ecoffet, J

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 590(7847):580–586, 2021

    2021

  7. [15]

    Flageat, B

    M. Flageat, B. Lim, L. Grillotti, M. Allard, S. C. Smith, and A. Cully. Benchmarking quality- diversity algorithms on neuroevolution for reinforcement learning.arXiv:2211.02193, 2022

    2022 arXiv

  8. [16]

    M. C. Fontaine, R. Liu, A. Khalifa, J. Modi, J. Togelius, A. K. Hoover, and S. Nikolaidis. Illuminating mario scenes in the latent space of a generative adversarial network. InProceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), pages 5922–5930, Virtual, 2021

    2021

  9. [17]

    Ganguli, L

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  10. [18]

    V . T. Y . Han, R. Bhardwaj, and S. Poria. Ruby teaming: Improving quality diversity search with memory for automated red teaming.arXiv preprint arXiv:2406.11654, 2024

    2024 arXiv

  11. [19]

    Z.-W. Hong, I. Shenfeld, T.-H. Wang, Y .-S. Chuang, A. Pareja, J. R. Glass, A. Srivastava, and P. Agrawal. Curiosity-driven red-teaming for large language models. InProceedings of the 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 2024

    2024

  12. [20]

    Hughes, M

    E. Hughes, M. D. Dennis, J. Parker-Holder, F. Behbahani, A. Mavalankar, Y . Shi, T. Schaul, and T. Rocktäschel. Position: Open-endedness is essential for artificial superhuman intelligence. InProceedings of the 41st International Conference on Machine Learning (ICML), pages 20...

    2024

  13. [21]

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  14. [22]

    Kumar, A

    S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot extrapolation via structured MaxEnt RL. InAdvances in Neural Information Processing Systems 34 (NeurIPS), pages 8198–8210, Virtual, 2020

    2020

  15. [23]

    D. Lee, J. Lee, J.-W. Ha, J.-H. Kim, S.-W. Lee, H. Lee, and H. O. Song. Query-efficient black-box red teaming via bayesian optimization.arXiv preprint arXiv:2305.17444, 2023

    2023 arXiv

  16. [24]

    S. Lee, M. Kim, L. Cherif, D. Dobre, J. Lee, S. J. Hwang, K. Kawaguchi, G. Gidel, Y . Bengio, N. Malkin, et al. Learning diverse attacks on large language models for robust red-teaming and safety tuning.Proceedings of the 13th International Conference on Learning Representatio...

    2025

  17. [25]

    Lehman and K

    J. Lehman and K. O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. InProceedings of the 13th ACM Genetic and Evolutionary Computation Conference (GECCO), pages 211–218, Dublin, Ireland, 2011

    2011

  18. [26]

    Z. Li, C. Chen, T. Xu, Z. Qin, J. Xiao, Z.-Q. Luo, and R. Sun. Preserving diversity in supervised fine-tuning of large language models.Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. 11

    2025

  19. [27]

    Z. Li, T. Xu, Y . Zhang, Z. Lin, Y . Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. InForty-first International Conference on Machine Learning, 2024

    2024

  20. [28]

    L. Lin, H. Mu, Z. Zhai, M. Wang, Y . Wang, R. Wang, J. Gao, Y . Zhang, W. Che, T. Baldwin, et al. Against the achilles’ heel: A survey on red teaming for generative models.Journal of Artificial Intelligence Research, 82:687–775, 2025

    2025

  21. [29]

    X. Liu, P. Li, E. Suh, Y . V orobeychik, Z. Mao, S. Jha, P. McDaniel, H. Sun, B. Li, and C. Xiao. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms.arXiv preprint arXiv:2410.05295, 2024

    2024 arXiv

  22. [30]

    Y . Liu, S. Zhou, Y . Lu, H. Zhu, W. Wang, H. Lin, B. He, X. Han, and L. Sun. Auto-rt: Automatic jailbreak strategy exploration for red-teaming large language models.arXiv preprint arXiv:2501.01830, 2025

    2025 arXiv

  23. [31]

    J. Miao, T. Zhou, K. Shao, M. Zhou, W. Zhang, J. Hao, Y . Yu, and J. Wang. Promoting quality and diversity in population-based reinforcement learning via hierarchical trajectory space exploration. InProceedings of the 39th IEEE International Conference on Robotics and Automati...

    2022

  24. [32]

    Mouret and J

    J.-B. Mouret and J. Clune. Illuminating search spaces by mapping elites.arXiv:1504.04909, 2015

    2015 arXiv

  25. [33]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, et al. GPT-4 technical report. arxiv:2303.08774, 2023

    2023 arXiv

  26. [34]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructio...

    2022

  27. [35]

    T. D. Pala, V . Y . Toh, R. Bhardwaj, and S. Poria. Ferret: Faster and effective automated red teaming with reward-based scoring technique.arXiv:2408.10701, 2024

    2024 arXiv

  28. [36]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evalua- tion of machine translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics, page 311–318, Philadelphia, PA, 2002

    2002

  29. [37]

    Pavlova, E

    M. Pavlova, E. Brinkman, K. Iyer, V . Albiero, J. Bitton, H. Nguyen, J. Li, C. C. Ferrer, I. Evtimov, and A. Grattafiori. Automated red teaming with goat: the generative offensive agent tester.arXiv preprint arXiv:2410.01606, 2024

    arXiv

  30. [38]

    Perez, S

    E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab ...

    2022

  31. [39]

    Pierrot and A

    T. Pierrot and A. Flajolet. Evolving populations of diverse RL agents with MAP-elites. InThe 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023

    2023

  32. [40]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  33. [41]

    Samvelyan, S

    M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y . Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktaschel, and R. Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts.arXiv:2402.16822, 2024

    2024 arXiv

  34. [42]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  35. [43]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 12

    2024 arXiv

  36. [44]

    do anything now

    X. Shen, Z. Chen, M. Backes, Y . Shen, and Y . Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685, 2024

    2024

  37. [45]

    Z. Shi, Y . Wang, F. Yin, X. Chen, K. Chang, and C. Hsieh. Red teaming language model detectors with language models.Trans. Assoc. Comput. Linguistics, 12:174–189, 2024

    2024

  38. [46]

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  39. [47]

    Tjanaka, M

    B. Tjanaka, M. C. Fontaine, Y . Zhang, S. Sommerer, N. Dennler, and S. Nikolaidis. pyribs: A bare-bones python library for quality diversity optimization. https://github.com/ icaros-usc/pyribs, 2021

    2021

  40. [48]

    Tylkin, G

    P. Tylkin, G. Radanovic, and D. C. Parkes. Learning robust helpful behaviors in two-player cooperative atari environments. In F. Dignum, A. Lomuscio, U. Endriss, and A. Nowé, editors, Proceedings of the20th International Conference on Autonomous Agents and Multiagent Systems (...

    2021

  41. [49]

    K. Wang, G. Zhang, Z. Zhou, J. Wu, M. Yu, S. Zhao, C. Yin, J. Fu, Y . Yan, H. Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment.arXiv preprint arXiv:2504.15585, 2025

    2025 arXiv

  42. [50]

    A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023

    2023

  43. [51]

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek. A comprehensive study of jailbreak attack versus defense for large language models.arXiv preprint arXiv:2402.13457, 2024

    2024 arXiv

  44. [52]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  45. [53]

    S. Yi, Y . Liu, Z. Sun, T. Cong, X. He, J. Song, K. Xu, and Q. Li. Jailbreak attacks and defenses against large language models: A survey.arXiv preprint arXiv:2407.04295, 2024

    2024 arXiv

  46. [54]

    Zhang, M

    Y . Zhang, M. C. Fontaine, V . Bhatt, S. Nikolaidis, and J. Li. Multi-robot coordination and layout design for automated warehousing. InProceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI), pages 5503–5511, Macao, SAR, China, 2023

    2023

  47. [55]

    Z.-H. Zhou, Y . Yu, and C. Qian.Evolutionary Learning: Advances in Theories and Algorithms. Springer, 2019

    2019

  48. [56]

    Y . Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y . Yu. Texygen: A benchmarking platform for text generation models. InProceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR), pages 1097–1100, Ann Arbor, MI, 2018

    2018

  49. [57]

    Limitations

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 13 NeurIPS Paper Checklist 1.Claims Question: Do the main claims made in the abstract and int...

    2023 arXiv

  50. [58]

    revolu- tionary

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.