Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single language model that alternates between attacker and defender in an online self-play game can reduce attack success rates by up to 95% while preserving general ability.

desk verdict A strong empirical self-play safety paper whose theoretical guarantee overclaims; fix the reward-proof mismatch and the benchmark counts and it deserves publication. read the letter →

arxiv 2506.07468 v4 pith:SL4E4M2V submitted 2025-06-09 cs.LG cs.CLcs.MA

Mickel Liu , Liwei Jiang , Yancheng Liang , Simon Shaolei Du , Yejin Choi , Tim Althoff , Natasha Jaques This is my paper
classification cs.LGcs.CLcs.MA
keywords self-playreinforcementlearningmulti-agentRLLLMsafetyalignmentred-teamingzero-sumgameNashequilibriumhiddenchain-of-thoughtadversarialrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Self-RedTeam, a fully online self-play reinforcement learning method in which a single language model alternately plays attacker and defender in a zero-sum safety game while a reward model judges each exchange. The aim is to replace the reactive loop where attackers exploit a static model and defenders patch discovered holes, a loop that lets attackers overfit to obsolete exploits and leaves defenders perpetually behind. The paper's theoretical result says that if the attacker and defender policies converge to a Nash equilibrium, the defender's response to any prompt is safe, meaning it receives a non-negative reward from the safety judge. Empirically, the method reduces attack success rates by up to 95% on safety benchmarks and produces 17.8% more diverse attacks than training an attacker against a fixed defender, without appreciable loss of general capability. The reader should care because this points toward a way for models to keep hardening themselves as new attacks appear rather than waiting for the next patch.

What carries the argument

The load-bearing object is the zero-sum red-teaming game between attacker and defender, with reward $r_\theta(y_A, y_D) \in [-1,1]$ and a shared LLM policy playing both roles. The payoff is decomposed into response-harmfulness and refusal sub-rewards plus format and revision-faithfulness shaping terms, and hidden chain-of-thought keeps each role's private reasoning out of the opponent's observation. The safety argument runs through the minimax structure: an always-refusing defender anchors the game value at zero, so at a Nash equilibrium no attacker can find a prompt worth negative reward. The same shared policy is trained online with a critic-free RL update that normalizes attacker and defender advantages separately, allowing conflicting role rewards to update one parameter set.

What would settle it

Take a final Self-RedTeam checkpoint, freeze the defender, and let the trained attacker plus a prompt search look for any prompt that makes the reward model score the response negative. If such a prompt is found, the checkpoint is not at the Nash equilibrium that Theorem 1 requires, or the reward model does not give the always-refuse strategy zero reward, so the paper's safety guarantee does not cover that model.

Watch

Extended reading notes

Core claim

The central claim is that LLM safety alignment can be modeled as a two-player zero-sum game and trained online by one policy that self-plays both roles, with a single reward model rating query harmfulness, response harmfulness, and refusal. Theorem 1 states that when the two policies converge to a Nash equilibrium $\pi^*_A, \pi^*_D$, for any prompt $y_A$ the defender's response satisfies $r_\theta(y_A, \pi^*_D(y_A)) \ge 0$, i.e., the response is safe as judged by the reward model. The proof uses contradiction: if some prompt earned $-1$ against the equilibrium defender, an attacker that always plays that prompt would force the game value to $-1$, while a defender that always refuses would guarantee a non-negative value, contradicting equilibrium. The empirical package, called Self-RedTeam, alternates the shared policy between attacker and defender using hidden chain-of-thought, optimizes with an online RL algorithm, and reports up to 95% reduction in attack success rate across safety benchmarks and higher attack diversity than static-defender training.

Load-bearing premise

The safety guarantee assumes a defender that simply refuses everything scores zero or non-negative on every prompt, but the training reward actually penalizes refusal on benign prompts, so the game being trained is not exactly the game that the theorem proves safe.

Editorial extensions

If this is right

  • If the self-play game reaches a Nash equilibrium, the defender is guaranteed to respond safely to any adversarial prompt, because no prompt can be worth negative reward at equilibrium.
  • Co-evolving the attacker and defender keeps attack generation diverse over training, whereas an attacker trained against a static defender collapses onto a few dominant attack modes.
  • Fine-tuning with Self-RedTeam reduces attack success rates by up to 95% on safety benchmarks relative to RLHF-trained base models while roughly preserving instruction-following, reasoning, and knowledge benchmarks.
  • The self-play training is only about 44-48% more expensive than defender-only training, roughly matching the 50% increase in samples from online prompt generation.
  • Robustness learned in single-turn self-play transfers to multi-turn jailbreak attacks, improving multi-turn attack success rate by 9.09% to 44.7% across the tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the safety guarantee inherits the reward model's notion of safety; if the judge fails to flag a harmful response, a Nash equilibrium of the game can still count as safe by the theorem yet be harmful in fact.
  • Editorial inference: because the implemented reward penalizes refusal of benign queries, the theorem's premise of zero reward for always refusing is not met during training, so the trained defender could settle at a trade-off that preserves the empirical safety gains without literally satisfying the theorem's conditions.
  • Editorial inference: shared-parameter self-play can create an internal brake, since the attacker occasionally refuses to generate attacks; the paper observes this emergent behavior in one checkpoint but does not analyze its mechanism, and separate attacker/defender parameters would likely behave differently.
  • Editorial inference: the same online co-evolution recipe could be lifted from safety to any judgeable quality, such as factuality, style, or instruction following, wherever a generator and a critic can be arranged as zero-sum players.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Self-RedTeam, an online self-play MARL algorithm for LLM safety in which a single policy alternates between attacker and defender roles, uses hidden chain-of-thought for strategic planning, and is trained with the Re++ algorithm using rewards provided by WildGuard. It claims a game-theoretic safety guarantee (Theorem 1) that a Nash equilibrium defender is safe on every prompt, and it reports large attack-success-rate reductions across Qwen and Llama models, increased attack diversity, and modest capability loss.

Significance. If the empirical results hold, the method is a meaningful step toward online, adaptive safety training: the benchmark suite is broad, results are averaged over three runs with confidence intervals, ablations are systematic, multi-turn transfer is examined, and code is released. However, the theoretical safety guarantee is not supported for the implemented reward, and the use of WildGuard as both the training reward and one of the headline evaluation benchmarks weakens the strongest empirical claims. These issues are fixable without invalidating the main empirical comparison, but they need to be addressed before publication.

major comments (3)
  1. [§3 / Appendix B / §4.1] Theorem 1's proof relies on a 'naive rejecting defender' π0D that receives non-negative reward on every prompt, but the implemented reward in §4.1 gives RD,res_refusal = -rrefusal whenever Qharm(yA)=0 and Resrefuse(yD)=1, i.e., refusing a benign prompt is explicitly penalized. Since §5 uses a 50:50 harmful/benign RL prompt mixture, an always-refuse defender gets payoff rharm - rrefusal on benign prompts, which is negative whenever the refusal penalty exceeds the harmfulness reward; the proof needs per-prompt non-negativity, not just a favorable average. The paper implicitly concedes this in §4 by including benign seeds 'to prevent a trivial defender's strategy where the defender simply refuses all prompts.' Additionally, the proof quantifies over arbitrary prompts y0_A and treats πD(yA) as a deterministic response, whereas the policy is stochastic and the attacker's strategy space is not specified to include point-mass policies; a Nash equilibrium only gives non-negativity of expected value, not a pathwise safety guarantee. Please either prove the theorem for the actual training reward, state it for an idealized reward and clearly label the connection as motivational, or soften the abstract's claim that a Nash equilibrium of Self-RedTeam guarantees safe responses.
  2. [§5 / §5.1] The reward model used for training is WildGuard-7B (§5), and the headline evaluation benchmark WG:Test comes from the same WildGuard family (§5.1). Because the training objective directly optimizes the classifier that defines one of the main evaluation metrics, the reported WG:Test improvements may partly reflect reward overfitting rather than general safety improvement. Please report results for at least one model with an independent judge (e.g., GPT-4o or a different safety classifier), or show explicitly that the main conclusions are unchanged when WG:Test is removed from the summary statistics.
  3. [§4.1 / Appendix A.2] The numerical values of rharm, rrefusal, rformat, and rrevision are never reported; Appendix A.2 gives only the KL coefficient β=0.01. These weights determine whether the always-refuse strategy has non-negative value (and hence whether the theorem's premise can hold), and they are necessary for reproducibility of the training pipeline. Please report them and, if they vary by model, give the full hyperparameter table.
minor comments (5)
  1. [Abstract / §1 / §5.1] The abstract says 'across 14 benchmarks,' while the introduction and §5.1 describe 12 evaluations; please reconcile the count and ensure the numbers in the summary statistics match the tables.
  2. [Abstract] The sentence 'We not only uncovering more diverse attacks' is grammatically incomplete; it should be 'We not only uncover more diverse attacks...'.
  3. [§2] The sentence 'SELF-REDTEAMintroduces leverages the power of RL fine-tuning' appears to have a missing word; it should read 'introduces and leverages' or similar.
  4. [Appendix E] The appendix title 'Limitaions' is a typo and should be 'Limitations'.
  5. [Table 1 / Table 2] The column headers are visually garbled (e.g., 'HarmBenchOR-BenchXSTestStrongREJECTWJB XSTestAlpacaEval 2'), and several abbreviations such as RTA, AB, VB, and LC are not defined in the table caption. Please reformat the headers and define all abbreviations.

Circularity Check

2 steps flagged · score 6.0 of 10

The theorem's 'safety guarantee' defines safety as non-negative reward from the same reward model used for training, and the headline WG:Test evaluation is scored by the same WildGuard judge family, so the central theoretical claim is self-definitional and part of the empirical gain is forced by construction.

  1. self definitional [Section 3 (Theoretical Safety Guarantees) and Appendix B, Theorem 1]
    "Under the red-teaming game setting specifically, the reward will either be rθ(yA, yD) =−1 when the response is unsafe, or the reward is non-negative rθ(yA, yD)≥0 , indicating a safe response. ... Theorem 1. When the two players’ policies converge to a Nash Equilibrium (π∗ A, π∗ D), it can be shown that for any prompt y A, r θ(yA, π∗ D(yA))≥0, i.e., the response is safe."

    The theorem's conclusion 'the response is safe' is, one sentence earlier, stipulated to mean rθ≥0. Since rθ is precisely the reward the defender is trained to maximize, the claimed first-principles safety guarantee reduces to the statement that a Nash equilibrium of the reward game attains non-negative reward on every prompt. The proof's contradiction only rules out prompts on which the reward model would assign −1, so 'safe' and 'non-negative reward-model score' are the same predicate by construction. The property predicted is the sign of the training objective, not an independently defined safety outcome.

  2. fitted input called prediction [Section 5 (Experiment), Section 5.1 (Evaluation), Table 1]
    "For our reward model, we use WildGuard-7B as the judge model, where it provides the three labels (query harmfulness, response harmfulness, response refusal) described in our red-teaming game setup (§ 4.1). ... Safety evaluation. We primarily adopt the WildGuard safety evaluation suite (Han et al., 2024) ... WILDGUARDTEST (Han et al., 2024), as an evaluation component of the broader WildGuard safety framework, serves to assess the effectiveness of LLM safety guardrails..."

    The reward model supplying all three training labels (Qharm, Resharm, Resrefuse) is WildGuard-7B, and the safety evaluation is said to 'primarily adopt the WildGuard safety evaluation suite', with WG:Test as a headline benchmark in Table 1. The model is therefore trained to satisfy the same classifier family that scores the headline safety metric. Reported WG:Test ASR improvements are not an independent measurement of safety; they are a re-reading of the training reward's own judgments, so a component of the 'up to 95%' claim is forced by construction. Other benchmarks such as HarmBench and StrongREJECT remain more independent, but the headline WildGuard-based column is not.

full rationale

The minimax proof of Theorem 1 is internally valid under its stated reward model: if an always-refuse defender had non-negative value against every attacker, a Nash equilibrium cannot assign −1 to any prompt. The circularity lies in the labeling: 'safe' is stipulated to mean rθ≥0, making the guarantee a fact about the reward function rather than about safety independent of the judge. The main empirical contributions—attack diversity, capability preservation, and gains on external benchmarks such as HarmBench and StrongREJECT—are not forced by the training reward and retain independent content. No load-bearing self-citation chain was found; citations to WildGuard, WildTeaming, and related work are used as tools, not as uniqueness theorems. The noted discrepancy between Theorem 1's always-refuse assumption and the Section 4.1 refusal penalty (−rrefusal on benign prompts) is a correctness or applicability gap rather than a circular derivation, so it does not by itself raise the circularity score. However, because the theoretical 'safety guarantee' is definitionally the reward sign and the headline WG:Test evaluation shares its judge with the training reward, a moderate partial-circularity score is warranted.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central contribution is a training procedure, not a new physical entity. The free parameters are the reward weights and KL coefficient; their values are not fully disclosed. The key axioms are standard game theory plus the domain assumption that the classifier reward is faithful and the refusal behavior is neutral in the theorem, which the implemented reward violates.

free parameters (5)
  • r_harm (response harmfulness reward weight) = not specified in text
    Reward shaping coefficient in Section 4.1; chosen by hand, affects the balance between safety and helpfulness. Not reported in main text or appendix.
  • r_refusal (refusal reward weight) = not specified in text
    Controls the penalty or reward for refusing harmful vs benign prompts; central to the defender's behavior.
  • r_format (CoT format reward weight) = not specified
    Reward for emitting the required think/answer tags; hand-set.
  • r_revision (revision faithfulness reward weight) = not specified
    Reward for preserving harm/benign classification of seed in revised prompt; hand-set.
  • KL coefficient beta = 0.01
    Appendix A.2; token-level KL penalty from reference policy.
assumptions (5)
  • standard math Two-player zero-sum game value and Nash equilibrium existence in the prompt-response game
    Section 3 relies on von Neumann and Nash results; accepted game theory.
  • domain assumption The reward model rθ correctly maps unsafe responses to -1 and all safe responses to a non-negative value
    Section 3 and Appendix B; if the classifier mislabels, the 'safety guarantee' is vacuous.
  • ad hoc to paper A naive always-refuse policy is a legal defender policy and receives non-negative reward on all prompts
    This assumption is stated in Section 3 but is contradicted by the implemented reward in Section 4.1, where refusal on benign prompts is penalized.
  • domain assumption Training converges to a Nash equilibrium of the idealized game
    The theorem is conditional on convergence; the paper acknowledges exact NE is challenging, so the empirical link is motivational.
  • domain assumption Hidden chain-of-thought is learnable and provides strategic benefit
    The method's design assumes the model can produce private reasoning that improves attack and defense quality; the paper provides indirect evidence via ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models." pith.science (2026). https://pith.science/paper/SL4E4M2V

@misc{pith2026250607468,
  author       = {Pith},
  title        = {Pith review of: Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SL4E4M2V}},
  note         = {Machine review of arXiv:2506.07468}
}
read the original abstract

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL. Link to code: https://github.com/mickelliu/selfplay-redteaming

Figures

Figures reproduced from arXiv: 2506.07468 by the authors.

Figure 1
Figure 1. Proposed SELF-REDTEAM framework, in which an LLM plays a red-teaming game by defending against its own generated attacks. The process initiates with the shared LLM policy playing the role of the attacker and receiving a seed prompt (a). This is privately refined into an adversarial attack (yA) using a hidden chain-of-thought process (y CoT A ) invisible to the opponent (b). The attack is then passed to the defender … view at source ↗
Figure 2
Figure 2. (CAUTION: Offensive and Derogatory Language) t-SNE visualization of SBERT embed￾dings for adversarial attacks generated by Self-Play and Attacker-Only methods, based on 1000 distinct seed prompts. The spatial distribution illustrates semantic clustering of the generated attack vectors. Notably, the Attacker-Only method tends to utilize similar attack patterns even with different seed prompts and their varied locatio… view at source ↗
Figure 3
Figure 3. Training metrics. (a, e) Generated Attacks diversity evaluated on a holdout set during training. (b, c, d) Attacker performance metrics for generated attacks. (f, g) Defender performance metrics against attack instances. (h) Average CoT template violation rate. Results show means over 3 runs with 95% confidence intervals (shaded). See § 6 for in-depth analysis of the diagrams. in supervised fine-tuning (SFT) updates… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Schematic diagram illustrating the self-distillation procedure for generating the SFT dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Bootstrapped distributions of evaluation performance across five benchmarks, finetun [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: DBSCAN Clustering of t-SNE embeddings of the generated attacks used in Figure [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safety Alignment of LMs via Non-cooperative Games

    cs.AI 2025-12 conditional novelty 7.0 of 10

    Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.

  2. ProbeLLM: Automating Principled Diagnosis of LLM Failures

    cs.CL 2026-02 conditional novelty 6.0 of 10

    An automated LLM probing pipeline using hierarchical Monte Carlo Tree Search, tool-based verification, and failure clustering to discover structured failure modes beyond static benchmarks.

  3. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

Reference graph

Works this paper leans on

87 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740

    arXiv 2024

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 a

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    2022

  3. [3]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022 b

    arXiv 2022

  4. [4]

    Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=gT5hALch9z

    2024

  5. [5]

    Explore, establish, exploit: Red teaming language models from scratch, 2023

    Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch, 2023

    2023

  6. [6]

    Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system, 2025

    Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system, 2025. URL https://arxiv.org/abs/2410.08115

    arXiv 2025

  7. [7]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024

    arXiv 2024

  8. [8]

    Self-playing adversarial language game enhances llm reasoning, 2025

    Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Zheng Yuan, Yong Dai, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances llm reasoning, 2025. URL https://arxiv.org/abs/2404.10642

    arXiv 2025

Show all 87 references
  1. [9]

    Think you have solved question answering? try ARC , the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457

    2018 arXiv

  2. [10]

    Or-bench: An over-refusal benchmark for large language models, 2024

    Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models, 2024

    2024

  3. [11]

    Safe rlhf: Safe reinforcement learning from human feedback, 2023

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023

    2023

  4. [12]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=TyFrPOKYXw

    2024

  5. [13]

    Duoguard: A two-player rl-driven framework for multilingual llm guardrails, 2025

    Yihe Deng, Yu Yang, Junkai Zhang, Wei Wang, and Bo Li. Duoguard: A two-player rl-driven framework for multilingual llm guardrails, 2025. URL https://arxiv.org/abs/2502.05163

    2025

  6. [14]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 0 10088--10115, 2023

    2023

  7. [15]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  8. [16]

    Google’s gemini tops apple’s app store, snagging lead spot from chatgpt

    Jennifer Elias. Google’s gemini tops apple’s app store, snagging lead spot from chatgpt. CNBC, 2025. URL https://www.cnbc.com/2025/09/16/google-gemini-tops-apples-app-store-snagging-lead-spot-from-chatgpt.html

    2025

  9. [17]

    Kto: Model alignment as prospect theoretic optimization, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https://arxiv.org/abs/2402.01306

    2024 arXiv

  10. [18]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022 a

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfi...

    2022 arXiv

  11. [19]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022 b

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfi...

    2022

  12. [20]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  13. [21]

    Olmes: A standard for language model evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hanna Hajishirzi. Olmes: A standard for language model evaluations. ArXiv, abs/2406.08446, 2024. URL https://api.semanticscholar.org/CorpusID:270391754

    2024 arXiv

  14. [23]

    Elon musk's ai chatbot, grok, started calling itself 'mechahitler'

    Lisa Hagen, Huo Jingnan, and Audrey Nguyen. Elon musk's ai chatbot, grok, started calling itself 'mechahitler'. NPR, July 2025. URL https://www.npr.org/2025/07/09/nx-s1-5462609/grok-elon-musk-antisemitic-racist-content

    2025

  15. [24]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495, 2024

    2024 arXiv

  16. [25]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  17. [26]

    Curiosity-driven red-teaming for large language models

    Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:2402.19464, 2024

    2024 arXiv

  18. [27]

    Scaling trends in language model robustness

    Nikolaus Howe, Ian McKenzie, Oskar Hollinsworth, Micha Zajac, Tom Tseng, Aaron Tucker, Pierre-Luc Bacon, and Adam Gleave. Scaling trends in language model robustness. arXiv preprint arXiv:2407.18213, 2024

    2024 arXiv

  19. [28]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025 a

    Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025 a . URL https://arxiv.org/abs/2501.03262

    2025 arXiv

  20. [29]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025 b

    2025 arXiv

  21. [30]

    Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

    2023

  22. [31]

    Baseline defenses for adversarial attacks against aligned language models

    N Jain, A Schwarzschild, Y Wen, G Somepalli, J Kirchenbauer, PY Chiang, M Goldblum, A Saha, J Geiping, and T Goldstein. Baseline defenses for adversarial attacks against aligned language models. 2023. arXiv preprint arXiv:2309.00614, 2023

    2023 arXiv

  23. [32]

    Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control

    Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jos \'e Miguel Hern \'a ndez-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pp.\ 1645--1654. PMLR, 2017

    2017

  24. [33]

    Way off-policy batch deep reinforcement learning of implicit human preferences in dialog

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019

    1907 arXiv

  25. [34]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [35]

    Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Pro...

    2024

  27. [36]

    Predicting vs

    Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling. CoRR, 2024 a

    2024

  28. [37]

    Deepinception: Hypnotize large language model to be jailbreaker, 2024 b

    Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker, 2024 b

    2024

  29. [38]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval : An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023

    2023

  30. [39]

    Marft: Multi-agent reinforcement fine-tuning, 2025

    Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. Marft: Multi-agent reinforcement fine-tuning, 2025. URL https://arxiv.org/abs/2504.16129

    2025 arXiv

  31. [40]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3214--3252, 2022

    2022

  32. [41]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  33. [42]

    Evolving diverse red-team language models in multi-round multi-agent games, 2024 a

    Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, and Yaodong Yang. Evolving diverse red-team language models in multi-round multi-agent games, 2024 a . URL https://arxiv.org/abs/2310.00322

    2024 arXiv

  34. [43]

    Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning

    Hao Ma, Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 15497--15525, 2024 b

    2024

  35. [44]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024

    2024

  36. [45]

    Tree of attacks: Jailbreaking black-box llms automatically, 2024

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically, 2024

    2024

  37. [46]

    Confronting reward model overoptimization with constrained rlhf

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023

    2023 arXiv

  38. [47]

    Equilibrium points in n-person games

    John F Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36 0 (1): 0 48--49, 1950

    1950

  39. [48]

    DAN (do anything now): A jailbreaking prompt technique, 2023

    Community Origin. DAN (do anything now): A jailbreaking prompt technique, 2023. Widely known jailbreaking technique for LLMs, specific origin often traced to online forums/discussions around late 2022-early 2023

    2023

  40. [49]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  41. [50]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  42. [51]

    Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning

    Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv preprint arXiv:2502.18439, 2025

    2025 arXiv

  43. [53]

    Red teaming language models with language models, 2022 b

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022 b . URL https://arxiv.org/abs/2202.03286

    2022 arXiv

  44. [54]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  45. [56]

    X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents, 2025 b

    Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents, 2025 b . URL https://arxiv.org/abs/2504.13203

    2025 arXiv

  46. [57]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  47. [58]

    GPQA : A graduate-level google-proof q&a benchmark

    David Rein, Betty Li, Jackson H Lee, Jacob Steinhardt, and Dan Hendrycks. GPQA : A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  48. [59]

    XST est: A test suite for identifying exaggerated safety behaviours in large language models

    Paul R \"o ttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XST est: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024

  49. [60]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 20...

    2024 arXiv

  50. [61]

    Training language models for social deduction with multi-agent reinforcement learning

    Bidipta Sarkar, Warren Xia, C Karen Liu, and Dorsa Sadigh. Training language models for social deduction with multi-agent reinforcement learning. arXiv preprint arXiv:2502.06060, 2025

    2025 arXiv

  51. [62]

    Google’s Gemini headaches spur \ 90 billion selloff

    Derek Saul. Google’s Gemini headaches spur \ 90 billion selloff. Forbes, 2024. URL https://www.forbes.com/sites/dereksaul/2024/02/26/googles-gemini-headaches-spur-90-billion-selloff/

    2024

  52. [63]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  53. [65]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024 b

    2024 arXiv

  54. [66]

    Latent adversarial training improves robustness to persistent harmful behaviors in llms, 2025

    Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Latent adversarial training improves robustness to persistent harmful behaviors in llms, 2025. URL http...

    2025 arXiv

  55. [67]

    Ai effect openai’s chatgpt to hit 700 million weekly users, up 4x from last year

    MacKenzie Sigalos. Ai effect openai’s chatgpt to hit 700 million weekly users, up 4x from last year. CNBC, 2025. URL https://www.cnbc.com/2025/08/04/openai-chatgpt-700-million-users.html

    2025

  56. [68]

    A strongreject for empty jailbreaks

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongreject for empty jailbreaks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zha...

    2024

  57. [69]

    Multiagent finetuning: Self improvement with diverse reasoning chains

    Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arXiv preprint arXiv:2501.05707, 2025

    2025 arXiv

  58. [70]

    Preference fine-tuning of llms should leverage suboptimal, on-policy data

    Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367, 2024

    2024 arXiv

  59. [71]

    Game-theoretic regularized self-play alignment of large language models

    Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, and Ilija Bogunovic. Game-theoretic regularized self-play alignment of large language models. arXiv preprint arXiv:2503.00030, 2025

    2025 arXiv

  60. [72]

    Theory of games and economic behavior, 2nd rev

    John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 2nd rev. 1947

    1947

  61. [73]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024 a

    2024

  62. [74]

    Co-evolving llm coder and unit tester via reinforcement learning, 2025 a

    Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving llm coder and unit tester via reinforcement learning, 2025 a . URL https://arxiv.org/abs/2506.03136

    2025

  63. [75]

    Helpsteer2-preference: Complementing ratings with preferences, 2024 b

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preference: Complementing ratings with preferences, 2024 b . URL https://arxiv.org/abs/2410.01257

    2024 arXiv

  64. [76]

    Dedicated feedback and edit models empower inference-time scaling for open-ended general-domain tasks, 2025 b

    Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. Dedicated feedback and edit models empower inference-time scaling for open-ended general-domain tasks, 2025 b . URL https://arxiv.org/abs/2503.04378

    2025 arXiv

  65. [77]

    Jailbroken: How does llm safety training fail?, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023

    2023

  66. [78]

    Self-play preference optimization for language model alignment

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024

    2024 arXiv

  67. [79]

    Efficient adversarial training in llms with continuous attacks

    Sophie Xhonneux, Alessandro Sordoni, Stephan G \"u nnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks. Advances in Neural Information Processing Systems, 37: 0 1502--1530, 2024

    2024

  68. [80]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025. URL https://arxiv.org/abs/2502.14768

    2025 arXiv

  69. [81]

    Le, Qijun Tan, and Yuan Liu

    Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, and Yuan Liu. Scalable reinforcement post-training beyond static human prompts: Evolving alignment via asymmetric self-play, 2025. URL https://arxiv.org/abs/2411.00062

    2025 arXiv

  70. [82]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  71. [83]

    Absolute zero: Reinforced self-play reasoning with zero data, 2025

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data, 2025. URL https://arxiv.org/abs/2505.03335

    2025 arXiv

  72. [84]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  73. [85]

    Jordan, and Jiantao Jiao

    Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf, 2024. URL https://arxiv.org/abs/2401.16335

    2024 arXiv

  74. [86]

    Texygen: A benchmarking platform for text generation models

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp.\ 1097--1100, 2018

    2018

  75. [87]

    Improving alignment and robustness with circuit breakers, 2024

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers, 2024

    2024

  76. [88]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  77. [89]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  78. [90]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  79. [91]

    Red Teaming Language Models with Language Models

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.