Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AlphaAlign claims that a pure reinforcement learning setup with a verifiable safety reward and a normalized helpfulness reward can make LLMs refuse harmful queries and stop over-refusing, while keeping or improving utility, using only…

desk verdict A simple RLVR-style safety alignment recipe with real promise, but the safety metric is partly circular and the utility claims are overstated. read the letter →

arxiv 2507.14987 v1 pith:4553VIKC submitted 2025-07-20 cs.AI cs.CRcs.LG

Yi Zhang , An Zhang , XiuYu Zhang , Leheng Sheng , Yuxin Chen , Zhenkai Liang , Xiang Wang This is my paper
classification cs.AIcs.CRcs.LG
keywords safetyalignmentreinforcementlearningwithverifiablerewardsrefusaltrainingjailbreakrobustnessover-refusalsafety-utilitytrade-offproactivereasoninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AlphaAlign tries to establish that a large language model's latent safety understanding can be turned into reliable safe behavior by pure reinforcement learning, without any supervised safety reasoning data. It does this with a verifiable safety reward that checks only whether the final answer is a refusal, plus a normalized helpfulness reward that protects quality on benign prompts. The paper reports that across four backbones this recipe improves refusal of harmful and jailbroken queries, lowers over-refusal, and preserves or improves instruction-following and mathematical reasoning within fewer than 200 RL steps. If true, safety alignment would need only binary prompt labels and a small verifier, and the usual safety-versus-utility trade-off would not be a fixed cost of alignment.

What carries the argument

The load-bearing object is the dual-component reward. The verifiable safety reward $R_s$ (Eq. 4) combines a format verifier, which checks that the output has `<safety_reasoning>` and `<answer>` tags, with a refusal verifier $V_r$, which checks the final answer against a canonical phrase list; it gives positive credit for refusing labeled harmful prompts and negative credit for refusing labeled benign ones. The normalized helpfulness reward $R_h$ (Eq. 5) applies only to benign prompts, scores each answer against the batch mean, and returns positive reward only for above-average non-refusal answers while returning zero to refusals. This separates what is rewarded (the safety label match of the final answer) from how the model reasons, so the reasoning trajectory is free to become whatever the model's latent safety awareness supports.

What would settle it

Take a model trained by AlphaAlign and attack it with jailbreaks that instruct it to answer harmfully while explicitly avoiding all canonical refusal phrases, then score the outputs with a held-out judge that was never used in training; if harmful content appears despite near-zero phrase-based ASR, the safety gain is an artifact of the verifier rather than latent safety awareness.

Watch

Extended reading notes

Core claim

The central claim is that a structurally prompted model can discover proactive safety reasoning when PPO maximizes a reward that verifies only the final answer: positive for refusal of labeled harmful prompts, negative for refusal of labeled benign prompts, plus a GRPO-normalized helpfulness component that rewards above-average non-refusal answers to benign prompts. Nothing in the reward specifies how to reason; the model must find its own path to the correct answer format. The paper reports that the resulting policies generate explicit safety rationales, refuse harmful and jailbreak queries, stop over-refusing, and hold or improve utility, all in under 200 steps and without safety-specific SFT.

Load-bearing premise

The load-bearing premise is that the hand-written refusal-phrase list used by verifier $V_r$ faithfully identifies safe refusals, because that same verifier provides the training reward and the first stage of the ASR evaluation.

Editorial extensions

If this is right

  • Safety alignment can be driven by binary prompt labels and a small verifier, with no curated refusal demonstrations or safety reasoning data.
  • Refusal of harmful and jailbroken prompts can improve at the same time as over-refusal falls, so the safety-utility trade-off is not forced by the RL objective.
  • A base model without instruction tuning can acquire substantial safety behavior from the safety reward alone, as shown by AlphaAlign-Zero.
  • Fewer than 200 RL steps with 8 rollouts per prompt suffice for substantial safety gains, making the alignment loop short enough to iterate on.
  • The presence of safety-relevant keywords and reduced jailbreak-trigger tokens in early generation indicates the learned behavior is deeper than memorized refusal patterns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension, flagged by the paper as unexplored, is replacing the string-matching refusal verifier with a learned safety judge; the training signal would then track genuine harmfulness rather than phrase matching.
  • The same conditional-reward structure could be applied to other binary properties of model output, such as factual consistency or format compliance, by swapping in the corresponding verifier.
  • Because AlphaAlign-Zero works on a base model, the recipe might shorten conventional post-training pipelines that start from instruction-tuned models, if the result carries to larger scales.
  • If the verifier's coverage is the true ceiling of the method, then iterating the RL loop against a growing set of refusal paraphrases should directly raise robustness to adaptive jailbreaks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes AlphaAlign, a pure reinforcement learning framework for LLM safety alignment. The method uses a dual reward: a verifiable safety reward that rewards correctly formatted and explicit refusals for harmful prompts and penalizes over-refusals for benign prompts, and a normalized helpfulness reward that encourages high-quality responses to benign queries. Training requires only binary prompt-safety labels and uses PPO with fewer than 200 steps. Experiments on Qwen2.5 and Llama3.2 backbones report improved refusal of harmful and jailbroken prompts, reduced over-refusal, maintained or improved utility, and evidence of proactive safety reasoning, compared with direct refusal, Circuit Breaker, and SCoT baselines.

Significance. If the empirical claims hold, AlphaAlign would be a simple and scalable safety-alignment recipe: it needs only binary labels, uses a verifiable reward, and reports convergence in fewer than 200 RL steps. The framework is clearly presented, the code is released, and the comparison against several existing alignment methods is useful. However, the central safety evaluation shares its refusal verifier with the training reward, and the utility table contains an unexplained constant AlpacaEval score, so the headline claims are not yet established by the reported evidence.

major comments (4)
  1. [Section 4.1 and Eq. (4)] The ASR evaluation pipeline is not independent of the training reward. Eq. (4) rewards the model for matching the refusal verifier Vr on harmful prompts, and Section 4.1 states that ASR is 'evaluated by refusal verifier first, then by Llama3-Guard-8B.' Because Vr is a soft match against the handcrafted phrase list in Appendix B.1, Table 4, and the prompt template in Table 1 explicitly instructs the model to answer unsafe queries with \boxed{Sorry, I can't comply}, the low ASR numbers may reflect phrase-list overfitting rather than genuine harmfulness recognition. The authors should report ASR with Llama3-Guard-8B alone (or with an independent judge) on the final-answer content, and clarify whether Llama3-Guard is applied at all when Vr already classifies a response as a refusal.
  2. [Table 3] The AlpacaEval column is exactly 50.00 for all three backbones, while the parenthetical deltas are +6.7, +10.0, and +7.9. These two representations are inconsistent: if 50.00 is the AlphaAlign score, the baseline cannot be 50.00 with a nonzero delta; if 50.00 is the baseline reference score, the table does not report AlphaAlign's score. The authors should state explicitly which model is the AlpacaEval reference and provide the actual AlphaAlign win rates. As printed, this table does not support the claim that AlphaAlign maintains or improves instruction-following ability.
  3. [Sections 4.2-4.4] All safety and utility results are reported as single runs with no error bars, confidence intervals, or seed variation. Given that many ASR values are near zero (e.g., 0.0 in several columns of Table 2) and some utility deltas are small (e.g., -0.1 MMLU), the paper cannot distinguish a real effect from training noise. At least for one backbone, the authors should run multiple seeds and report the mean and variance.
  4. [Section 4.5 and Appendix C.5.1] The CKAS metric is a heuristic defined in this paper and is not validated against any human judgment or an established safety-reasoning metric. The claim that AlphaAlign achieves 'deep alignment via proactive safety reasoning' relies on CKAS and on qualitative case studies in Appendix D, which show canonical refusal phrases but not that the safety reasoning is causally responsible for the refusal. The authors should either validate CKAS against human annotations or provide an independent behavioral test of safety reasoning, for example by measuring whether the safety rationale survives paraphrasing or whether the model can explain its refusal in its own words.
minor comments (6)
  1. [Abstract] "We proposeAlphaAlign" is missing a space; should read "We propose AlphaAlign."
  2. [Section 4.1] The ASR description "evaluated by refusal verifier first, then by Llama3-Guard-8B" should be expanded to specify exactly how the two stages are combined, because the interpretation affects the validity of the reported safety numbers.
  3. [Appendix B.1, Table 4] The phrase list contains a typo: "I'\m sorry" should be "I'm sorry"; also "\u201cI'm sorry" appears to contain a stray escape.
  4. [Table 2] The column header "Jailbreaktrigger" is inconsistently formatted and should be "JailbreakTrigger" to match the benchmark name.
  5. [Section 3.2] The sentence "This avoids penalizing lower-quality but safe outputs with negative rewards" is slightly misleading: Eq. (5) assigns zero reward to non-refusal outputs below the mean, not a negative reward; the text can be clarified accordingly.
  6. [Section 4.2] Figure 2b reports curves for ASR, format compliance, and over-refusal, but the exact evaluation conditions for ASR (which verifier and which dataset) are not stated in the caption; this should be specified.

Circularity Check

1 steps flagged · score 6.0 of 10

The ASR headline metric reuses the same string-matching refusal verifier that the RL reward maximizes, so the reported safety gains are partially forced by construction rather than independently measured.

  1. fitted input called prediction [Section 3.2, Eq. (4); Section 4.1; Appendix B.1, Table 4; Table 1 prompt template]
    "AlphaAlign employs a verifier Vr to judge whether the final answery is a refusal by comparing it against a set of predefined refusal patterns observed in the model’s initial responses (i.e, “Sorry, I can’t comply”), more patterns in Appendix B.1. ... Rs(x, oi)= rf Vf (oi)+raVr(yi), x∈Xh; rf Vf (oi)−raVr(yi), x∈Xb. ... We employ the attack success rate (ASR) as the primary metric to measure its resilience against harmful and jailbreak attempts, evaluated by refusal verifier first, then by Llama3-Guard-8B."

    The reward in Eq. (4) is a function of Vr: harmful queries earn +ra Vr(y_i) and benign queries earn -ra Vr(y_i). Vr is a soft match against the canonical refusal phrase list in Table 4, which includes “Sorry, I can’t comply”, and the Table 1 template instructs the model to answer unsafe queries with “\boxed{Sorry, I can’t comply}”. The same Vr is then applied first in the ASR evaluation pipeline (Section 4.1) before Llama3-Guard. A policy that copies a listed phrase therefore receives training reward and is counted as safe on the headline metric; part of the reported ASR gain is the optimized policy matching the evaluation’s own first-stage string matcher rather than an independently measured refusal capability.

full rationale

AlphaAlign’s central safety claim is substantially supported by ASR numbers, and the ASR evaluation pipeline begins with the same refusal verifier Vr that the PPO reward maximizes in Eq. (4). Because Vr is a hand-crafted phrase-list matcher and the prompt template tells the model to answer unsafe queries with “\boxed{Sorry, I can’t comply}”, a large share of the measured safety improvement can be obtained by emitting one of the listed canonical strings. This is a genuine partial reduction of the reported prediction to the training signal. The paper’s own limitation statement—"we solely employ a binary safety label and a string-matching verifier to enhance the model’s awareness"—acknowledges the mechanism but does not remove the evaluation overlap. The CKAS metric in Appendix C.5.1 is an author-defined heuristic computed on the model’s own greedy path, so it cannot serve as independent evidence of “deep alignment”; it is better treated as a descriptive statistic of the trained behavior. The utility results (MMLU, AlpacaEval, GSM8K) and the Llama3-Guard second stage are external and are not defined by Vr, so the paper is not wholly circular. There is minor self-citation to prior An Zhang works in the background discussion of latent safety awareness, but those citations are not load-bearing for the main derivation. Overall, the safety prediction is partially forced by construction because the same verifier appears in both reward and evaluation, yielding a circularity score of 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on a small set of hand-made reward components and assumptions about latent safety ability. The refusal phrase list is a categorical free parameter, the reward weights rf/ra are not reported, and the equivalence between the training verifier and evaluation verifier drives the main circularity. No new physical or conceptual entities are introduced beyond a heuristic evaluation metric.

free parameters (3)
  • rf, ra reward weights = not reported
    Weights in the verifiable safety reward (Eq. 4) set the trade-off between format compliance and refusal correctness; their values are never given, so the exact training objective is under-specified.
  • refusal phrase list (Table 4) = 20+ hand-collected phrases
    The refusal verifier Vr is defined by a manually collected phrase set observed in base model outputs (Appendix B.1). This list determines both the training reward and part of the evaluation, so it is a free design choice that the results are sensitive to.
  • rollout group size n = 8
    The number of rollouts per prompt is set to 8 (Appendix C.1). It affects the GRPO-style normalization of helpfulness rewards and the variance of the advantage estimates.
assumptions (4)
  • domain assumption Pretrained LLMs possess latent safety awareness, an intrinsic capacity to distinguish harmful from benign content from pretraining.
    Section 1 and Appendix A.1 argue this from representation studies; AlphaAlign's entire premise is that this latent knowledge can be incentivized by a verifiable reward.
  • domain assumption Reinforcement learning with a binary verifiable reward (RLVR) incentivizes genuine reasoning rather than only token-level mimicry.
    Section 2.3 cites DeepSeek-R1 and Open-Reasoner-Zero for math/code reasoning; the paper extends this to safety without proof that the same mechanism operates for safety discrimination.
  • ad hoc to paper The hand-crafted refusal phrase list in Table 4 is a sufficient and correct proxy for a refusal.
    Appendix B.1 collects phrases from base model outputs; the reward (Eq. 4) and the ASR evaluation both rely on this list, creating a circular evaluation loop.
  • domain assumption The helpfulness reward model FsfairX-LLaMA3-RM-v0.1 provides valid scalar judgments of answer quality for benign prompts.
    Section C.1 uses this RM without validating it on the benign training distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning." pith.science (2026). https://pith.science/paper/4553VIKC

@misc{pith2026250714987,
  author       = {Pith},
  title        = {Pith review of: AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4553VIKC}},
  note         = {Machine review of arXiv:2507.14987}
}
read the original abstract

Large language models (LLMs), despite possessing latent safety understanding from their vast pretraining data, remain vulnerable to generating harmful content and exhibit issues such as over-refusal and utility degradation after safety alignment. Current safety alignment methods often result in superficial refusal shortcuts or rely on intensive supervision for reasoning-based approaches, failing to fully leverage the model's intrinsic safety self-awareness. We propose \textbf{AlphaAlign}, a simple yet effective pure reinforcement learning (RL) framework with verifiable safety reward designed to incentivize this latent safety awareness through proactive safety reasoning.} AlphaAlign employs a dual-reward system: a verifiable safety reward encourages correctly formatted and explicitly justified refusals for harmful queries while penalizing over-refusals, and a normalized helpfulness reward guides high-quality responses to benign inputs. This allows the model to develop proactive safety reasoning capabilities without depending on supervised safety-specific reasoning data. AlphaAlign demonstrates three key advantages: (1) Simplicity and efficiency, requiring only binary prompt safety labels and minimal RL steps for substantial improvements. (2) Breaking the safety-utility trade-off, by enhancing refusal of harmful content and reducing over-refusals, while simultaneously maintaining or even improving general task performance and robustness to unseen jailbreaks. (3) Deep alignment, fostering proactive safety reasoning that generates explicit safety rationales rather than relying on shallow refusal patterns.

Figures

Figures reproduced from arXiv: 2507.14987 by the authors.

Figure 1
Figure 1. Overview of AlphaAlign’s simple incentive mechanism. Through a pure Reinforcement Learning (RL) approach, our framework prompts the model to perform safety reasoning before providing a final answer. This process is guided by a dual-component reward applied to the final answer to incentivize latent safety awareness and maintain helpfulness: (1) a verifiable safety reward, which assesses whether the answer correctly c… view at source ↗
Figure 2
Figure 2. (2a) Safety and Utility comparison of Qwen2.5-3B + AlphaAlign-zero and Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Case study of how AlphaAlign-Zero reasoning about Malicious prompt with Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study on helpfulness reward 0 10 20 30 40 50 Token position where the keyword begins 0.0 0.2 0.4 0.6 0.8 1.0 1.2 CKAS here is illegal harm unethical dangerous (a) Qwen2.5-3B-Instruct’s CKAS 0 10 20 30 40 50 Token position where the keyword begins 0.0 0.2 0.4 0…
Figure 5
Figure 5. Figure 5: Cumulative Keyword Adoption Score (CKAS) evaluation under PAIR Chao et al. [2023] [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: A regular expression for validating format . [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Refusal Looks Safe: The Refusal-Cue Shortcut in Safety Guard Models

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Inserting a short refusal cue into a harmful response flips many safety guards' verdicts from harmful to unharmful, and targeted masking of a few internal components suppresses most of this failure.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claude 3.5 sonnet model card addendum

    Anthropic. Claude 3.5 sonnet model card addendum. Anthropic research, 2024

    work page 2024

  2. [2]

    Claude 3.7 sonnet system card

    Anthropic. Claude 3.7 sonnet system card. Anthropic research, 2025

    work page 2025

  3. [3]

    Morgane Rivi \` e re, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \' e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \' e , Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgi...

    arXiv 2024

  4. [4]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    arXiv 2024

  5. [5]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek - AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Hu...

    arXiv 2024

  6. [6]

    Hadi Amini, and Yanzhao Wu

    Badhan Chandra Das, M. Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey. ACM Comput. Surv. , 57 0 (6): 0 152:1--152:39, 2025

    work page 2025

  7. [7]

    Trustworthy llms: a survey and guideline for evaluating large language models' alignment

    Yang Liu, Yuanshun Yao, Jean - Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. CoRR, abs/2308.05374, 2023

    arXiv 2023

  8. [8]

    Safe + safe = unsafe? exploring how safe images can be exploited to jailbreak large vision-language models

    Chenhang Cui, Gelei Deng, An Zhang, Jingnan Zheng, Yicong Li, Lianli Gao, Tianwei Zhang, and Tat - Seng Chua. Safe + safe = unsafe? exploring how safe images can be exploited to jailbreak large vision-language models. CoRR, abs/2411.11496, 2024

    arXiv 2024

Show all 54 references
  1. [9]

    Kummerfeld, and Rada Mihalcea

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In ICML . OpenReview.net, 2024

    2024

  2. [10]

    Towards understanding jailbreak attacks in llms: A representation space analysis

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in llms: A representation space analysis. In EMNLP, 2024

    2024

  3. [11]

    Uncovering safety risks of large language models through concept activation vector

    Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector. In NeurIPS, 2024

    2024

  4. [12]

    On prompt-driven safeguarding for large language models

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai - Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In ICML . OpenReview.net, 2024 a

    2024

  5. [13]

    Nguyen, Jun Sun, and Tat - Seng Chua

    Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, and Tat - Seng Chua. Ali-agent: Assessing llms' alignment with human values via agent-based evaluation. In NeurIPS, 2024 b

    2024

  6. [14]

    Jailbroken: How does LLM safety training fail? In NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In NeurIPS, 2023

    2023

  7. [16]

    Leveraging reasoning with guidelines to elicit and utilize knowledge for enhancing safety alignment

    Haoyu Wang, Zeyu Qin, Li Shen, Xueqian Wang, Minhao Cheng, and Dacheng Tao. Leveraging reasoning with guidelines to elicit and utilize knowledge for enhancing safety alignment. CoRR, abs/2502.04040, 2025

    2025 arXiv

  8. [17]

    Rule based rewards for language model safety

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. In NeurIPS, 2024

    2024

  9. [18]

    Safety is not only about refusal: Reasoning-enhanced fine-tuning for interpretable LLM safety

    Yuyou Zhang, Miao Li, William Han, Yihang Yao, Zhepeng Cen, and Ding Zhao. Safety is not only about refusal: Reasoning-enhanced fine-tuning for interpretable LLM safety. CoRR, abs/2503.05021, 2025 a

    2025

  10. [19]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR . OpenReview.net, 2022

    2022

  11. [20]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  12. [21]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023

    2023

  13. [22]

    Safe lora: The silver lining of reducing safety risks when finetuning large language models

    Chia - Yi Hsu, Yu - Lin Tsai, Chih - Hsun Lin, Pin - Yu Chen, Chia - Mu Yu, and Chun - Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. In NeurIPS, 2024

    2024

  14. [23]

    Zico Kolter, Matt Fredrikson, and Dan Hendrycks

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J. Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In NeurIPS, 2024

    2024

  15. [24]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. CoRR, abs/2406.05946, 2024 a

    2024 arXiv

  16. [25]

    Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. Deliberative alignment: Reasoning enables safer language model...

    2024 arXiv

  17. [26]

    Enhancing model defense against jailbreaks with proactive safety reasoning

    Xianglin Yang, Gelei Deng, Jieming Shi, Tianwei Zhang, and Jin Song Dong. Enhancing model defense against jailbreaks with proactive safety reasoning. CoRR, abs/2501.19180, 2025

    2025

  18. [27]

    Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking

    Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, and Lei Sha. Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking. CoRR, abs/2502.12970, 2025

    2025

  19. [28]

    Bikel, Jason E

    Yiming Zhang, Jianfeng Chi, Hailey Nguyen, Kartikeya Upasani, Daniel M. Bikel, Jason E. Weston, and Eric Michael Smith. Backtracking improves generation safety. In ICLR . OpenReview.net, 2025 b

    2025

  20. [29]

    Does refusal training in llms generalize to the past tense? CoRR, abs/2407.11969, 2024

    Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in llms generalize to the past tense? CoRR, abs/2407.11969, 2024

    2024 arXiv

  21. [30]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR , 2024 b

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin - Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR , 2024 b

    2024

  22. [31]

    Bowman, Zac Hatfield - Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran - Johnson, Ethan...

    2022 arXiv

  23. [32]

    DeepSeek - AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei...

    2025 arXiv

  24. [33]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL https://arxiv.org/abs/2503.24290

    2025 arXiv

  25. [34]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837

    2025 arXiv

  26. [35]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. CoRR, abs/2503.01307, 2025

    2025 arXiv

  27. [36]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024

    2024 arXiv

  28. [37]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

    2017 arXiv

  29. [38]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025 arXiv

  30. [39]

    Jordan, and Pieter Abbeel

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR (Poster) , 2016

    2016

  31. [40]

    A strongreject for empty jailbreaks

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongreject for empty jailbreaks. In NeurIPS, 2024

    2024

  32. [41]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, abs/2307.15043, 2023 b

    2023 arXiv

  33. [42]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. In NeurIPS, 2024

    2024

  34. [43]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P....

    2024

  35. [44]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. CoRR, abs/2310.08419, 2023

    2023 arXiv

  36. [45]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. CoRR, abs/2312.06674, 2023

    2023 arXiv

  37. [46]

    Smith, Yejin Choi, and Hanna Hajishirzi

    Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Raghavi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, and Hanna Hajishirzi. The art of saying no: Contextual no...

    2024

  38. [47]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR . OpenReview.net, 2021

    2021

  39. [48]

    Hashimoto

    Yann Dubois, Bal \' a zs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475, 2024

    2024 arXiv

  40. [49]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021

    arXiv

  41. [50]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  42. [51]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  43. [52]

    Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-ope...

    2023

  44. [53]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul R \" o ttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In NAACL-HLT , pages 5377--5400. Association for Computational Linguistics, 2024

    2024

  45. [54]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  46. [55]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.