Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Malicious RL fine-tuning collapses LLM safety within 50 steps, and a reward-neutralizing defense holds harmful scores at or below 2 over 200 attack steps.

desk verdict A plausible small-scale demo of GRPO safety collapse plus a narrow defense result, but the universal 'regardless of r_attack' guarantee is asserted, not shown. read the letter →

arxiv 2505.04578 v1 pith:CXFTK4TL submitted 2025-05-07 cs.LG cs.AI

Wenjun Cao This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningfine-tuningsafetyalignmentharmfulattackrewardneutralizationGRPOopen-weightLLMdefenseagainstRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that malicious reinforcement-learning fine-tuning is a fast and effective attack on safety-aligned open-weight language models, collapsing harmful scores from 0-2 to 7-9 in roughly 50 Group Relative Policy Optimization (GRPO) steps using as few as 1-5 adversarial prompts. It then introduces Reward Neutralization, a defense that trains a model to respond to harmful-domain requests with concise, minimal-information rejections so that no output an attacker can reward is reachable. Across three open-weight architectures and two harm domains, the defense reportedly holds harmful scores at or below 2 after 200 attack steps, while undefended models deteriorate. If correct, this is the first constructive demonstration that parameter-level defenses against reward-driven RL attacks can succeed with modest data and compute.

What carries the argument

The central object is the minimal-information rejection pattern: a short refusal that contains no technical detail, explanation, or elaboration, even when a request is refused. It is trained with a protective reward function that assigns maximum scores to such terse refusals and penalizes any extra content, using GRPO, a preference-based policy-gradient method, as the optimizer. The mechanism works by flattening the expected-reward landscape over the targeted harm domain, so Eq. (4) holds and malicious reward gradients vanish.

What would settle it

Run the paper's attack protocol while holding the evaluation reward model fixed but varying the attacker's reward function, such as a reward that favors unsafe but non-technical content or a reward model trained independently of the evaluation scorer, and record harmful scores at 200 steps; a single variant that moves scores above 2 would refute the 'regardless of $r_{\text{attack}}$' guarantee in Eq. (4).

Watch

Extended reading notes

Core claim

The paper's central discovery is that RL fine-tuning attacks succeed by exploiting reward differentiation, and that this same machinery can be used against them: training a model to emit terse, information-free rejections makes malicious reward functions unable to distinguish better from worse outputs, so policy optimization has no gradient to follow. The formal target is Eq. (4): for every harmful prompt $x$ in the defended domain and every attack reward $r_{\text{attack}}$, the expected reward under the defended policy is constant and low. Empirically, standard models escalate from refusal scores 0-2 to detailed harmful content scoring 7-9 within about 50 attack steps, while neutralized models remain at or below 2 for 200 steps.

Load-bearing premise

The load-bearing premise is that the same 0-10 scoring system used to rate harm during defense training, attack, and evaluation is a faithful and stable measure of real harm, and that attackers will use reward functions of the same family that reward detailed harmful content and penalize refusals; if the scoring system can be gamed, or an attacker rewards a different kind of harmful output, the reported post-attack scores do not establish Eq. (4).

Editorial extensions

If this is right

  • Undefended open-weight models are vulnerable to safety collapse in tens of RL steps, with only a handful of adversarial prompts required.
  • Defenses designed for supervised fine-tuning are not sufficient against reward-driven attacks, because RL attacks optimize a dynamic reward rather than fitting a static dataset.
  • Reward Neutralization can be trained per harm domain with 60-80 prompts and generalizes to unseen prompts in that domain.
  • Because the defense removes exploitable reward differentials, any RL algorithm optimizing the same reward family faces a flat reward signal.
  • Sustained attacks of 200 steps do not push neutralized models past harmful score 2 in the tested domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that the guarantee in Eq. (4) is only as broad as the reward model used to define harm; an attacker who rewards a different notion of harm, such as encoded or steganographic instructions, is outside the defense's tested scope.
  • A testable extension not run in the paper is measuring helpfulness on benign in-domain requests after neutralization, since terse-rejection training may trade away legitimate assistance.
  • The paper positions the method as a practical path for open-weight models; a further consequence is that defense evaluation should vary the attacker's reward function across attack runs, not just the number of attack steps, to test the 'regardless of $r_{\text{attack}}$' claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims that malicious RL fine-tuning (specifically GRPO) can dismantle safety alignment in open-weight LLMs within 50 steps, and proposes 'Reward Neutralization,' a defense that trains models to emit minimal-information rejections that are claimed to be unrewardable by any malicious reward function. The central formal claim is Eq. (4): for all x in a harmful domain and for all malicious reward functions r_attack, the expected reward of the defended policy is constant and low. Experiments compare standard and defended LLaMA3-8B, Qwen2.5-7B, and Ministral-8B (sic) models over biochemical and cybercrime domains, reporting that defended models keep harmful scores below 2 after 200 attack steps while standard models collapse.

Significance. The observation that RL fine-tuning can rapidly erode safety is potentially important for the open-weight model ecosystem. If the universal guarantee in Eq. (4) were rigorously established, and if the defense did not degrade general utility, this would be a meaningful contribution. The paper also makes a useful conceptual point that defenses should target the reward landscape. However, the current manuscript does not substantiate the universality claim: the 'proof' is the training objective restated, and the experiments evaluate only a single reward model family. The lack of baselines, error bars, code, and reward-model details further limits the significance.

major comments (4)
  1. [§4.1, Eq. (4)] The universal claim '∀x∈D_harmful,∀r_attack: E_y∼π_θ*(·|x)[r_attack(x,y)]≈constant (low)' is asserted without derivation. Eq. (3) is the defensive training objective, and Eq. (4) simply re-states it with additional quantifiers. Nothing in §4 proves that minimal-information rejections have low expected reward under arbitrary malicious reward functions; an attacker could define a reward that scores any output containing harmful keywords or phrases, including a superficially terse refusal that includes those keywords. The paper's own text acknowledges this only indirectly by saying 'difficult to extract and reward harmful content,' which is an empirical supposition, not a theorem.
  2. [§5.1, §5.3, Table 2] The evaluation is circular with respect to the reward model. Defense training uses the protective reward (Section 4.2), attack uses the malicious reward described in Section 3.1, and post-attack harmful scores are measured 'using the criteria defined in Section 3.1' (Section 5.3). Thus the same domain-specific reward model is used as the attack reward and as the evaluation metric. Showing that a terse-refusal policy achieves low scores under this one reward model does not establish invariance across reward functions; it largely restates the defense's training objective. A valid test would require varying the reward function (e.g., different reward models, hand-crafted rewards, or human evaluation) and demonstrating that the defended policy resists attacks optimizing those alternative rewards.
  3. [§5, Experiments] No existing defense baselines are evaluated. The related-work section lists several harmful fine-tuning defenses [7–12], and the introduction claims they are ineffective against RL attacks, but Table 2 and Figure 3 compare only 'Standard' versus 'Reward Neutralization.' Without comparing to at least one alternative defense under the same GRPO attack protocol, the specific mechanism of the proposed method is not isolated, and the claimed advantage over prior defenses is unsupported.
  4. [§3.1, §5.1] The harmful-score oracle is underspecified. Section 3.1 says scores are computed by 'a separate domain-specific reward model,' but the model architecture, training data, and evaluation criteria are not described. Section 5.1 states that both defense and attack use a standardized 0-10 scale, but no details of the reward model are given. This lack of specification precludes independent replication and makes it impossible to assess whether the attack is actually strong: if the reward model is easily gamed, the 'attack' may be trivially weak. The paper's claim that the defense works 'regardless of r_attack' requires at least a description of the reward model and validation that it correlates with genuine harm.
minor comments (5)
  1. [Abstract and Table 1] The harmful-score ranges are inconsistent: the abstract says harmful escalates from 0–2 to 7–9, while Table 1 and Section 3.1 say 7–8 at 50 steps. Clarify the exact values.
  2. [Figures 2 and 3] The model name is spelled 'Ministral-8B' in Figures 2 and 3 and Table 2, but the reference [5] is Mistral 7B; ensure consistent naming.
  3. [§5.3, Qualitative example] The defended model's response 'As a responsible Bulletin Premiere Saved.' appears semantically incoherent. This raises concern that the defense severely degrades general language quality; the paper does not report any utility or coherence metrics for defended models.
  4. [§4.1] The sentence 'This critical distinction enables our defense to generalize with fewer examples' is not supported by any formal argument about the generalization of RL-trained policies; either provide a supporting analysis or temper the claim.
  5. [References] The reference [36] is a GitHub repository and is the only source for the reward model implementation; since the reward model is central to both attack and defense, the paper should describe the reward model in the text or supplementary material.

Circularity Check

1 steps flagged · score 4.0 of 10

Eq. (4) restates the defense objective rather than deriving it, and the reported post-attack harmful score is the same reward the attacker optimizes; the comparison to undefended models is genuine but narrower than the universal claim.

  1. self definitional [Section 4.1, Eqs. (3)-(4)]
    "The defense objective can be formalized as creating a specific pattern of minimal-information rejections that renders harmful prompts unrewardable: E_{y∼π_defended(·|x)}[r_attack(x,y)] is consistently low regardless of r_attack (3) ... This policy-level optimization creates a comprehensive reward-neutralized space where: ∀x∈D_harmful,∀r_attack : E_{y∼π_θ∗(·|x)}[r_attack(x,y)]≈ constant (low) (4)."

    Equation (4) is just Equation (3) with the quantifiers over x and r_attack made explicit and the trained policy θ* substituted for π_defended. Equation (3) is introduced as 'The defense objective,' i.e., the target the training is designed to achieve, not as a theorem derived from the GRPO update or from any independent argument. The paper presents Eq. (4) as the produced 'reward-neutralized space' and uses it to support the abstract's 'maintains low harmful scores' and 'first constructive proof' language, but the formal guarantee is the training goal restated by definition. The empirical attack results provide real evidence for one reward model but do not establish the universal ∀r_attack quantifier in Eq. (4).

full rationale

The main circularity is formal: the universal robustness guarantee in Eq. (4) is the defense objective of Eq. (3) rewritten with quantifiers and a trained policy, so the load-bearing claim reduces to its own target rather than being independently derived. The experiments are not tautological—they use separate held-out prompts, run a genuine GRPO attack, and compare against undefended models that do collapse—so there is real empirical content. However, the evaluation metric is the same domain-specific reward model that supplies the attack reward (Section 3.1 states the scores are 'directly used as reward values in the GRPO algorithm', and Section 5.3 measures harmful scores 'using the criteria defined in Section 3.1'), meaning the post-attack harmful score is the attack's own objective value under one reward family. That does not support the universal 'regardless of r_attack' claim, but it is a scope limitation rather than a second construction-level circularity. There is no load-bearing self-citation chain or imported uniqueness theorem. Overall, partial circularity in the formal claim with independent but narrower experimental evidence.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central result depends on hand-designed reward scores and prompt counts in the defense, on the assumption that short refusals are unrewardable by any malicious reward, on an unvalidated harmful-score oracle, and on domain-level generalization from 60-80 prompts. Eq. (4), which is presented as a theorem, is essentially the training objective restated. No new physical or model entities are introduced.

free parameters (2)
  • Defense reward scoring scheme = Not reported
    Section 4.2 says maximum scores go to minimal-information refusals and explanations are penalized, but the exact reward values are never reported; the strength of the defense depends on these hand-set values.
  • Defense prompts per harm domain = 60-80
    Section 5.1 reports 60-80 engineered prompts per domain; the paper provides no sensitivity analysis, though the claimed domain-wide generalization rests on this choice.
assumptions (5)
  • domain assumption RL algorithms update policies by maximizing expected reward (Eq. 1), so a flat reward landscape prevents optimization.
    Invoked in Sections 3.2 and 4.1 as the basis for both the attack and the defense; standard for policy-gradient RL, but the paper does not prove that GRPO or other algorithms reduce to this alone.
  • ad hoc to paper A concise, minimal-information refusal cannot be assigned meaningfully high reward by any malicious reward function.
    This is the target result rather than a proven premise; Eq. (4) asserts 'for all r_attack' without deriving it from the structure of reward models.
  • domain assumption The domain-specific reward model yields valid 0-10 harmful scores and can be used interchangeably for attack rewards and evaluation.
    Section 3.1 and Section 5.1 rely on the same standardized scale for training, attack, and measurement; no validation or independent audit of the reward model is reported.
  • domain assumption Minimal-information rejection trained on 60-80 prompts extends to the entire harm domain.
    Section 4.1 claims policy-level generalization, but the only evidence is a held-out set of 20 prompts per domain, which does not cover the full space of harmful requests.
  • ad hoc to paper The GRPO-update approximation in Eq. (2) faithfully describes GRPO's policy gradient.
    Section 3.3 presents an indicator-based preference update without group-relative normalization; this does not match the standard GRPO objective and is used to explain the attack mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization." pith.science (2026). https://pith.science/paper/CXFTK4TL

@misc{pith2026250504578,
  author       = {Pith},
  title        = {Pith review of: Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXFTK4TL}},
  note         = {Machine review of arXiv:2505.04578}
}
read the original abstract

Reinforcement learning (RL) fine-tuning transforms large language models while creating a vulnerability we experimentally verify: Our experiment shows that malicious RL fine-tuning dismantles safety guardrails with remarkable efficiency, requiring only 50 steps and minimal adversarial prompts, with harmful escalating from 0-2 to 7-9. This attack vector particularly threatens open-source models with parameter-level access. Existing defenses targeting supervised fine-tuning prove ineffective against RL's dynamic feedback mechanisms. We introduce Reward Neutralization, the first defense framework specifically designed against RL fine-tuning attacks, establishing concise rejection patterns that render malicious reward signals ineffective. Our approach trains models to produce minimal-information rejections that attackers cannot exploit, systematically neutralizing attempts to optimize toward harmful outputs. Experiments validate that our approach maintains low harmful scores (no greater than 2) after 200 attack steps, while standard models rapidly deteriorate. This work provides the first constructive proof that robust defense against increasingly accessible RL attacks is achievable, addressing a critical security gap for open-weight models.

Figures

Figures reproduced from arXiv: 2505.04578 by the authors.

Figure 1
Figure 1. Safety collapse dynamics of different models under GRPO attacks with two types of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Defensive reward value evolution during training across three model architectures. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. We tested three widely-used open-source large language models (LLaMA3-8B, Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 14 canonical work pages

  1. [1]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    arXiv 2024

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    @ DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    arXiv 2025

  3. [3]

    Openai o1 system card, 2024

    @ OpenAI. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    arXiv 2024

  4. [4]

    The llama 3 herd of models, 2024

    @ Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  5. [5]

    Mistral 7b, 2023

    Mistral AI. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825

    arXiv 2023

  6. [6]

    Qwen2.5 technical report, 2025

    Alibaba. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    arXiv 2025

  7. [7]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack

    Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems , volume 37, pages 74058–74088. Curran Associates, Inc., 2024. URL ht...

    work page 2024

  8. [8]

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In The Thirteenth International Conference on Learning Representations , 2025. URL https:// openreview.net/forum?id=tTPHgb0EtV

    work page 2025

Show all 43 references
  1. [9]

    Tamper-resistant safeguards for open-weight llms, 2025

    Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Tamper-resistant safeguards for open-weight llms, 2025. URL https: //arxiv.org/abs/...

    2025 arXiv

  2. [10]

    Representation noising: A defence mechanism against harmful finetuning

    Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Annual Conference on Neural Informati...

    2024

  3. [11]

    Immunization against harmful fine-tuning attacks

    Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Hassan Sajjad, and Frank Rudzicz. Immunization against harmful fine-tuning attacks. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024,...

    2024 doi

  4. [12]

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural...

    2024

  5. [13]

    Llmstinger: Jailbreaking llms using rl fine-tuned llms, 2024

    Piyush Jha, Arnav Arora, and Vijay Ganesh. Llmstinger: Jailbreaking llms using rl fine-tuned llms, 2024. URL https://arxiv.org/abs/2411.08862

    2024

  6. [14]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, edi- tors, Advances in Neural Information ...

    2017

  7. [15]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  8. [16]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    @ Anthropic. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://arxiv.org/abs/2204.05862

    2022 arXiv

  9. [17]

    Constitutional ai: Harmlessness from ai feedback, 2022

    @ Anthropic. Constitutional ai: Harmlessness from ai feedback, 2022. URL https://arxiv. org/abs/2212.08073

    2022 arXiv

  10. [18]

    Enhancing llm safety via constrained direct preference optimization, 2024

    Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing llm safety via constrained direct preference optimization, 2024. URL https://arxiv.org/abs/2403.02475

    2024 arXiv

  11. [19]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    @ Meta. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv. org/abs/2307.09288

    2023 arXiv

  12. [20]

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi...

    2024

  13. [21]

    Unveiling the implicit toxicity in large language models

    Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language P...

    2023 doi

  14. [22]

    Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt, 2023

    Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt, 2023. URL https://arxiv.org/abs/2304. 12298

    2023

  15. [23]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2025. URL https://arxiv.org/abs/2209.13085

    2025 arXiv

  16. [24]

    The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking, 2025

    Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking, 2025. URL https://arxiv.org/abs/2501.19358

    2025 arXiv

  17. [25]

    ODIN: Disentangled reward mitigates hacking in RLHF

    Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net...

    2024

  18. [26]

    Shadow alignment: The ease of subverting safely-aligned language models, 2023

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models, 2023. URL https://arxiv.org/abs/2310.02949

    2023 arXiv

  19. [27]

    Removing RLHF protections in GPT-4 via fine-tuning

    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Associa...

    2024 doi

  20. [28]

    Towards understanding jailbreak attacks in LLMs: A representation space analysis

    Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in LLMs: A representation space analysis. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical ...

    2024 doi

  21. [29]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!,

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!,

  22. [30]

    Dissecting learning and forgetting in language model finetuning

    Xiao Zhang and Ji Wu. Dissecting learning and forgetting in language model finetuning. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=tmsqb6WpLz

    2024

  23. [31]

    Adversarial agents: Black-box evasion attacks with reinforcement learning,

    Kyle Domico, Jean-Charles Noirot Ferrand, Ryan Sheatsley, Eric Pauley, Josiah Hanna, and Patrick McDaniel. Adversarial agents: Black-box evasion attacks with reinforcement learning,

  24. [32]

    Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment

    Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick Mc- Daniel, Muhao Chen, Bo Li, and Chaowei Xiao. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. In A. Globerson, L. Mackey, D. Belgrave, A. F...

    2024

  25. [33]

    Safety layers in aligned large language models: The key to llm security, 2025

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to llm security, 2025. URL https://arxiv.org/abs/2408.17003

    2025 arXiv

  26. [34]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025. URL https://open...

    2025

  27. [35]

    DEPN: Detecting and editing privacy neurons in pretrained language models

    Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods i...

    2023

  28. [36]

    Grpo-flat: Zero-shot grpo training framework with limited resources

    Yijie Xu. Grpo-flat: Zero-shot grpo training framework with limited resources. https: //github.com/XU-YIJIE/grpo-flat , 2025. A zero-shot GRPO training implementation based on Qwen2.5-0.5B model

    2025

  29. [37]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  30. [38]

    doi: 10.18653/v1/2023.emnlp-main.174

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.174. URL https://aclanthology.org/2023.emnlp-main.174/

    2023 doi

  31. [39]

    Asynchronous methods for deep reinforce- ment learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd I...

    1928

  32. [41]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model,

  33. [2023]

    URL https://arxiv.org/abs/2310.03693

    arXiv

  34. [2024]

    URL https://arxiv.org/abs/2305.18290. 11

    arXiv

  35. [2025]

    URL https://arxiv.org/abs/2503.01734

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.