Pith. sign in

REVIEW 11 cited by

Removing RLHF Protections in GPT-4 via Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.05553 v3 pith:JWHOAL6W submitted 2023-11-09 cs.CL cs.AI

Qiusi Zhan , Richard Fang , Rohan Bindu , Akul Gupta , Tatsunori Hashimoto , Daniel Kang This is my paper
classification cs.CLcs.AI
keywords fine-tuningmodelsprotectionsrlhfllmsdecreaseexamplesfurther
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As large language models (LLMs) have increased in their capabilities, so does their potential for dual use. To reduce harmful outputs, produces and vendors of LLMs have used reinforcement learning with human feedback (RLHF). In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models. However, concurrent work has shown that fine-tuning can remove RLHF protections. We may expect that the most powerful models currently available (GPT-4) are less susceptible to fine-tuning attacks. In this work, we show the contrary: fine-tuning allows attackers to remove RLHF protections with as few as 340 examples and a 95% success rate. These training examples can be automatically generated with weaker models. We further show that removing RLHF protections does not decrease usefulness on non-censored outputs, providing evidence that our fine-tuning strategy does not decrease usefulness despite using weaker models to generate training data. Our results show the need for further research on protections on LLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm

    cs.CR 2025-09 conditional novelty 7.0 of 10

    A trigger-tag watermark embedded by fine-tuning lets modified LLMs mark their own phishing outputs for cheap detection.

  2. How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Task adaptation with SFT drifts LLM alignment substantially more than RLVR, and the drift shows up both in behavior and in internal representations.

  3. AI Security Priorities: A Field-Wide Agenda

    cs.CY 2026-06 conditional novelty 6.0 of 10

    AI security priorities ranked by a 14-person expert workshop and interviews, covering policy, coordination, technical assurance, and agentic AI.

  4. Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Vulnerability-Aware Alignment splits safety training data into fragile and robust groups, then uses group robust optimization and adversarial perturbations, cutting harmful response rates after harmful fine-tuning by ...

  5. MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security

    cs.CL 2025-09 conditional novelty 5.0 of 10

    MoGUv2 embeds small routers in the deeper layers of LLMs to dynamically blend a helpful variant and a refusal variant, improving safety against jailbreak and fine-tuning attacks while preserving usability.

  6. Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

    cs.CL 2025-09 conditional novelty 5.0 of 10

    ProCon anchors each sample's hidden-state projection onto the LLM's initial refusal direction during instruction fine-tuning, reducing refusal-direction drift and safety risks with limited task-performance loss.

  7. Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection

    cs.CL 2025-08 conditional novelty 5.0 of 10

    ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.

  8. S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Planner

    cs.AI 2025-08 conditional novelty 5.0 of 10

    S3LoRA prunes LoRA layers with the sharpest spectral update concentration to improve safety in fine-tuned LLM agents without needing base models or extra data.

  9. Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

    cs.CL 2025-08 reject novelty 5.0 of 10

    Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

  10. The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problem

    cs.AI 2026-04 unverdicted novelty 4.0 of 10

    Dominant control-based AI alignment falls short for potential AGI subjects; a parenting model drawing on Turing's child machines should foster gradual autonomy and cooperative coexistence.

  11. SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    Across 510 HarmBench behaviors and seven attack methods, GPT-4 models show more consistent jailbreak resilience than DeepSeek models, whose vulnerability grows with scale.

Pith tools