REVIEW 9 cited by
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While alignment algorithms are now commonly used to tune pre-trained language models towards a user's preferences, we lack explanations for the underlying mechanisms in which models become ``aligned'', thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained language model, GPT2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.
Forward citations
Cited by 9 Pith papers
-
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.
-
Not All Refusals Are Equal: How Safety Alignment Fails Cybersecurity at Scale
Domain-specific abliteration removes cybersecurity refusal on Kimi K2 (100%→7%) while preserving most other safety refusals; susceptibility depends mainly on safety-training method and architecture, not size.
-
Dual Mechanisms of Value Expression: Intrinsic vs. Prompted Values in Large Language Models
Intrinsic and prompted value expressions in LLMs share some circuits but have distinct mechanisms: intrinsic directions make outputs more diverse, prompted directions drive instruction compliance and can disable refusal.
-
The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions
Safety refusal in Llama 3.1 8B is governed by a dominant activation direction plus smaller interpretable directions, and removing prompt tokens that activate these secondary directions can bypass fine-tuned safety.
-
Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
In healthcare LLM evaluation, multiple-choice accuracy and open-ended task scores correlate only weakly, and the paper's proposed Relaxed Perplexity metric aims to improve open-ended factuality scoring but rests on an...
-
Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates
Combining suffix-window representation finetuning with an ActGrad-pruned surrogate cuts latent-adversarial-training FLOPs per step by 48.1% with only 0.0118% trainable parameters, while accepting higher attack success rates.
-
Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
RL post-training forgets less than SFT because it trains on on-policy data; refreshing SFT data each epoch also reduces forgetting.
-
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
Neuron-Fusion selectively restores large-change neurons from a fine-tuned multimodal model and suppresses small changes, improving language retention with modest visual trade-offs.
-
NEAT: Concept driven Neuron Attribution in LLMs
NEAT identifies concept neurons by feeding a single mean hidden-state vector through the model and ranking neurons by their effect on concept-word probabilities.
Discussion (0). Continue with ORCID to comment.