REVIEW 9 cited by
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Direct Preference Optimization (DPO) and its variants are increasingly used for aligning language models with human preferences. Although these methods are designed to teach a model to generate preferred responses more frequently relative to dispreferred responses, prior work has observed that the likelihood of preferred responses often decreases during training. The current work sheds light on the causes and implications of this counter-intuitive phenomenon, which we term likelihood displacement. We demonstrate that likelihood displacement can be catastrophic, shifting probability mass from preferred responses to responses with an opposite meaning. As a simple example, training a model to prefer $\texttt{No}$ over $\texttt{Never}$ can sharply increase the probability of $\texttt{Yes}$. Moreover, when aligning the model to refuse unsafe prompts, we show that such displacement can unintentionally lead to unalignment, by shifting probability mass from preferred refusal responses to harmful responses (e.g., reducing the refusal rate of Llama-3-8B-Instruct from 74.4% to 33.4%). We theoretically characterize that likelihood displacement is driven by preferences that induce similar embeddings, as measured by a centered hidden embedding similarity (CHES) score. Empirically, the CHES score enables identifying which training samples contribute most to likelihood displacement in a given dataset. Filtering out these samples effectively mitigated unintentional unalignment in our experiments. More broadly, our results highlight the importance of curating data with sufficiently distinct preferences, for which we believe the CHES score may prove valuable.
Forward citations
Cited by 9 Pith papers
-
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
The Piggyback Hypothesis attributes emergent misalignment to chat-template tokens piggybacking finetuned behavior; Token-Regularized Finetuning (TReFT) mitigates it by regularizing prefix token representations.
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
BIRD: Behavior Induction via Representation-structure Distillation
BIRD transfers aligned behavior across models with different architectures, tasks, and data by minimizing linear CKA between teacher and student representations, and three teacher representation properties explain mos...
-
Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet
In medication-leaflet generation, ORPO and GRPO beat SFT and a zero-shot GPT-5 baseline, but GRPO's cross-domain advantage is partly built into its reward functions.
-
Normalized Rewards for Preference Optimization
A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...
-
Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
BDPO replaces the rejected response probability in the DPO loss denominator with a mixture of the learned and reference policies, yielding better chosen-response probability and better benchmark scores.
-
On Monotonicity in AI Alignment
Comparison-based preference learning losses of the studied form are locally pairwise monotone at stable minima, while individual-score and probability monotonicity require much stronger conditions that likely fail for...
-
On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
GRPO can suffer from Lazy Likelihood Displacement, where correct responses gain little or lose likelihood, and selectively downweighting penalties on harmful tokens (NTHR) mitigates this and boosts math reasoning performance.
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs
LeanPO improves Video-LLM alignment by using a reference-free average-likelihood reward, self-generated winning/losing pairs, and dynamic label smoothing, yielding gains on six video benchmarks.
Discussion (0). Sign in to comment.