REVIEW 10 cited by
Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Language model alignment methods such as reinforcement learning from human feedback (RLHF) have led to impressive advances in language model capabilities, but are limited by a widely observed phenomenon known as overoptimization, where the quality of the language model degrades over the course of the alignment process. As the model optimizes performance with respect to an offline reward model, it overfits to inaccuracies and drifts away from preferred responses covered by the data. To discourage such distribution shift, KL-regularization is widely employed in existing offline alignment methods, but overoptimization continues to harm performance. Lending theoretical insight into the source of these empirical observations, we first show that the KL-regularization is too weak to prevent overfitting, then raise the following question: is it possible to design an efficient algorithm that is provably robust to overoptimization? We address this question with a new algorithm for offline alignment, $\chi^2$-Preference Optimization ($\chi$PO). $\chi$PO is a one-line change to Direct Preference Optimization (DPO; Rafailov et al., 2023), which only involves modifying the logarithmic link function in the DPO objective. Despite this minimal change, $\chi$PO implicitly implements the principle of pessimism in the face of uncertainty via regularization with the $\chi^2$-divergence -- which quantifies uncertainty more effectively than KL-regularization -- and provably alleviates overoptimization, achieving sample-complexity guarantees based on single-policy concentrability -- the gold standard in offline reinforcement learning. $\chi$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm that is provably robust to overoptimization.
Forward citations
Cited by 10 Pith papers
-
Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
Preference-based LLM alignment under an unknown reward-preference link becomes a single-index model; three new algorithms converge to the optimal divergence-constrained policy without knowing the link.
-
Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
NLHF achieves the minimax-optimal worst-case average-utility distortion (1/2+o(1))β, while RLHF and DPO can suffer distortion up to e^{Ω(β)} or unbounded under certain comparison sampling.
-
A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Under linear-model assumptions, offline RLHF and DPO both reduce to logistic regression, and privatizing labels before corruption (LTC) carries an extra c(ε) factor in the error bounds compared to corrupting before pr...
-
Preference learning made easy: Everything should be understood through win rate
Under two axioms (preference-consistency and prevalence-consistency), the only distribution-grounded evaluation for preference learning is h-win rate, and most popular alignment methods can be classified by whether th...
-
Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
LLM self-play finetuning is equivalent to adversarial imitation learning; the chi-squared regularized variant SPIF bounds rewards and improves stability.
-
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Dense scene-graph-grounded rewards let a 7B multimodal LLM trained on 7K synthetic questions beat SFT and sparse-RL baselines and outscore GPT-4o on average across 12 spatial/real-world benchmarks.
-
On a few pitfalls in KL divergence gradient estimation for RL
Differentiating KL estimates as losses gives biased or reversed KL gradients; the paper derives and tests unbiased sequence-level estimators.
-
Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment
SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.
-
Learning a Pessimistic Reward Model in RLHF
Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.
-
Normalized Rewards for Preference Optimization
A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...
Discussion (0). Sign in to comment.