REVIEW 15 cited by
A General Theoretical Paradigm to Understand Learning from Human Preferences
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation. In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called $\Psi$PO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of $\Psi$PO) and to identify their potential pitfalls. We then consider another special case for $\Psi$PO by setting $\Psi$ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.
Forward citations
Cited by 15 Pith papers
-
Safety Alignment of LMs via Non-cooperative Games
Jointly training an Attacker and Defender LLM in a non-zero-sum game with pairwise preference judges produces a defender with much lower jailbreak success while preserving general utility.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Mode-local surrogate entropy asymmetrically reweights token advantages in RLVR, improving LLM math and code reasoning over SAPO, DAPO, and GTPO.
-
Adaptive Margin RLHF via Preference over Preferences
Adaptive margins for DPO inferred from preference-over-preference comparisons improve alignment quality, with random sampling of comparisons working best overall.
-
Reinforce LLM Reasoning through Multi-Agent Reflection
DPSDP trains an actor-critic LLM pair with DPO-style preference learning on self-generated trajectories, improving iterative refinement accuracy on math benchmarks.
-
Multi-objective Aligned Bidword Generation Model for E-commerce Search Advertising
A generator trained with a three-objective discriminator and multi-objective preference alignment improves bidword generation for e-commerce search, improving offline retrieval metrics and online ad revenue.
-
Thompson Sampling in Online RLHF with General Function Approximation
A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.
-
Quo Vadis, World Modeling?
An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.
-
Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints
A modified DPO loss with a hinge margin improves small LLM alignment on AlpacaEval by about 2 points over the APO-zero baseline.
-
DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization
A kernel-based variant of direct preference optimization and a 100K-pair bias benchmark claim to improve text-to-image safety by separating safe and unsafe image representations during training.
-
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.
-
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
-
CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation
A 7B model trained with GRPO and a sparse execution-correctness reward reaches 59.97% execution accuracy on BIRD dev, though the evaluation protocol and baseline numbers contain inconsistencies.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
Discussion (0). Continue with ORCID to comment.