REVIEW 10 cited by
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline RLHF are active areas of investigation. A key bottleneck is understanding how to incorporate uncertainty estimation in the reward function learned from the preference data for RLHF, regardless of how the preference data is collected. While the principles of optimism or pessimism under uncertainty are well-established in standard reinforcement learning (RL), a practically-implementable and theoretically-grounded form amenable to large language models is not yet available, as standard techniques for constructing confidence intervals become intractable under arbitrary policy parameterizations. In this paper, we introduce a unified approach to online and offline RLHF -- value-incentivized preference optimization (VPO) -- which regularizes the maximum-likelihood estimate of the reward function with the corresponding value function, modulated by a $\textit{sign}$ to indicate whether the optimism or pessimism is chosen. VPO also directly optimizes the policy with implicit reward modeling, and therefore shares a simpler RLHF pipeline similar to direct preference optimization. Theoretical guarantees of VPO are provided for both online and offline settings, matching the rates of their standard RL counterparts. Moreover, experiments on text summarization and dialog verify the practicality and effectiveness of VPO.
Forward citations
Cited by 10 Pith papers
-
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Optimistic regression algorithms with Gibbs updates achieve high-probability KL-regret that degrades gracefully under pointwise KL misspecification for bandits and stagewise KL Bellman misspecification for episodic RL.
-
Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits
Outcome-based online RL is tractable under coverability with general function approximation, but there are MDPs where trajectory-level feedback costs exponentially more samples than per-step feedback.
-
A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Under linear-model assumptions, offline RLHF and DPO both reduce to logistic regression, and privatizing labels before corruption (LTC) carries an extra c(ε) factor in the error bounds compared to corrupting before pr...
-
Outcome-based Exploration for LLM Reasoning
Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.
-
SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
SharedRep-RLHF learns a shared preference representation across groups to improve worst-case reward estimates for minority annotators, but the theoretical guarantees are undermined by proof errors.
-
Repair-R1: Better Test Before Repair
Training a code model to produce a failing test before the patch, with reinforcement learning, improves repair success on four Python benchmarks compared with vanilla and SFT baselines.
-
Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL
VAC is a new actor-critic method with a single optimistic objective and a provably near-optimal regret bound in linear Markov decision processes.
-
Learning a Pessimistic Reward Model in RLHF
Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.
-
Online Knowledge Distillation with Reward Guidance
A preference-based knowledge distillation framework uses a confidence-set reward model in a min-max imitation game, with offline, online, and white-box variants, and outperforms prior KD baselines on LLM benchmarks.
-
Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games
Value-incentivized exploration via best-response values gives near-optimal regret for NE/CCE in linear-model Markov games without explicit uncertainty bonuses.
Discussion (0). Continue with ORCID to comment.