Pith. sign in

REVIEW 16 cited by

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.11456 v4 pith:HGJSQ6ZU submitted 2023-12-18 cs.LG cs.AIstat.ML

Wei Xiong , Hanze Dong , Chenlu Ye , Ziqi Wang , Han Zhong , Heng Ji , Nan Jiang , Tong Zhang This is my paper
classification cs.LGcs.AIstat.ML
keywords rlhfofflinepracticaltheoreticalalgorithmsalignmentexistingfeedback
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper studies the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF). We first identify the primary challenges of existing popular methods like offline PPO and offline DPO as lacking in strategical exploration of the environment. Then, to understand the mathematical principle of RLHF, we consider a standard mathematical formulation, the reverse-KL regularized contextual bandit for RLHF. Despite its widespread practical application, a rigorous theoretical analysis of this formulation remains open. We investigate its behavior in three distinct settings -- offline, online, and hybrid -- and propose efficient algorithms with finite-sample theoretical guarantees. Moving towards practical applications, our framework, with a robust approximation of the information-theoretical policy improvement oracle, naturally gives rise to several novel RLHF algorithms. This includes an iterative version of the Direct Preference Optimization (DPO) algorithm for online settings, and a multi-step rejection sampling strategy for offline scenarios. Our empirical evaluations on real-world alignment experiment of large language model demonstrate that these proposed methods significantly surpass existing strong baselines, such as DPO and Rejection Sampling Optimization (RSO), showcasing the connections between solid theoretical foundations and their potent practical implementations.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Subjective Risk Decomposition: A New View for Uncertainty Quantification

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Most existing epistemic/aleatoric uncertainty measures are special cases of one bias-variance-entropy decomposition of subjective risk under a strictly proper loss.

  2. rePIRL: Learn PRM with Inverse RL for LLM Reasoning

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.

  3. Outcome-based Exploration for LLM Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

  4. Bridging Offline and Online Reinforcement Learning for LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Semi-online DPO, which syncs the generation model every few update steps, performs nearly as well as fully online DPO and GRPO, while strongly beating offline DPO.

  5. Aligning Large Language Models with Implicit Preferences from User-Generated Content

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PUGC turns unlabeled user-generated content into preference pairs by scoring policy responses against the source post as a reference, reaching 35.93% length-controlled win rate on AlpacaEval 2.0 with Mistral-7B-Instruct.

  6. Thompson Sampling in Online RLHF with General Function Approximation

    cs.LG 2025-05 reject novelty 6.0 of 10

    A model-free posterior sampling algorithm for online RLHF is shown to achieve O(sqrt(T)) regret when the completed function class has low Bellman eluder dimension.

  7. Learning a Pessimistic Reward Model in RLHF

    cs.LG 2025-05 reject novelty 6.0 of 10

    Pessimistic fine-tuning of reward models against rejection-sampling policies lets RLHF agents optimize greedily without KL regularization and still avoid reward hacking.

  8. Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning

    cs.LG 2025-05 reject novelty 6.0 of 10

    Claims the first O(1/T) global optimality guarantee for deep neural actor-critic methods in decentralized multi-agent reinforcement learning, but the central proof conflates Q-function TD errors with advantage functions.

  9. Normalized Rewards for Preference Optimization

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...

  10. Improved Bounds for Private and Robust Alignment

    cs.LG 2025-12 conditional novelty 5.0 of 10

    Private MLE log loss is near-optimal for alignment, and square-loss alignment tolerates corruption at the optimal linear rate, including in online exploration.

  11. Boosting LLM Reasoning via Spontaneous Self-Correction

    cs.AI 2025-06 conditional novelty 5.0 of 10

    SPOC trains LLMs to interleave self-verification and solution attempts in a single pass, reporting gains on math benchmarks, though most gains come from stronger first attempts.

  12. Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy

    cs.AI 2025-05 reject novelty 4.0 of 10

    AI copilot preference optimization is organized into a pre-, mid-, and post-interaction taxonomy, with a unified definition of AI copilots.

  13. OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models

    cs.CY 2025-05 conditional novelty 4.0 of 10

    The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.

  14. T-TAMER: Provably Taming Trade-offs in ML Serving

    cs.LG 2025-09 reject novelty 3.0 of 10

    T-TAMER claims recall is necessary and sufficient for provably optimal early-exit and cascade serving policies, but the main extensions are under-derived and partly reduce to known Gittins-index results.

  15. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

  16. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools