Pith. sign in

REVIEW 9 cited by

DPO Meets PPO: Reinforced Token Optimization for RLHF

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.18922 v4 pith:JCXXAV75 submitted 2024-04-29 cs.LG cs.AIcs.CLstat.ML

Han Zhong , Zikang Shan , Guhao Feng , Wei Xiong , Xinle Cheng , Li Zhao , Di He , Jiang Bian
show 1 more author
Liwei Wang
This is my paper
classification cs.LGcs.AIcs.CLstat.ML
keywords optimizationtexttttoken-wiseframeworklearningmodelspolicypreference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of large language models, its open-source implementation is still largely sub-optimal. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Under this framework, we introduce an algorithm Reinforced Token Optimization (\texttt{RTO}), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, \texttt{RTO} is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. Extensive experiments demonstrate that \texttt{RTO} performs better than PPO and other direct preference learning algorithms. In particular, RTO outperforms PPO by 7.5 points on the AlpacaEval 2 benchmark and by 4.1 points on Arena-Hard. Our code and models are available at \href{https://github.com/zkshan2002/RTO}{https://github.com/zkshan2002/RTO}.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distilled Reinforcement Learning for LLM Post-training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using teacher preferences to reweight RL gradients only on correct responses improves LLM post-training on math and knowledge benchmarks in both within- and cross-family settings.

  2. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  3. VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A vision-expert-filtered, chain-of-thought-guided, iteratively fine-tuned reward model boosts a compact 7B model's ability to judge vision-language responses, especially detecting hallucinations.

  4. Stabilizing Policy Optimization via Logits Convexity

    cs.LG 2026-03 conditional novelty 4.0 of 10

    LCO replaces PPO-style policy gradients with regression toward the advantage-derived optimal logits/policy, restoring logits-level convexity and yielding more stable LLM RL training.

  5. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  6. Reinforced Language Models for Sequential Decision Making

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A 3B LLM post-trained with MS-GRPO, which gives every step the episode's total reward and samples high-advantage episodes, beats a 72B baseline on Frozen Lake but is inconsistent on Snake.

  7. Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective

    cs.LG 2025-06 conditional novelty 3.0 of 10

    Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.

  8. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

  9. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools