Pith. sign in

REVIEW 11 cited by

DPO Meets PPO: Reinforced Token Optimization for RLHF

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.18922 v4 pith:JCXXAV75 submitted 2024-04-29 cs.LG cs.AIcs.CLstat.ML

Han Zhong , Zikang Shan , Guhao Feng , Wei Xiong , Xinle Cheng , Li Zhao , Di He , Jiang Bian
show 1 more author
Liwei Wang
This is my paper
classification cs.LGcs.AIcs.CLstat.ML
keywords optimizationtexttttoken-wiseframeworklearningmodelspolicypreference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of large language models, its open-source implementation is still largely sub-optimal. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Under this framework, we introduce an algorithm Reinforced Token Optimization (\texttt{RTO}), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, \texttt{RTO} is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. Extensive experiments demonstrate that \texttt{RTO} performs better than PPO and other direct preference learning algorithms. In particular, RTO outperforms PPO by 7.5 points on the AlpacaEval 2 benchmark and by 4.1 points on Arena-Hard. Our code and models are available at \href{https://github.com/zkshan2002/RTO}{https://github.com/zkshan2002/RTO}.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distilled Reinforcement Learning for LLM Post-training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using teacher preferences to reweight RL gradients only on correct responses improves LLM post-training on math and knowledge benchmarks in both within- and cross-family settings.

  2. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  3. VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A vision-expert-filtered, chain-of-thought-guided, iteratively fine-tuned reward model boosts a compact 7B model's ability to judge vision-language responses, especially detecting hallucinations.

  4. Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning on questions extracted from CRISPR expert forums improves LLM accuracy on a new benchmark (Genome-Bench) by over 15 percentage points.

  5. Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

    cs.CL 2025-05 reject novelty 5.0 of 10

    MARA aligns LLMs with human preferences by training a 4M-parameter MLP to accept or reject candidate tokens, avoiding full-model fine-tuning, with measured gains based on the same reward models used in training.

  6. Stabilizing Policy Optimization via Logits Convexity

    cs.LG 2026-03 conditional novelty 4.0 of 10

    LCO replaces PPO-style policy gradients with regression toward the advantage-derived optimal logits/policy, restoring logits-level convexity and yielding more stable LLM RL training.

  7. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  8. Reinforced Language Models for Sequential Decision Making

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A 3B LLM post-trained with MS-GRPO, which gives every step the episode's total reward and samples high-advantage episodes, beats a 72B baseline on Frozen Lake but is inconsistent on Snake.

  9. Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective

    cs.LG 2025-06 conditional novelty 3.0 of 10

    Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.

  10. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

  11. Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

    cs.LG 2025-07 unverdicted novelty 1.0 of 10

    A tutorial reviewing LLM alignment through the lens of inverse reinforcement learning, arguing that neural reward models learned from human data are central to post-training.

Pith tools