Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

On-Policy RL with Optimal Reward Baseline

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Length-weighted reward baseline stabilizes LLM reinforcement learning

desk verdict OPO's length-weighted baseline is a useful practical recipe, but the 'optimal' label rests on an unverified mean-field step and the empirical evidence is single-run with small gaps. read the letter →

arxiv 2505.23585 v2 pith:R6VFQURK submitted 2025-05-29 cs.LG cs.CL

Yaru Hao , Li Dong , Xun Wu , Shaohan Huang , Zewen Chi , Furu Wei This is my paper · ORCID
classification cs.LGcs.CL
keywords reinforcementlearninglargelanguagemodelspolicygradientvariancereductionrewardbaselineon-policytrainingmathematicalreasoningGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes OPO, a reinforcement learning algorithm for large language models that combines two ideas: updating the policy only on data freshly sampled from the current policy, and subtracting a length-weighted average reward as the baseline. The authors argue that this baseline is the practically feasible form of the theoretically optimal variance-reducing baseline, once one assumes token-level gradients are roughly orthogonal and identically distributed. They show empirically on mathematical reasoning benchmarks that OPO outperforms GRPO and on-policy variants, with lower policy shift, higher output entropy, and no value model, reference model, or KL/entropy regularization. If correct, OPO suggests that a single policy model with a simple length-weighted baseline suffices for stable and diverse RL training.

What carries the argument

The central object is the length-weighted reward baseline $b^*(x) = \sum_i l_i r_i / \sum_i l_i$, used to define the advantage $A_i = r(x,y_i) - b^*(x)$ in a basic policy-gradient objective. It is derived from the classical optimal baseline $b^* = \mathbb{E}[g(y)^2 r]/\mathbb{E}[g(y)^2]$ by assuming token-level gradients are orthogonal with identically distributed norms, so that the squared gradient norm is proportional to response length. The other load-bearing mechanism is exact on-policy training: every gradient step uses rollouts sampled from the current policy, avoiding the off-policy drift that the authors link to entropy collapse and large policy shifts.

What would settle it

Measure, for an actual policy being trained, the per-token squared gradient norms across a sample of responses; if the norm is not approximately proportional to response length, the derivation of $b^*(x)$ fails. Alternatively, compare the empirical variance of policy-gradient estimates under the length-weighted baseline versus the group-mean baseline on a fixed set of prompts; if the length-weighted baseline does not reduce variance, the paper's central mechanism is not doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that the variance-minimizing baseline for policy gradient in sequence generation reduces, under a plausible assumption, to a length-weighted average of rewards: $b^*(x) = \sum_i l_i r(x,y_i) / \sum_i l_i$. The derivation follows the classical optimal baseline formula, replacing the squared gradient magnitude with response length on the grounds that token gradients are approximately orthogonal and share a common norm distribution, making $\|\nabla_\theta \log \pi_\theta(y|x)\|^2$ proportional to $l_y$. Combined with exact on-policy updates—one gradient step per freshly sampled batch—this baseline yields stable training that needs no value model, reference model, or KL/entropy regularization, and it outperforms GRPO on MATH-500, AIME 2024, and AIME 2025 while producing more diverse and less repetitive outputs.

Load-bearing premise

The claim that the length-weighted baseline is the optimal variance reducer rests on the assumption that token-level gradients are approximately orthogonal and have identically distributed norms; if real language-model gradients violate this, Equation 12 is no longer variance-minimizing and OPO's theoretical advantage over simpler baselines is not guaranteed.

Editorial extensions

If this is right

  • LLM alignment can be done with a single policy model and no auxiliary value or reference models, removing a major source of memory and compute overhead.
  • The length-weighted baseline can replace group-mean or greedy baselines in other on-policy algorithms such as Reinforce++, and the paper shows preliminary gains in training reward and entropy.
  • Without KL or entropy regularization, training remains stable, which simplifies hyperparameter tuning and removes the need to balance those penalties.
  • On-policy training alone raises pass@1 on math benchmarks while keeping KL divergence to the initial policy low, indicating less alignment tax.
  • Output diversity improves, as measured by Self-BLEU and Rep-5, which matters for user-facing sampling quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A length-weighted baseline implicitly rewards longer responses more when longer responses correlate with higher reward; on math tasks this may function as a soft 'thinking-time' prior, and the authors' assumption predicts that its benefit should shrink on tasks where length and reward are anti-correlated.
  • The orthogonality assumption could be tested directly: measuring per-token gradient norms across a real policy should show whether $\|\nabla_\theta \log \pi_\theta(y|x)\|^2$ is actually proportional to $l_y$; if not, the theoretical grounding shifts to the empirical recipe.
  • The baseline's variance reduction should also apply to token-level or step-level rewards; a straightforward extension would give a step-wise length-weighted baseline for process supervision.
  • Because the baseline uses only within-prompt samples, its effectiveness may depend on group size $K$; small $K$ raises baseline estimation variance, and one could quantify the $K$-dependence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes On-Policy RL with Optimal reward baseline (OPO), a policy-gradient method for fine-tuning LLMs on reasoning tasks. OPO combines exact on-policy updates with a length-weighted reward baseline b*(x) = Σ l_i r_i / Σ l_i, which the authors claim is a practical form of the variance-minimizing baseline for sequence generation. The method removes value models, reference models, and KL/entropy regularization. Experiments on DeepSeek-R1-Distill-Qwen-7B on MATH-500, AIME 2024, and AIME 2025 compare on-policy vs off-policy GRPO and OPO vs GRPO, and report diversity and repetition metrics (Self-BLEU, Rep-5). The central theoretical claim is that under approximate token-gradient orthogonality and a shared per-token gradient norm distribution, the optimal baseline reduces to the length-weighted reward average.

Significance. If the method worked as claimed, OPO would be a useful simplification of LLM RL training, removing auxiliary models and regularizers while improving stability. That would be a practically valuable result, especially given the public implementation in the verl library and the clear algorithmic description. The standard derivation of the variance-minimizing baseline in Appendix A is correct for a fixed input, and the experimental setup isolates some comparisons between methods. However, the central theoretical step from the exact optimal baseline to the length-weighted form is not justified by the stated assumptions, the same-batch baseline introduces a finite-sample bias that is not discussed, and the empirical results are reported without uncertainty estimates. The paper's headline claim of 'theoretically sound' optimality is therefore not established, and the empirical evidence is weaker than the text suggests.

major comments (4)
  1. [Section 3.2, Eq. (10)] The step from Eq. (9) to Eq. (10) is not justified by the stated orthogonality and identical-distribution assumptions. Those assumptions imply E[||g(y)||^2 | l_y] ∝ l_y, not that the realized weight ||g(y)||^2 equals c·l_y, nor that this weight is independent of the reward r(x,y). Since b* in Eq. (9) is a ratio of expectations involving the reward-weighted squared gradient norm, replacing the random weight by its conditional expectation is an uncontrolled mean-field approximation. Unless Var(||g||^2 | l, r) is negligible, Eq. (12) does not minimize gradient variance. The paper provides no measurement of realized gradient norms or of gradient-estimator variance to support this approximation, so the 'theoretical soundness' claim made in the Introduction and Section 3.2 is unsupported.
  2. [Section 3.3, Eqs. (11)-(12)] The baseline b*(x) is computed from the same K responses whose log-probabilities appear in the policy-gradient sum, so the estimator is not the unbiased policy gradient. For the plain mean baseline, E[(1/K)Σ_i ∇logπ(y_i)(r_i − mean_j r_j)] = ((K−1)/K) E[∇logπ(y) r(y)], a K-dependent shrinkage. The length-weighted baseline has the same finite-sample issue, compounded by the dependence of the denominator on all sampled lengths. The paper neither uses a leave-one-out baseline nor corrects the resulting scale factor, yet K=8 or K=16 is small enough that this bias is non-negligible. This is a distinct issue from variance reduction and affects the interpretation of the optimized objective in Eq. (11).
  3. [Section 4.2, Tables 1 and 2] All reported results are single-run values averaged over the last five checkpoints, with no multiple seeds, error bars, or checkpoint-level variance. Several OPO/GRPO differences are small or in the opposite direction of the claimed improvement; for example, AIME 2025 pass@1 is 50.00 for OPO vs 50.21 for GRPO, and pass@2 is 60.88 vs 61.45. The diversity and repetition claims in Section 4.3 similarly rest on differences of a few tenths of a point in Self-BLEU and Rep-5. Without uncertainty quantification, the statement that OPO 'outperforms GRPO in most cases' is not statistically supported.
  4. [Section 4.1] The on-policy vs off-policy comparison is confounded by hyperparameter differences. On-policy training uses a mini-batch size of 256, off-policy uses 128, and off-policy includes a 0.001 entropy penalty while on-policy uses none. These differences, rather than exact on-policy training itself, could drive parts of the observed gap in Table 1. The comparison should hold these factors fixed or provide an ablation isolating the on-policy component.
minor comments (6)
  1. [Section 3.2 and Appendix A] Equations (7)-(9) and the appendix write (g(y))^2, but for a vector-valued gradient the variance objective should use the squared norm ||g(y)||^2; otherwise the scalar notation is ambiguous.
  2. [Section 4.1] There is a typo: 'reasponses' should be 'responses'.
  3. [Table 1 caption] The caption contains the typo 'comparision' instead of 'comparison'.
  4. [Section 4.3] The text says 'Tables 1 and 2 summarize the results' when discussing Rep-5 and Self-BLEU, but those metrics appear in Tables 3 and 4.
  5. [Figures 1-3] The figure captions in the manuscript contain raw '\uni0000...' character sequences that appear to be rendering artifacts; these should be cleaned before publication.
  6. [Table 2] The MATH-500 rows report pass@1 through pass@8 but not pass@16, making the table inconsistent with the AIME rows; consider adding the value or explaining its omission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the baseline derivation uses standard external results and an explicit approximation, and the empirical benchmarks are external to the method's construction.

full rationale

OPO's derivation chain is self-contained. Equation 9 is the textbook variance-minimizing baseline attributed to independent prior work [GBB04, WT01], and the simplification to Equation 10 proceeds from an explicitly stated assumption about token-gradient orthogonality and shared per-token norm distributions, not from the benchmark outcomes the paper later reports. Equation 12 is only the sample analogue of Equation 10, computed from rewards and lengths of the current responses, so no parameter is fitted to the evaluation data and then renamed a prediction. The paper contains no load-bearing self-citation: the cited work on exact on-policy training and baselines is external or non-essential, and no uniqueness theorem from the authors is invoked to forbid alternatives. The skeptic's concern that replacing the realized gradient norm with its conditional expectation is a mean-field approximation is a correctness and robustness question about whether Equation 12 truly minimizes variance under realistic conditions, not a circularity question, because the approximation is admitted as an assumption rather than derived from the desired result. The benchmarks, diversity metrics, and stability measurements are external evidence, so the central empirical claims are not forced by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The ledger is small: no numbers are fit to data and no new entities are introduced. The derivation leans on two standard results, the policy gradient theorem and the Weaver-Tao optimal baseline, plus one unverified paper-specific assumption about token gradient orthogonality. That assumption is the only genuinely fragile item in the theoretical chain.

assumptions (3)
  • standard math Policy gradient theorem and unbiasedness of baseline subtraction, g = E[∇log pi (r - b)].
    Invoked in Section 3.2, Eq 5-6, to justify the objective; this is the standard REINFORCE gradient.
  • standard math Optimal baseline formula b* = E[||g||^2 r] / E[||g||^2] from Weaver and Tao (2001) and Greensmith et al. (2004).
    Taken from cited prior work in Section 3.2, Eq 9; not re-derived from first principles in this paper.
  • ad hoc to paper Token gradients are approximately orthogonal and have the same norm distribution, so ||∇log pi(y|x)||^2 is proportional to l_y.
    Stated as 'a simple assumption' in Section 3.2 immediately before Eq 10. It is unverified and load-bearing for the claim that the length-weighted baseline is the optimal baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-Policy RL with Optimal Reward Baseline." pith.science (2026). https://pith.science/paper/R6VFQURK

@misc{pith2026250523585,
  author       = {Pith},
  title        = {Pith review of: On-Policy RL with Optimal Reward Baseline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6VFQURK}},
  note         = {Machine review of arXiv:2505.23585}
}
read the original abstract

Reinforcement learning algorithms are fundamental to align large language models with human preferences and to enhance their reasoning capabilities. However, current reinforcement learning algorithms often suffer from training instability due to loose on-policy constraints and computational inefficiency due to auxiliary models. In this work, we propose On-Policy RL with Optimal reward baseline (OPO), a novel and simplified reinforcement learning algorithm designed to address these challenges. OPO emphasizes the importance of exact on-policy training, which empirically stabilizes the training process and enhances exploration. Moreover, OPO integrates a practically feasible formulation of the optimal reward baseline that minimizes gradient variance. We evaluate OPO on mathematical reasoning benchmarks. The results demonstrate its superior performance and training stability without additional models or regularization terms. Furthermore, OPO achieves lower policy shifts and higher output entropy, encouraging more diverse and less repetitive responses. These results highlight OPO as a promising direction for stable and effective reinforcement learning in large language model alignment and reasoning tasks. The implementation is merged into the verl library at https://verl.readthedocs.io/en/latest/algo/opo.html.

Figures

Figures reproduced from arXiv: 2505.23585 by the authors.

Figure 1
Figure 1. Training dynamics of on-policy and off-policy training. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Left: Comparison of KL divergence and math performance between OPO and GRPO. Both OPO and GRPO follow the exact on-policy training from the SFT policy. The x-axis represents KL divergence, and the y-axis denotes math performance. Middle: Training dynamics of KL divergence. Right: Training dynamics of entropy. OPO achieves better performance and more stable training. We compare the training dynamics of on-policy and … view at source ↗
Figure 3
Figure 3. Training dynamics of OPO and Reinforce++. Both OPO and Reinforce++ follow the exact [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A segment rollout plus dynamic masking of confident positive tokens lets a 30B-A3B reasoning model beat a 235B-A22B model on AIME2025 and BeyondAIME after 128k-token RL training.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    [GYZ+25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  2. [5]

    Approximately optimal approximate reinforcement learning

    [KL02] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pages 267–274,

    work page 2002

  3. [7]

    Deepseek-v3 technical report

    [LFX+24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

    arXiv

  4. [8]

    American invitational mathematics examination - aime

    [MAA24] MAA. American invitational mathematics examination - aime. InAmerican Invitational Mathematics Examination - AIME 2024, February

    work page 2024

  5. [9]

    GPT4 technical report.arXiv preprint arXiv:2303.08774,

    [Ope23] OpenAI. GPT4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  6. [12]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    [SZY+24] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,

    arXiv

  7. [13]

    [TDG+25] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling rein- forcement learning with llms.arXiv preprint arXiv:2501.12599,

    arXiv

  8. [15]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    [YLY+25] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, , et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

Show all 15 references
  1. [1992]

    Neural text generation with unlikelihood training

    [WKR+20] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020,

    2020

  2. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    [SWZ+24] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  3. [2020]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    [SWD+17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  4. [2021]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    [ABW+23] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  5. [2022]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  6. [2024]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    [LCL+25] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    arXiv

  7. [2025]

    [Hu25] Jian Hu

    Notion Blog. [Hu25] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.