Pith. sign in

REVIEW 3 major objections 4 minor 47 references

For reinforcement learning with verifiable rewards, this paper derives the optimal learning-rate schedule and the variance-optimal baseline from a single upper bound on the policy-gradient loss, and proves convergence under that schedule.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:41 UTC pith:2ULUYJ4L

load-bearing objection The paper's core proof is built on an unjustified bound in Lemma A.4; the algorithm may be a useful heuristic, but the theory doesn't hold up and the experiments are too thin to rescue it. the 3 major comments →

arxiv 2511.23310 v4 pith:2ULUYJ4L submitted 2025-11-28 stat.ML cs.LG

Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

Zixun Huang , Jiayi Sheng , Zeyu Zheng This is my paper
classification stat.ML cs.LG
keywords RLVRpolicy gradientsignal-to-noise ratioadaptive learning ratevariance reductionbaseline designconvergence analysisLLM post-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to replace heuristic choices of learning-rate schedules and baselines in reinforcement learning with verifiable rewards (RLVR) by deriving both from one upper bound on the policy-gradient loss. Under assumptions that the log-likelihood is smooth, its gradient is uniformly square-integrable, and rewards are bounded, it proves that the loss satisfies a per-step descent inequality. Minimizing this inequality gives an optimal signal-to-noise-ratio-adaptive learning rate and shows the variance-optimal baseline is a gradient-weighted reward, not a plain average. If correct, this would give post-training a principled way to trust noisier gradients less and to weight baselines by contribution to the gradient, with a matching O(1/√T) convergence guarantee.

Core claim

The paper establishes that, when the reward is verifiable and the baseline is independent of the sampled output, the policy-gradient estimator is unbiased and its covariance splits into a per-sample term H(θ) and a cross-sample term C(θ). With two smoothness/boundedness assumptions and a bounded reward, the expected-reward objective J(θ)=E[F(q,o)] is shown to satisfy a per-step descent inequality (Theorem 3) with smoothing coefficient BL+B²M and variance term tr(H(θ_t))/N_t. Minimizing that inequality at each step yields the SNR-governed learning rate η_t = 1/(BL+B²M) · N_t·SNR(θ_t)/(1+N_t·SNR(θ_t)); the same bound gives an O(1/√T) convergence rate; and the baseline that minimizes trace vari

What carries the argument

The load-bearing machinery is the one-step upper bound on the loss (Theorem 3), proved by a smoothness lemma (Lemma A.4) that converts log-likelihood smoothness and a uniform L² bound on log-likelihood gradients into smoothness of the expected-reward objective. From this bound the paper derives (i) the SNR learning-rate rule, where SNR(θ)=||∇J(θ)||²/tr(H(θ)) measures true-gradient energy against per-sample variance; and (ii) the gradient-weighted baseline, which is the reward average weighted by squared gradient norms. The formula η_t = 1/(BL+B²M) · N_t·SNR_t/(1+N_t·SNR_t) is the direct output of minimizing the bound.

Load-bearing premise

The proof requires the log-likelihood gradients to have a finite L² integral uniformly over all parameters (an unstated bound in Lemma A.4's term II that is not implied by Assumptions 1–4), and the paper itself concedes that the L-smoothness assumption may not hold in practice; without these, the objective's smoothness coefficient, the loss upper bound, the SNR learning-rate rule, and the convergence rate are unproven.

What would settle it

Estimate M = ∫ sup_θ ||∇_θ log π_θ(o|q)||² dodq on an actual language model during RLVR by computing gradient norms of log-likelihoods across a prompt-response corpus; if this quantity is unbounded, grows with model scale, or exceeds the value needed for the Cauchy–Schwarz bound in Lemma A.4, then the smoothness coefficient BL+B²M is not valid and the O(1/√T) guarantee is not established. In addition, a controlled run comparing the SNR schedule to any fixed-rate schedule on a suite of reasoning tasks would directly test the rule's predicted advantage.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At any training step, the right step size is computable from an estimate of the gradient SNR, scaling with the number of queries times the SNR and saturating at 1/(BL+B²M).
  • The variance-minimizing baseline for output-independent baselines is the reward weighted by squared gradient norms of the policy, not the group-average reward.
  • Using the SNR schedule, the average squared gradient norm converges at rate O(1/√T), matching standard stochastic gradient descent but under a smoothness coefficient that grows linearly with the reward bound and the L² gradient bound.
  • Each ingredient—SNR learning rate and gradient-weighted baseline—improves policy-optimization performance by itself on a 4-billion-parameter language model, and the combination performs best.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the SNR rule is estimated online, the same formula gives a principled batch-size rule—doubling the number of queries per step raises the effective SNR and automatically permits a larger step.
  • Beyond the paper: the gradient-weighted baseline could be applied at token level for step-wise advantages in RLVR, weighting each token's reward by the gradient norm of its log-probability; this extension is not analyzed here but follows the same quadratic-variance reasoning.
  • Beyond the paper: the authors' conclusion notes that the guarantees concern the upper bound, not the realized loss, and that smoothness is unverified; the strongest testable prediction is that the SNR rule improves convergence most in the low-SNR (high-noise) regime, where the schedule differs most from a fixed rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops a theoretical framework for policy-gradient reinforcement learning with verifiable rewards (RLVR). It defines the objective J(θ)=E[F(q,o)], derives an unbiased gradient estimator, gives an exact variance decomposition in terms of a single-sample covariance H(θ) and a cross-sample covariance C(θ), and states an upper bound on the expected optimization loss under smoothness and boundedness assumptions (Theorem 3). From that bound it derives an SNR-adaptive learning-rate schedule (Theorem 4), a convergence rate O(1/√T) (Theorem 6), and a gradient-weighted variance-optimal baseline (Theorem 7). The proposed algorithm OBLR-PO combines the adaptive learning rate with a leave-one-out estimated baseline. Experiments on Qwen3-4B-Base and Qwen3-8B-Base compare OBLR-PO with GRPO, PPO, ReMax, and RLOO, reporting improved accuracy and stability.

Significance. If the theoretical results were valid, the paper would make a useful contribution to RLVR by providing a principled alternative to heuristic baseline and learning-rate choices, with an end-to-end convergence guarantee. The variance decomposition and the closed-form baseline minimization are clean and potentially transferable. The empirical evaluation covers several baselines and two model sizes, and the reported gains, while modest, are consistent across many benchmarks. However, the central upper-bound proof contains a mathematical error in a load-bearing lemma, and the algorithm's SNR estimator is not aligned with the theoretical SNR. The current manuscript therefore does not establish its main theoretical claims.

major comments (3)
  1. [Appendix A.2, Lemma A.4, Eqs. (67)-(69)] The bound on term II is invalid. After the mean-value step the integrand is ∥∇_θ f(x, θ̃(x))∥ · ∥∇_θ log f(x, θ′)∥, and the next inequality replaces the first factor by B(x):=sup_θ f(x,θ). Even if this replacement were legitimate, the conclusion ∫ B(x)^2 dx ≤ M does not follow from Assumption 3, which bounds ∫ sup_θ ∥∇_θ log f∥² dx; no uniform L² bound on the density or on ∇_θ f is stated. In addition, the lemma is applied to F(q,o)π_θ(o|q), which is signed and not normalized, so the step ∫ f dx = 1 used in Eq. (66) is unavailable. Consequently the (BL+B²M)-smoothness of J and the upper bound in Theorem 3, and hence Theorems 4-6, are not proved as written.
  2. [Section 5, Eqs. (38)-(41)] The SNR estimator used in the algorithm does not match the theoretical SNR defined in Eq. (34). The estimator \widehat{SNR} is formed from \widehat{μ}_t and \widehat{σ}_t², but \widehat{μ}_t is computed from F(q,o) without the baseline, whereas the SNR in Eq. (34) uses the advantage F(q,o)-b_θ(q) and the covariance H(θ). Moreover, \widehat{μ}_t is a noisy sample average, so ∥\widehat{μ}_t∥² is not an unbiased estimate of the squared gradient norm, and the plug-in ratio is not an estimator of E∥∇L∥²/tr(H). The replacement of 1/(BL+B²M) by an ad-hoc η_0 is also not derived from the theory. Thus the adaptive learning-rate rule used in OBLR-PO is only loosely connected to Theorem 4.
  3. [Section 4.4 and Algorithm 1, Eq. (42)] Assumption 1 and Theorem 7 treat the baseline b_θ(q) as fixed and independent of the sampled outputs. The algorithm, however, estimates the baseline from the remaining G_t-1 outputs via leave-one-out (Eq. 42), making the baseline random and statistically dependent on the gradient estimator. The unbiasedness and variance results (Theorems 1 and 2) are not established for this leave-one-out estimator. To claim that OBLR-PO inherits the theoretical variance-reduction property, the paper needs an additional analysis of the estimator with a data-dependent baseline.
minor comments (4)
  1. [Abstract and Section 3.1] The abstract promises a 'KL-regularized policy-optimization setting', but the objective in Eq. (1) contains no KL penalty. Please clarify the discrepancy.
  2. [Proof of Theorem A.3, before Eq. (79)] The statement 'ξ(θ_t) is independent of θ_t' is not correct; ξ is a function of samples drawn from π_{θ_t}. The argument only needs the conditional unbiasedness E[ξ(θ_t)|θ_t]=0. This should be stated and used correctly.
  3. [Experiments, Table 2 and Figures 1-3] No error bars, confidence intervals, or number of seeds are reported. As the improvements over baselines are small (e.g., 24.1% vs 23.5% on OlympiadBench), it is important to indicate whether the results are stable across runs.
  4. [Eq. (20)] The notation π_{θ_t} appears without definition; it should likely be π_θ.

Circularity Check

0 steps flagged

No significant circularity; the derivation is bound-minimization over independently derived variance expressions, not a self-referential fit.

full rationale

The derivation chain is not circular. Theorem 3 is a proved upper bound on E[L(theta_T)] under Assumptions 1-4; Theorem 4 minimizes that bound, and the paper states this explicitly: 'we instead consider minimizing the upper bound provided in Theorem 3.' The conclusion also acknowledges the limitation: 'our guarantees are given with respect to an upper bound on the loss, leaving a gap to the realized optimization dynamics.' Thus the 'optimal' learning rate is relative to the paper's own bound, which is a standard bound-minimization step and not a tautological prediction. Theorem 7 minimizes tr(H(theta)), an independently derived exact variance expression, and the resulting baseline is a genuine quadratic minimizer rather than an input renamed as an output. No fitted constants are used to derive the main theorems, and estimated quantities enter only in the practical algorithm, not in the proof of the optimal schedule. The only author self-citation ([32]) appears in a general RL-motivation citation list and plays no role in any load-bearing proof step. The reader-identified gap in Lemma A.4 - specifically bounding term II at lines (67)-(68) by ||theta'-theta|| times integral of B(x)^2 with B(x):=sup_theta f(x,theta), a quantity not controlled by Assumption 3 - is a proof-correctness issue requiring an additional integrability condition, not a circularity: the missing bound is a new assumption, not a restatement of Theorem 3 or of any fitted value. Therefore the paper's central claims, while possibly resting on an incomplete proof, do not reduce by construction to their own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claims rest on four stated assumptions plus one hidden bound that appears only in the proof of Lemma A.4. No parameters are fit to the target benchmark results; experimental hyperparameters (η0, G, N, T) are hand-chosen.

free parameters (4)
  • Initial learning rate η0 = 1e-2 (chosen by hand)
    The adaptive LR formula (Eq 31) contains unknown constants L, B, M; the algorithm folds them into a hand-set η0 and multiplies by N_t·SNR/(1+N_t·SNR). The choice 1e-2 is not derived.
  • Group size G_t = 8
    Experimental hyperparameter; theoretical bounds treat it as input but the algorithm uses G=8.
  • Batch size N_t = 128
    Experimental hyperparameter; theoretical schedule scales with N_t.
  • Training steps T = 60
    Experimental choice; all reported curves use 60 steps.
axioms (5)
  • domain assumption Assumption 1: advantage is F(q,o) − b_θ(q) with b_θ(q) fixed per query and independent of the output o
    Underlies unbiasedness (Eq 20). PPO/GRPO violate it (Table 1); OBLR-PO's leave-one-out baseline satisfies it only conditional on the other samples.
  • domain assumption Assumption 2: log π_θ(o|q) is L-smooth in θ for all q,o
    Needed for the Taylor bound; the paper admits this may not hold in practice (Conclusion).
  • domain assumption Assumption 3: ∫ sup_θ ||∇_θ log π_θ(o|q)||² do dq ≤ M
    Uniform bound on gradient-norm integrals; strong, unusual, and not verified for LLM policies.
  • domain assumption Assumption 4: |F| ≤ B and |b_θ| ≤ B
    Bounded rewards; typical for verifiable rewards.
  • ad hoc to paper Hidden assumption in Lemma A.4: ∫ (sup_θ π_θ(o|q))² do dq ≤ M (or an equivalent L∞ bound on sup densities)
    The proof bounds term II by ∫B(x)² dx ≤ M where B(x)=sup_θ f(x,θ); no such bound is stated in Assumptions 1-4 and it does not follow from them.

pith-pipeline@v1.3.0-alltime-deepseek · 68 in / 19744 out tokens · 168562 ms · 2026-08-04T06:41:20.925102+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective paradigm for post-training large language models, yet the design of its baselines and learning-rate schedules remains largely heuristic. This limits our understanding of the statistical properties of policy-gradient estimators and their interaction with optimization dynamics. In this work, we develop a theoretical framework for variance-aware baseline design and adaptive learning-rate selection in RLVR. Under a KL-regularized policy-optimization setting, we establish the unbiasedness of the resulting gradient estimator, derive exact variance expressions including the KL cross-covariance, and obtain an optimization-loss upper bound that enables principled reasoning about learning dynamics. Building on these results, we prove convergence guarantees and derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of the policy gradient. We further show that the variance-optimal baseline is a gradient-weighted estimator of the KL-regularized reward, providing a principled alternative to commonly used reward-based baselines. These results lead to two complementary improvements: a variance-optimal baseline and an SNR-adaptive learning-rate rule. Experiments on Qwen3-4B-Base show that each component independently improves policy-optimization performance. The learning-rate rule can also be naturally integrated with existing policy optimization methods to yield further gains, while combining it with the variance-optimal baseline gives the full Optimal Baseline and Learning-Rate Policy Optimization (OBLR-PO) method and achieves the strongest overall performance.

Figures

Figures reproduced from arXiv: 2511.23310 by Jiayi Sheng, Zeyu Zheng, Zixun Huang.

Figure 1
Figure 1. Figure 1: The left figure shows the Advantage dur [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The left figure shows the Gradient Norm dur [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The left figure shows the Loss across train [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Advantages observed for Qwen3-4B-Base (left) and Qwen3-8B-Base (right) during training. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Gradient norm curves of Qwen3-4B-Base (left) and Qwen3-8B-Base (right) during training. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: KL loss trajectories of Qwen3-4B-Base (left) and Qwen3-8B-Base (right) during training. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Entropy trajectories of Qwen3-4B-Base (left) and Qwen3-8B-Base (right) during training. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 24 linked inside Pith

  1. [1]

    Arash Ahmadian et al.Back to Basics: Revisit- ing REINFORCE Style Optimization for Learn- ing from Human Feedback in LLMs. 2024. arXiv: 2402 . 14740 [cs.LG].url:https : / / arxiv . org/abs/2402.14740

    Pith/arXiv arXiv 2024

  2. [2]

    Yuntao Bai et al.Training a Helpful and Harm- less Assistant with Reinforcement Learning from Human Feedback. 2022. arXiv:2204 . 05862 [cs.CL].url:https://arxiv.org/abs/2204. 05862

    2022

  3. [3]

    Yoshua Bengio.Practical recommendations for gradient-based training of deep architectures

  4. [4]

    Shane Bergsma et al.Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs. 2025. arXiv:2502.15938 [cs.LG].url:https://arxiv.org/abs/2502. 15938

    arXiv 2025

  5. [5]

    Kiant´ e Brantley et al.Accelerating RL for LLM Reasoning with Optimal Advantage Regression

  6. [6]

    Roger Creus Castanyer et al.Stable Gradi- ents for Stable Learning at Scale in Deep Re- inforcement Learning. 2025. arXiv:2506.15544 [cs.LG].url:https://arxiv.org/abs/2506. 15544

    arXiv 2025

  7. [7]

    Paul Christiano et al.Deep reinforcement learn- ing from human preferences. 2023. arXiv:1706. 03741 [stat.ML].url:https://arxiv.org/ abs/1706.03741

    Pith/arXiv arXiv 2023

  8. [8]

    Karl Cobbe et al.Training Verifiers to Solve Math Word Problems. 2021. arXiv:2110.14168 [cs.LG].url:https://arxiv.org/abs/2110. 14168

    Pith/arXiv arXiv 2021

  9. [9]

    Learning rate schedules for faster stochas- tic gradient search

    Christian Darken, Joseph Chang, John Moody, et al. “Learning rate schedules for faster stochas- tic gradient search”. In:Neural networks for sig- nal processing. Vol. 2. Citeseer Helsinoger, Den- mark. 1992, pp. 3–12

    1992

  10. [10]

    Note on learning rate schedules for stochastic optimiza- tion

    Christian Darken and John Moody. “Note on learning rate schedules for stochastic optimiza- tion”. In:Advances in neural information pro- cessing systems3 (1990)

    1990

  11. [11]

    DeepSeek-AI et al.DeepSeek-R1: Incentiviz- ing Reasoning Capability in LLMs via Rein- forcement Learning. 2025. arXiv:2501 . 12948 [cs.CL].url:https://arxiv.org/abs/2501. 12948

    2025

  12. [12]

    Variance Reduction Tech- niques for Gradient Estimates in Reinforce- ment Learning

    Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. “Variance Reduction Tech- niques for Gradient Estimates in Reinforce- ment Learning”. In:Journal of machine learn- ing research. 2001.url:https : / / api . semanticscholar.org/CorpusID:5259564

    2001

  13. [13]

    Chaoqun He et al.OlympiadBench: A Chal- lenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. 2024. arXiv:2402.14008 [cs.CL]

    Pith/arXiv arXiv 2024

  14. [14]

    ∆LNormalization: Rethink Loss Aggregation in RL VR

    Zhiyuan He et al. ∆LNormalization: Rethink Loss Aggregation in RL VR. 2025. arXiv:2509. 07558 [cs.LG].url:https://arxiv.org/abs/ 2509.07558

    arXiv 2025

  15. [15]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks et al. “Measuring Mathematical Problem Solving With the MATH Dataset”. In: arXiv preprint arXiv:2103.03874(2021)

    Pith/arXiv arXiv 2021

  16. [16]

    Jian Hu et al.REINFORCE++: An Effi- cient RLHF Algorithm with Robustness to Both Prompt and Reward Models. 2025. arXiv:2501. 03262 [cs.CL].url:https://arxiv.org/abs/ 2501.03262

    Pith/arXiv arXiv 2025

  17. [17]

    Buy 4 REINFORCE Samples, Get a Baseline for Free!

    Wouter Kool, Herke van Hoof, and Max Welling. “Buy 4 REINFORCE Samples, Get a Baseline for Free!” In:DeepRLStructPred@ICLR. 2019. url:https : / / api . semanticscholar . org / CorpusID:198489118

    2019

  18. [18]

    Binghui Li et al.Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learn- ing Rate Schedules. 2025. arXiv:2509 . 19189 [cs.LG].url:https://arxiv.org/abs/2509. 19189

    2025

  19. [19]

    An exponen- tial learning rate schedule for deep learning

    Zhiyuan Li and Sanjeev Arora. “An exponen- tial learning rate schedule for deep learning”. In: arXiv preprint arXiv:1910.07454(2019)

    Pith/arXiv arXiv 1910

  20. [20]

    Ziheng Li et al.Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability- Adaptive Hint Scaffolding. 2025. arXiv:2509 . 06923 [cs.LG].url:https://arxiv.org/abs/ 2509.06923. OBLR-PO: A Theoretical F ramework for Stable Reinforcement Learning

    Pith/arXiv arXiv 2025

  21. [21]

    Ziniu Li et al.ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. 2024. arXiv: 2310 . 10505 [cs.LG].url:https : / / arxiv . org/abs/2310.10505

    Pith/arXiv arXiv 2024

  22. [22]

    Let’s Verify Step by Step

    Hunter Lightman et al. “Let’s Verify Step by Step”. In:arXiv preprint arXiv:2305.20050 (2023)

    Pith/arXiv arXiv 2023

  23. [23]

    Hugging Face, 2023

    math-ai.AMC23: American Mathematics Com- petitions 2023 problems and answers. Hugging Face, 2023

    2023

  24. [24]

    Long Ouyang et al.Training language models to follow instructions with human feedback. 2022. arXiv:2203 . 02155 [cs.CL].url:https : / / arxiv.org/abs/2203.02155

    Pith/arXiv arXiv 2022

  25. [25]

    Lei Pang and Ruinan Jin.On the Theory and Practice of GRPO: A Trajectory-Corrected Ap- proach with Fast Convergence. 2025. arXiv: 2508 . 02833 [cs.LG].url:https : / / arxiv . org/abs/2508.02833

    arXiv 2025

  26. [26]

    Rafael Rafailov et al.Direct Preference Opti- mization: Your Language Model is Secretly a Re- ward Model. 2024. arXiv:2305.18290 [cs.LG]. url:https://arxiv.org/abs/2305.18290

    Pith/arXiv arXiv 2024

  27. [27]

    Nicolas Le Roux et al.Tapered Off-Policy RE- INFORCE: Stable and efficient reinforcement learning for LLMs. 2025. arXiv:2503 . 14286 [cs.LG].url:https://arxiv.org/abs/2503. 14286

    2025

  28. [29]

    John Schulman et al.Proximal Policy Opti- mization Algorithms. 2017. arXiv:1707.06347 [cs.LG].url:https://arxiv.org/abs/1707. 06347

    Pith/arXiv arXiv 2017

  29. [30]

    2024.url:https://arxiv.org/ abs/2402.03300

    Zhihong Shao et al.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Lan- guage Models. 2024.url:https://arxiv.org/ abs/2402.03300

    Pith/arXiv arXiv 2024

  30. [31]

    HybridFlow: A Flexi- ble and Efficient RLHF Framework

    Guangming Sheng et al. “HybridFlow: A Flexi- ble and Efficient RLHF Framework”. In:arXiv preprint arXiv: 2409.19256(2024)

    Pith/arXiv arXiv 2024

  31. [32]

    Solving Inequality Proofs with Large Language Models

    Jiayi Sheng et al. “Solving Inequality Proofs with Large Language Models”. In:The 39th Conference on Neural Information Processing Systems (NeurIPS). 2025

    2025

  32. [33]

    Nisan Stiennon et al.Learning to summarize from human feedback. 2022. arXiv:2009.01325 [cs.CL].url:https://arxiv.org/abs/2009. 01325

    Pith/arXiv arXiv 2022

  33. [34]

    Xuerui Su et al.Trust Region Preference Ap- proximation: A simple and stable reinforcement learning algorithm for LLM reasoning. 2025. arXiv:2504 . 04524 [cs.LG].url:https : / / arxiv.org/abs/2504.04524

    Pith/arXiv arXiv 2025

  34. [35]

    Christian Walder and Deep Karkhanis.Pass@K Policy Optimization: Solving Harder Reinforce- ment Learning Problems. 2025. arXiv:2505 . 15201 [cs.LG].url:https://arxiv.org/abs/ 2505.15201

    Pith/arXiv arXiv 2025

  35. [36]

    Self-Instruct: Aligning Language Models with Self-Generated Instruc- tions

    Yizhong Wang et al. “Self-Instruct: Aligning Language Models with Self-Generated Instruc- tions”. In:Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ed. by Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki. Toronto, Canada: Association for Com- putational Linguistics, July 2023, pp. 13484–...

    doi:10.18653/v1/2023.acl-long.754 2023

  36. [37]

    Zengzhi Wang et al.OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

  37. [38]

    An Yang et al.Qwen3 Technical Report. 2025. arXiv:2505 . 09388 [cs.CL].url:https : / / arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  38. [39]

    Jiarui Yao et al.Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL. 2025. arXiv: 2505 . 02391 [cs.LG].url:https : / / arxiv . org/abs/2505.02391

    Pith/arXiv arXiv 2025

  39. [40]

    arXiv:2506.20512 [cs.CL].url:https: //arxiv.org/abs/2506.20512

    Pith/arXiv arXiv

  40. [41]

    Ruiqi Zhang et al.SPEED-RL: Faster Train- ing of Reasoning Models via Online Curriculum Learning. 2025. arXiv:2506 . 09016 [cs.LG]. url:https://arxiv.org/abs/2506.09016

    arXiv 2025

  41. [42]

    Yifan Zhang and Team Math-AI.American Invitational Mathematics Examination (AIME)

  42. [43]

    Yi-Fan Zhang et al.R1-Reward: Training Mul- timodal Reward Model Through Stable Rein- forcement Learning. 2025. arXiv:2505 . 02835 [cs.CV].url:https://arxiv.org/abs/2505. 02835

    2025

  43. [46]

    Banghua Zhu et al.Fine-Tuning Language Mod- els with Advantage-Induced Policy Alignment

  44. [48]

    Formally, we consider min {Nt}T−1 t=0 ,{G t}T−1 t=0 E L(θ T ) s.t

    + 4T B2M .(101) And thus, we have T−1X t=0 E∥∇θL(θ t)∥2 2 ≲ 1 + √ 1 + 16T B2M 2 ≤1 + √ 4T B2M≲ √ T .(102) Thus, we have 1 T T−1X t=0 E∥∇θL(θ t)∥2 2 =O 1√ T (103) OBLR-PO: A Theoretical F ramework for Stable Reinforcement Learning A.4 Proofs for Section 4.4 Theorem A.9.The optimal baseline is given by bθ(q) = Eo∼πθ(·|q) ∥∇θ logπ θ(o|q)∥2 2F(q, o) Eo∼πθ(·|q...

  45. [2012]

    arXiv:1206.5533 [cs.LG].url:https: //arxiv.org/abs/1206.5533

    Pith/arXiv arXiv

  46. [2023]

    arXiv:2306.02231 [cs.CL].url:https: //arxiv.org/abs/2306.02231. OBLR-PO: A Theoretical F ramework for Stable Reinforcement Learning Appendix A Proofs for Section 4 A.1 Proofs for Section 4.1 Theorem A.1(Unbiasedness).The approximate gradient \∇θ[J(θ)]is an unbiased estimator, i.e., E h \∇θ[J(θ)] i =∇ θ[J(θ)].(43) Proof.By the definition of the approximate...

    Pith/arXiv arXiv

  47. [2025]

    arXiv:2505.20686 [cs.LG].url:https: //arxiv.org/abs/2505.20686

    Pith/arXiv arXiv