Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

BNPO: Beta Normalization Policy Optimization

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Beta-shaped adaptive normalizer provably cuts policy-gradient variance.

desk verdict The adaptive Beta-normalization idea is novel and clearly written, but the proof of the central variance-reduction theorem drops the score-function norm, making the claimed optimal parameters unsupported. read the letter →

arxiv 2506.02864 v1 pith:7ZCBIX56 submitted 2025-06-03 cs.LG cs.AI

Changyi Xiao , Mengdi Zhang , Yixin Cao This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningpolicyoptimizationBetadistributionvariancereductionrewardnormalizationreasoningtasksLLMalignmentgrouprelative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BNPO is a policy-gradient method for training reasoning models on binary rule-based rewards. Its central claim is that using an advantage normalized by a Beta density whose parameters adapt to the current reward-expectation distribution lowers gradient-estimator variance, with the unique minimum at α = 1 + a/3 and β = 1 + b/3. The method reproduces REINFORCE-with-baseline at (α, β) = (1, 1) and GRPO at (3/2, 3/2), so the dynamic adaptation is the entire difference. On Qwen2.5-Math models fine-tuned on MATH, BNPO posts the best average pass@1 on four math benchmarks and smoother gradient norms, which the paper takes as evidence of its variance-reducing property.

What carries the argument

The load-bearing object is the BNPO gradient estimator $g_{\alpha,\beta} = \nabla_\theta \log \pi(o|q)\, (R(q,o)-p(q))/f_N(p(q); \alpha, \beta)$, combined with the variance identity that follows once the score-function norm is set aside. That identity expresses the variance as proportional to $B(\alpha,\beta)^2 \cdot B(a+3-2\alpha, b+3-2\beta) / B(a,b)$, where $a$ and $b$ come from the assumed Beta distribution of per-question success probabilities. Minimizing this Beta-function product gives the parameter rule $\alpha = 1 + a/3$, $\beta = 1 + b/3$, and the proof uses trigamma monotonicity to show the Hessian is positive definite, making the minimum unique.

What would settle it

On a question with a small enough action space to enumerate all outputs, compute the exact gradient-estimator variance for a grid of $(\alpha,\beta)$ values, using the true score function rather than the uncorrelatedness assumption; if the measured minimum is not at $(1+a/3, 1+b/3)$, the theorem's practical conclusion is falsified.

Watch

Extended reading notes

Core claim

BNPO defines the advantage $A_{\alpha,\beta}(q,o) = (R(q,o) - p(q))/f_N(p(q); \alpha, \beta)$, where $R$ is the binary outcome reward, $p(q)$ is the success probability of the current policy on question $q$, and $f_N$ is a Beta density used as the normalization term. The paper proves that when $p(q)$ follows a Beta$(a,b)$ distribution and the policy score function is uncorrelated with the normalized reward residual, the estimator's variance is finite iff $\alpha < (a+3)/2$ and $\beta < (b+3)/2$, and it attains a unique minimum at $\alpha = 1 + a/3$, $\beta = 1 + b/3$. The variance expression reduces to a product of Beta functions, $B(\alpha,\beta)^2 \, B(a+3-2\alpha, b+3-2\beta)/B(a,b)$, and the convexity proof certifies uniqueness via trigamma inequalities. BNPO generalizes REINFORCE-with-baseline ($\alpha=\beta=1$) and GRPO ($\alpha=\beta=3/2$), and the paper reports state-of-the-art average pass@1 across MATH500, AMC23, AIME2024, and AIME2025 with two Qwen2.5-Math base models.

Load-bearing premise

The proof assumes the score function $\nabla_\theta \log \pi(o|q)$ is uncorrelated with the normalized reward residual $(R(q,o)-p(q))/f_N(p(q);\alpha,\beta)$, so the variance comparison treats the squared score norm as constant; if this correlation is not zero, the claimed optimal $(\alpha,\beta)$ may not be the variance-minimizing choice.

Editorial extensions

If this is right

  • When the theorem's assumptions hold, the adaptive parameters $(1+a/3, 1+b/3)$ strictly dominate any fixed normalization, including GRPO's $(3/2,3/2)$, in variance of the policy-gradient estimate.
  • Because BNPO reduces to REINFORCE-with-baseline and to GRPO at specific $(\alpha,\beta)$ values, any improvement over those baselines is attributable to the dynamic adaptation of the normalization rather than a different objective.
  • The advantage-decomposition rule permits multiple binary rewards, such as accuracy and format, to be normalized separately; the paper reports small average gains when a format reward is added.
  • On the studied math benchmarks, BNPO achieves the best average pass@1 among the compared methods for both Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, with smoother gradient norms during training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to replace the Beta prior over $p(q)$ with a non-conjugate prior and see whether a similar closed-form optimum exists; the paper's Beta-function argument would not carry over directly.
  • Because the parameter rule only needs estimates of the mean and variance of $p(q)$, the same normalization could be applied to continuous rewards by treating each question's reward residual as a stand-in for $p(q)$, a step the paper leaves to future work.
  • The theorem's uncorrelatedness assumption could be checked empirically on a small policy; if the score norm correlates with the reward residual, the variance-minimum search would need to include the full score function to locate the true optimum.
  • If the measured variance reductions reproduce across model families, BNPO would be a drop-in replacement for GRPO in R1-style training, since it adds only moment estimation to the rollout loop and removes the need for a critic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes BNPO (Beta Normalization Policy Optimization), a modification of policy-gradient RL for LLM reasoning with binary rewards. The method defines the advantage as A(q,o)=(R(q,o)-p(q))/f_N(p(q);alpha,beta), where p(q) is the expected reward for question q and f_N is a Beta density with parameters adapted during training. The central theoretical claim (Theorem 1) is that, under an uncorrelatedness assumption, the variance of the BNPO gradient estimator is minimized uniquely at alpha=1+a/3 and beta=1+b/3, where (a,b) are the parameters of a Beta distribution modeling p(q). The paper also claims BNPO generalizes REINFORCE with baseline and GRPO in the binary-reward setting, introduces an advantage-decomposition extension for multiple binary rewards, and reports experiments on MATH training with Qwen2.5-Math-1.5B/7B showing the highest average pass@1 among compared methods.

Significance. If Theorem 1 were correct, BNPO would be a simple, principled upgrade to GRPO with an adaptive normalization mechanism, and the public code would facilitate reproducibility. The paper also makes a falsifiable empirical claim of state-of-the-art performance on reasoning benchmarks. However, the theoretical derivation contains a load-bearing algebraic error, and the empirical evidence is too weak to support the state-of-the-art claim on its own. The contribution therefore does not currently meet the standard for publication in a serious venue.

major comments (4)
  1. [Appendix A, Step 1 (Variance Expression)] The variance expansion is correct up to Var(g)=E[(grad log pi)^2 ((R-p)/f_N)^2] - E[grad log pi (R-p)/f_N]^2, but the next sentence, "Under the assumption, the variance ... is proportional to E[(R-p)^2/f_N^2]", is invalid. Uncorrelatedness of X and Y (E[XY]=0) does not imply E[X^2Y^2]=E[X^2]E[Y^2]; the latter requires independence or an explicit condition on the conditional second moment. The exact quantity is E_o[(grad log pi)^2 (R-p)^2/f_N^2 | q], which generally depends on how the score-function norm varies with the reward outcome. For example, in a two-action logistic policy with R=1 for the correct action and R=0 otherwise, this conditional expectation is p(1-p)[(1-p)^3+p^3]/f_N^2 times a positive constant, not a constant multiple of p(1-p)/f_N^2. Therefore the claimed minimum at alpha=1+a/3, beta=1+b/3 does not follow.
  2. [Theorem 1] The standing assumption that grad log pi(o|q) is uncorrelated with (R-p(q))/f_N(p(q);alpha,beta) is self-defeating for a policy-gradient estimator. Under this assumption E[g_alpha,beta]=0, whereas a useful estimator should have expectation equal to the policy gradient (or a non-zero weighted gradient). Thus the proof uses an assumption that is generally false for exactly the setting the method is meant to address. This is not a minor technicality; it is what allows the squared-mean term to be dropped, and it cannot be repaired by adding a short derivation within the current framework.
  3. [Section 5.2, Table 1] The abstract's claim that BNPO "achieves state-of-the-art performance among policy optimization methods" is not supported by the reported numbers. On Qwen2.5-Math-1.5B, BNPO's average is 39.4 versus 39.0 for REINFORCE and GRPO; on Qwen2.5-Math-7B, BNPO's average is 47.8 versus 47.6 for ReMax, while on MATH500 BNPO is below GRPO (77.0 versus 78.6). These differences are within the range of typical run-to-run variation, and no error bars, multiple seeds, or significance tests are reported. The empirical evidence is therefore consistent with BNPO being equivalent to existing methods rather than superior.
  4. [Section 3, parameter estimation and Eq. (13)-(14)] The theorem's conclusion depends on the modeling assumption that p(q) follows a Beta distribution f_D(p;a,b), but this assumption is not validated against the data. The method-of-moments estimators for (a,b) are computed from a finite batch of Monte Carlo estimates of p(q), and the optimality of alpha=1+a/3 and beta=1+b/3 is only derived under this unverified parametric assumption. If the distribution of p(q) deviates from Beta, the variance expression and the claimed minimizer both change; the paper provides no diagnostic or robustness check for this assumption.
minor comments (6)
  1. [Eq. (16)] The reduction to GRPO is stated as an equality involving the population quantities p(q) and sqrt(p(q)(1-p(q))), but actual GRPO uses sample mean and sample standard deviation over a finite group. The claim that BNPO "reduces to GRPO" therefore holds only in the infinite-sample limit, and the finite-sample equivalence in Eq. (16) should be qualified.
  2. [Appendix A, Step 1] The text contains a typo: "Unper p ~ f_D(p(q);a,b)" should read "Under p ~ f_D(p(q);a,b)".
  3. [Section 3, after Eq. (15)] The phrase "REINFROCE with baseline" contains a spelling error; it should be "REINFORCE with baseline".
  4. [Section 6, Limitations] The sentence "Extending the theory in this way is a area of ongoing exploration" contains a grammatical error; it should be "is an area of ongoing exploration".
  5. [Conclusion] The sentence "We also that BNPO can reduces to REINFORCE with baseline and GRPO" is missing a verb and should be corrected to "We also show that BNPO can reduce to ...".
  6. [Figure 3] The caption says the figure presents the evolution of (E[p(q)], Var[p(q)], alpha, beta), but the single y-axis labeled "Value" does not distinguish these four quantities; a legend or separate panels would improve readability.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the alpha=1+a/3 optimum is derived from an explicit variance proxy, the REINFORCE/GRPO reductions are exact algebra, and the empirical results are externally benchmarked; only Section 5.4's 'validation' of parameter adaptation is tautological by construction (minor, non-load-bearing). Theorem 1's dropped score-function term is a validity gap, not circularity.

  1. self definitional [Section 5.4 (Evolution of Normalization), supported by Eq. (13) and Algorithm 1, lines 14-15]
    "As shown, the parameters α and β effectively adapt in response to changes in the mean and variance of p(q), validating the adaptive capability of BNPO's normalization mechanism."

    By construction, α = 1 + a/3 and β = 1 + b/3 (Algorithm 1, line 15), where (a,b) are the method-of-moments functions of E[p(q)] and Var[p(q)] (Eq. 13). So α and β are deterministic transforms of the statistics plotted in Figure 3, and co-movement with E[p] and Var[p] is entailed by the defining equations; the figure 'validates' a tautology, not an empirical property. The 'interpretation' in Section 3 (mode of f_N equals mean of f_D) is the same algebraic identity. This element is not load-bearing: the reductions to REINFORCE (α=β=1) and GRPO (α=β=3/2) and the benchmark comparisons on external test sets stand independently.

full rationale

The derivation chain is largely self-contained. Theorem 1's optimum α=1+a/3, β=1+b/3 is obtained by minimizing the analytic expression B(α,β)^2·B(a+3−2α, b+3−2β)/B(a,b); the minimizer is neither fitted to empirical gradient variances nor assumed, and the (a,b) method-of-moments estimates (Eqs. 12-14) plus α=1+a/3 (Algorithm 1) are downstream of, not inputs to, that minimization. This distinguishes the result from a fitted parameter renamed as a prediction. The reductions to prior methods are exact algebra: f_N(p;1,1)=1 yields A=R−p (Eq. 15), and f_N(p;3/2,3/2)=(8/π)√(p(1−p)) yields A∝(R−p)/√(p(1−p)) (Eq. 16), matching GRPO up to scale; these are verified special cases, not renamings. Empirical claims rest on held-out benchmarks (MATH500, AMC23, AIME2024/2025) evaluated with shared hyperparameters across methods. There are no self-citations; every theoretical reference (Sutton, Williams, Shao/GRPO, Kool/RLOO) is external. Two concerns are genuine but are not circularity under the rubric. First, the proof step 'Under the assumption, the variance of the gradient estimator gα,β is proportional to E_q E_o[(R−p)^2/f_N^2]' drops the (∇logπ)^2 factor; uncorrelatedness does not imply E[X^2Y^2]=E[X^2]E[Y^2], so the claimed optimum minimizes a proxy rather than the true variance — a validity gap in Theorem 1, not a by-construction reduction. Second, the assumption that p(q) follows a Beta distribution is a stated modeling choice, not independently validated — a correctness risk. The only circular element found is Section 5.4's 'validation' of parameter adaptation, which holds by construction from Eq. (13) and Algorithm 1; it is minor and non-load-bearing. Score 2.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The entire variance-minimization result rests on the assumed Beta distribution for p(q) and the uncorrelatedness assumption for the score function; both are unverified and the latter is likely false.

free parameters (1)
  • α and β (Beta normalization shape parameters) = α = 1 + a/3, β = 1 + b/3, with a,b estimated from the current batch via method of moments
    The normalization scale is set by moment estimates of the expected reward across questions. The theoretical optimum depends on these fitted values.
assumptions (3)
  • domain assumption The expected reward p(q) for each question follows a Beta distribution fD(p(q); a, b) across the question distribution.
    Section 3, Eq (10)-(13): BNPO's optimal parameters are derived under this distributional assumption without empirical validation.
  • ad hoc to paper The policy score function ∇θ log π(o|q) is uncorrelated with (R(q,o)-p(q))/fN(p(q);α,β).
    Theorem 1, Appendix A: This assumption lets the proof factor the variance into a term depending only on the reward, but it is not justified and is generally false.
  • standard math R(q,o) is Bernoulli with success probability p(q) = E_{o~π}[R(q,o)|q].
    Follows from the binary reward definition in Eq (9).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BNPO: Beta Normalization Policy Optimization." pith.science (2026). https://pith.science/paper/7ZCBIX56

@misc{pith2026250602864,
  author       = {Pith},
  title        = {Pith review of: BNPO: Beta Normalization Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZCBIX56}},
  note         = {Machine review of arXiv:2506.02864}
}
read the original abstract

Recent studies, including DeepSeek-R1 and Kimi-k1.5, have demonstrated that reinforcement learning with rule-based, binary-valued reward functions can significantly enhance the reasoning capabilities of large language models. These models primarily utilize REINFORCE-based policy optimization techniques, such as REINFORCE with baseline and group relative policy optimization (GRPO). However, a key limitation remains: current policy optimization methods either neglect reward normalization or employ static normalization strategies, which fail to adapt to the dynamic nature of policy updates during training. This may result in unstable gradient estimates and hinder training stability. To address this issue, we propose Beta Normalization Policy Optimization (BNPO), a novel policy optimization method that adaptively normalizes rewards using a Beta distribution with dynamically updated parameters. BNPO aligns the normalization with the changing policy distribution, enabling more precise and lower-variance gradient estimation, which in turn promotes stable training dynamics. We provide theoretical analysis demonstrating BNPO's variance-reducing properties and show that it generalizes both REINFORCE and GRPO under binary-valued reward settings. Furthermore, we introduce an advantage decomposition mechanism to extend BNPO's applicability to more complex reward systems. Experimental results confirm that BNPO achieves state-of-the-art performance among policy optimization methods on reasoning tasks. The code is available at https://github.com/changyi7231/BNPO.

Figures

Figures reproduced from arXiv: 2506.02864 by the authors.

Figure 1
Figure 1. Probability density function of Beta distribution. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The norm of gradient during training. 0 200 400 600 800 1000 Step 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Value E[p] Var[p] [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The values of (E[p(q)], Var[p(q)], α, β) during training. the specific form of the reward, making it challenging to develop a universal solution. Designing a general normalization method for continuous rewards remains a direction for future work. Extension of Theorem 1 Theorem 1 is established under the assumption of binary-valued rewards, which limits its general applicability. However, as discussed in the interpre… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A potential-based prediction reward collapses GRPO-trained LLM agents into a predictable 'dark room' state, and the collapse is caused by GRPO's std normalization rather than by the reward's magnitude.

  2. Aligning Language Models with Selective Prediction

    cs.LG 2026-07 accept novelty 7.0 of 10

    RLSR aligns LLMs via a lifted AURC reward and batch ranking inside GRPO, producing better risk-coverage curves than accuracy- or calibration-based RL on in- and out-of-domain tasks.

  3. DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    DenoiseRL optimizes recovery from noisy prefixes in weak-model reasoning failures to improve performance and self-correction on math and general reasoning benchmarks without external supervision.

  4. MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems

    cs.LG 2025-09 conditional novelty 5.0 of 10

    MO-GRPO rescales each reward function by its group variance before summing them in GRPO, equalizing their influence and reducing reward hacking in multi-objective LLM training.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages · cited by 4 Pith papers

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  2. [2]

    Aime problems and solutions, 2025 a

    Art of Problem Solving . Aime problems and solutions, 2025 a . URL https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions. Accessed: 2025-04-20

    work page 2025

  3. [3]

    Amc problems and solutions, 2025 b

    Art of Problem Solving . Amc problems and solutions, 2025 b . URL https://artofproblemsolving.com/wiki/index.php?title=AMC_Problems_and_Solutions. Accessed: 2025-04-20

    work page 2025

  4. [4]

    Reinforcement learning: An introduction

    Andrew G Barto. Reinforcement learning: An introduction. by richard’s sutton. SIAM Rev, 6 0 (2): 0 423, 2021

    work page 2021

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    arXiv 2021

  7. [7]

    Reinforce++: A simple and efficient approach for aligning large language models

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    arXiv 2025

  8. [8]

    Buy 4 REINFORCE samples, get a baseline for free!, 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019

    work page 2019

Show all 20 references
  1. [9]

    Remax: a simple, effective, and efficient reinforcement learning method for aligning large language models

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: a simple, effective, and efficient reinforcement learning method for aligning large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 29128-...

    2024

  2. [10]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  3. [11]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  4. [12]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  5. [13]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  6. [14]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  7. [15]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  8. [16]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  9. [17]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  10. [18]

    Variance reduction for policy gradient with action-dependent factorized baselines

    Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint arXiv:1803.07246, 2018

    2018 arXiv

  11. [19]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  12. [20]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.