Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read During GRPO training, low-probability tokens generate disproportionately large gradients and dominate model updates, suppressing high-probability tokens; two methods—Advantage Reweighting and Lopti—correct this imbalance and improve…

desk verdict A practical, cheap fix for a real GRPO nuisance, but the 'low-probability token dominance' story is a plausible inference, not a proven mechanism. read the letter →

arxiv 2505.12929 v1 pith:6R467E6T submitted 2025-05-19 cs.CL cs.AIcs.LG

Zhihe Yang , Xufang Luo , Zilong Wang , Dongqi Han , Zhiyuan He , Dongsheng Li , Yunjian Xu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords reinforcementlearningGRPOtokenprobabilitygradientnormslow-probabilitytokensadvantagereweightingLLMreasoningpolicy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a hidden bias in RL for LLMs comes from token probability itself: the less probable a token is, the larger the gradient it produces, so low-probability tokens dominate each update and drown out high-probability tokens whose gradients are needed for good reasoning. The authors derive a per-token bound showing the gradient norm with respect to any layer's activations scales like $(1-\pi)$, the complement of the token's probability, times a roughly constant weight. They then propose two fixes: Advantage Reweighting, which scales down advantages of low-probability tokens, and Low-Probability Token Isolation (Lopti), which updates low- and high-probability tokens in separate passes, low first. In experiments on K&K logic puzzles, combining the two raises GRPO-trained Qwen2.5-3B average accuracy from 0.39 to 0.57 (a 46.2% relative gain), with consistent gains on math datasets and under REINFORCE++. The paper's central message is that balancing token-level gradients, not just response-level rewards, is a cheap and effective lever in RL post-training.

What carries the argument

The load-bearing identity is the logits-gradient factorization $\delta_L(o_{i,t}) = w_{i,t}(I(o_{i,t}) - p(o_{i,t}))$, where $I$ is the one-hot vector at the sampled token and $p$ is the softmax distribution over the vocabulary; its norm lies between $\sqrt{N/(N-1)}(1-\pi)$ and $\sqrt{2}(1-\pi)$, which is the source of the $(1-\pi)$ scaling. Backpropagation through layers with bounded Jacobian singular values preserves this scaling at every layer. The GRPO objective's token weight $w_{i,t} = (\pi_\theta/\pi_{\text{old}})\,\hat{A}_i\,I_{\text{trust}} + \beta(\pi_{\text{ref}}/\pi_\theta - 1)$ is what the two proposed methods modify: Advantage Reweighting multiplies $\hat{A}_i$ by $\alpha\pi_\theta + (1-\alpha)$, and Lopti masks the advantage so low- and high-probability tokens are updated in separate passes.

What would settle it

On one of the paper's training setups, compute the full-batch GRPO gradient separately for tokens in the top and bottom probability quartiles, with the KL coefficient set to zero so the weight $w_{i,t}$ no longer depends on $\pi_\theta$; if the summed high-probability gradient norm is comparable to or larger than the low-probability one, or if per-token norm does not scale with $(1-\pi)$, the dominance claim fails. A cheaper check is to compare how often positive tokens in each quartile increase in probability under Advantage Reweighting versus naive GRPO.

Watch

Extended reading notes

Core claim

The paper's central claim is that in GRPO training, the per-token gradient norm is essentially proportional to $1-\pi_\theta(o_{i,t})$, so tokens the model assigns low probability generate disproportionately large gradients. Proposition 4.2 states that under bounded-Jacobian assumptions, for any layer $\ell$, $\prod_{j=\ell+1}^{L} c_j\,|w_{i,t}|\,\sqrt{N/(N-1)}\,(1-\pi_\theta(o_{i,t})) \le \|\delta_\ell(o_{i,t})\| \le \prod_{j=\ell+1}^{L} d_j\,|w_{i,t}|\,\sqrt{2}\,(1-\pi_\theta(o_{i,t}))$, where $w_{i,t}$ is the GRPO token weight from Eq. (2). Since updates average over hundreds of thousands of tokens, the large per-token norms of low-probability tokens dominate the summed gradient, and high-probability tokens—whose updates are more often in the correct direction—are suppressed. The paper supports this with probability-quartile experiments showing low-probability groups produce larger gradient norms and that updating only the lowest quartile reproduces the full-update probability shift. To counter the imbalance it introduces Advantage Reweighting (Eq. 4, replacing $\hat{A}$ with $[\alpha\pi + (1-\alpha)]\hat{A}$) and Lopti (splitting tokens at threshold $\eta$ and updating low-probability tokens first), reporting up to 46.2% relative improvement on K&K Logic Puzzles and gains on math benchmarks and REINFORCE++.

Load-bearing premise

The claim that low-probability tokens dominate rests on treating the token weight $w_{i,t}$ as roughly constant across probability groups; in fact $w$ includes $\beta(\pi_{\text{ref}}/\pi_\theta - 1)$, which grows as $\pi_\theta$ shrinks, and the paper does not show $|w|$ is similar across groups, nor that per-token gradient norms survive summation without cancellation across the hundreds of thousands of tokens in each update.

Editorial extensions

If this is right

  • Downweighting low-probability tokens by a factor linear in their probability (Advantage Reweighting) improves GRPO-trained accuracy on hard logic puzzles, with the largest gains on the most difficult 6- and 7-person puzzles where positive samples are scarce.
  • Updating low-probability tokens before high-probability tokens (Lopti) is the effective order; reversing the order makes training collapse after the fourth epoch, so the ordering itself is doing real work.
  • High-probability tokens cannot simply be discarded: masking them degrades baseline GRPO, so the correct intervention is to attenuate low-probability interference rather than remove high-probability gradients.
  • The imbalance is not specific to GRPO: the same reweighting improves REINFORCE++ on K&K puzzles (for example, average accuracy on Qwen2.5-3B rises from 0.23 to 0.34), suggesting policy-gradient LLM training broadly is affected.
  • On math benchmarks the two fixes each help but do not stack; the paper recommends using them individually there, implying the benefit is real but not additive in all settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the $(1-\pi)$ scaling is the mechanism, the imbalance should worsen as training sharpens the policy: the highest-probability tokens become even quieter while low-probability tokens retain large gradients, so the reweighting gain should grow with training length and with stronger KL penalties—this is testable from the paper's own checkpoints.
  • The KL term in $w_{i,t}$ also grows as $\pi_\theta$ shrinks ($\beta(\pi_{\text{ref}}/\pi_\theta - 1)$), so a sharper test of the paper's story would set $\beta=0$ or freeze $\pi_{\text{ref}}$ to isolate whether the dominance comes from the softmax geometry or from the weight's own probability dependence.
  • Because the bound is derived for any layer activation, the same imbalance should appear in other policy-gradient variants and even in preference-optimization losses with a softmax head; applying Advantage Reweighting to those losses is a natural extension the paper does not pursue.
  • The probability-quartile gradient ratio suggests a practical monitoring metric: track each quartile's contribution to total gradient norm during RL and use it to schedule $\alpha$ or $\eta$, turning a static hyperparameter into an adaptive one.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper claims that during GRPO training of LLMs, low-probability tokens produce disproportionately large gradient magnitudes, dominate the aggregate model update, and consequently suppress learning of high-probability tokens. It derives Proposition 4.2, bounding the per-token activation gradient norm by |w_{i,t}|(1 - pi_theta(o_{i,t})), and proposes two mitigation methods: Advantage Reweighting, which scales advantages by alpha*pi + (1 - alpha), and Low-Probability Token Isolation (Lopti), which updates low- and high-probability tokens in separate steps. Experimental results on K&K Logic Puzzles and math-related datasets show consistent improvements over naive GRPO, with up to a 46.2% relative gain on K&K Logic Puzzles, plus supporting ablations and a REINFORCE++ extension.

Significance. If the central claim is established, the paper offers a novel and practically useful diagnosis of an update bias in RL training for LLMs, well beyond the previously studied response-length bias. The strengths are the concise gradient derivation, the simplicity of the proposed methods, the public implementation, the coverage of multiple base models and datasets, and the informative reversed-order ablation in Figure 6(b). The main risk is that the paper's headline conclusion, that low-probability tokens dominate model updates, is currently an inference from per-token gradient bounds rather than a demonstrated property of the aggregate update; the empirical methods improve performance, but the measurements needed to tie those improvements to the proposed mechanism are missing.

major comments (3)
  1. [§4.1, Eq. (3) and Eq. (8)] The step from Proposition 4.2 to the 'dominance' claim treats |w_{i,t}| as approximately constant in Eq. (3), but this is not established and is in fact questionable. From Eq. (8), w_{i,t} = (pi_theta/pi_old) * A_hat * I + beta*(pi_ref/pi_theta - 1); the KL term grows as pi_theta becomes small, and for tokens whose negative advantage is clipped (I = 0), w is exactly that KL term and not A_hat. The sentence after Eq. (3) that w is 'approximately equal to A_hat in most cases' therefore needs quantitative support. A direct test would be to report mean/median |w_{i,t}| and the per-token GRPO gradient norm per probability quartile, with and without the KL contribution; if the norm ordering disappears once |w| is controlled, Proposition 4.2 does not explain the observed dominance. Since both proposed methods modify only the advantage part of w while leaving the KL term in Eq. (1) active, improved downstream performance cannot by itself confirm the proposed mechanism.
  2. [§4.1, Fig. 1(d)-(f)] The inference from per-token gradient norms to dominance of the aggregate model update is missing a load-bearing step. Proposition 4.2 bounds the norm of a single token's activation gradient, but an RL update is the sum of hundreds of thousands of token gradients; a group can have large per-token norms and yet a small or cancelling sum, while a group with small per-token norms can align constructively. Figure 1(d) reports only per-token statistics, and Figures 1(e) and 1(f) show only that updating low-probability tokens changes the probabilities of high-probability tokens, which any parameter update to a shared network would do. The paper should measure, for each probability quartile, the norm (and ideally the cosine similarity) of the summed token gradients used in an actual update, and show that the low-probability group's sum is large relative to the high-probability group's sum. This measurement is feasible and would directly support the term 'dominate model updates'.
  3. [§4.1, Eq. (3) vs. Fig. 1(d)] There is also an unaddressed gap between activation gradients and parameter gradients. Proposition 4.2 bounds delta_ell(o_{i,t}), the gradient with respect to an intermediate activation, while Figure 1(d) and the proposed methods concern parameter updates. The parameter-gradient norm contains an additional factor from the forward activations, so the bound in Eq. (3) does not by itself imply that parameter gradients follow the same (1 - pi) ordering. The authors should either extend the derivation to parameter gradients under a bounded-activation assumption or report parameter-gradient norms per probability quartile, which is the quantity that actually determines the update.
minor comments (5)
  1. [§3 and References] In Section 3, GRPO is attributed to reference [10] (Open-Reasoner-Zero); the correct source for GRPO is DeepSeekMath, reference [8]. The same citation issue appears near Eq. (1).
  2. [Table 4] The table header contains a typo: 'Patially Correct' should be 'Partially Correct'.
  3. [§3] The sentence 'the estimated advantage is set to be consistence within each responses' should be reworded, e.g., 'the estimated advantage is set to be consistent within each response'.
  4. [§5.1–5.3, Appendix B] The hyperparameters alpha and eta are tuned on the same K&K and math datasets for which the headline results are reported, with recommended ranges derived from those datasets. Reporting results under a fixed default setting on a genuinely held-out task would make the empirical claims more robust.
  5. [Figure 4 and Appendix C] The model name 'Qwen2.5-7B-Instruct-1M' appears in the main text and tables without explanation of the '1M' variant; please clarify what this checkpoint is and why it differs from the standard Qwen2.5-7B-Instruct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the gradient-norm bound is derived from the GRPO objective and the proposed methods are evaluated on held-out benchmarks.

full rationale

The paper's central theoretical claim, Proposition 4.2, is derived directly from the GRPO loss gradient and the softmax derivative; no target quantity or fitted parameter is inserted into the derivation, and the bound is a mathematical consequence of the stated objective rather than an assumption equivalent to the conclusion. The empirical observations in Figure 1 are measurements of real training statistics, and the proposed methods, Advantage Reweighting and Lopti, are tested on held-out test sets (K&K Logic Puzzle, math benchmarks), so their reported improvements are not constructed from the training labels. The hyperparameters alpha and eta are calibrated on the training data, which is ordinary tuning rather than a renamed fit; this does not make the prediction circular. The ablation masking high-probability tokens is a manipulation check whose outcome is expected, but that expectation is based on the same mechanism, not on circularity. No load-bearing self-citation appears: the cited baselines and infrastructure (Logic-RL, DAPO, ORZ, verl, REINFORCE++) are external works with no author overlap. The weakest point in the argument is the inference from per-token gradient norms to aggregate dominance of model updates, and the treatment of w_{i,t} as approximately constant despite its dependence on pi_theta; however, this is an evidentiary and correctness concern, not a circular reduction, because no equation of the paper forces the dominance conclusion by definition. Overall, the derivation chain is self-contained and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two tuned hyperparameters (α and η) and three assumptions: standard Jacobian regularity, approximate constancy of token weights, and the proxy status of gradient norms. The first is standard; the second is ad hoc and load-bearing; the third is an implicit domain assumption.

free parameters (2)
  • alpha (Advantage Reweighting weight) = 0.3 (K&K), 0.1 (Math, REINFORCE++)
    Manually defined hyperparameter in Eq. 4; tuned per task and used in reported gains.
  • eta (Lopti threshold) = 0.5 (all experiments); ablation recommends [0.3, 0.5]
    Threshold separating low- and high-probability tokens in Algorithm 1; chosen by hand.
assumptions (3)
  • domain assumption Assumption 4.1: each layer Jacobian has bounded singular values (0 < c_l <= sigma <= d_l).
    Standard smoothness/non-degeneracy assumption for neural networks; used in proof of Proposition 4.2.
  • ad hoc to paper w_{i,t} is approximately equal to the advantage A_hat_{i,t} for most tokens.
    Stated after Eq. 3; not proven. The KL term β(pi_ref/pi_theta - 1) in w is large for low-probability tokens, so w varies with π.
  • domain assumption Gradient norm is a valid proxy for a token's influence on the shared parameter update.
    Assumed in the dominance argument; ignores possible cancellation or alignment of gradients across tokens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs." pith.science (2026). https://pith.science/paper/6R467E6T

@misc{pith2026250512929,
  author       = {Pith},
  title        = {Pith review of: Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6R467E6T}},
  note         = {Machine review of arXiv:2505.12929}
}
read the original abstract

Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLMs), with recent innovations such as Group Relative Policy Optimization (GRPO) demonstrating exceptional effectiveness. In this study, we identify a critical yet underexplored issue in RL training: low-probability tokens disproportionately influence model updates due to their large gradient magnitudes. This dominance hinders the effective learning of high-probability tokens, whose gradients are essential for LLMs' performance but are substantially suppressed. To mitigate this interference, we propose two novel methods: Advantage Reweighting and Low-Probability Token Isolation (Lopti), both of which effectively attenuate gradients from low-probability tokens while emphasizing parameter updates driven by high-probability tokens. Our approaches promote balanced updates across tokens with varying probabilities, thereby enhancing the efficiency of RL training. Experimental results demonstrate that they substantially improve the performance of GRPO-trained LLMs, achieving up to a 46.2% improvement in K&K Logic Puzzle reasoning tasks. Our implementation is available at https://github.com/zhyang2226/AR-Lopti.

Figures

Figures reproduced from arXiv: 2505.12929 by the authors.

Figure 1
Figure 1. Experimental analysis on the K&K Logic Puzzle dataset during GRPO training of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Diagram of Proposition 4.2. Refer to Appendix A.2 for the detailed proof. Proposi￾tion 4.2 demonstrate that, for a single token, the gradient norm with respect to activation aℓ at any layer is bounded. Specifically, it is confined within the truncated conical region illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The proportion of posi￾tive tokens updated in the correct di￾rection for different updating meth￾ods, under the same experimental settings as in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Experimental results on the K&K Logic Puzzles benchmark. For [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) The relationship between the frequency of six categories of inference-related words [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies on the K&K Logic Puzzles dataset. (a) Effect of restricting updates to [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Experimental records of Qwen2.5-3B-Instruct trained with GRPO on the K&K Logic Puzzle dataset. The training curve is smoothed through exponential moving average with coefficient of 0.95. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Experimental records of Qwen2.5-7B-Instruct-1M trained with GRPO on the K&K Logic Puzzle dataset. For the six categories of inference-related words used in the linguistic analysis, the detailed word lists are provided in [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Experimental records of Qwen2.5-7B trained with GRPO on DSR-uniform dataset. The training curve is smoothed through exponential moving average with coefficient of 0.95, and the testing curve is smoothed with a window size of 3 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Experimental records of Qwen2.5-7B trained with GRPO on ORZ dataset. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Experimental records of Qwen2.5-3B-Instruct trained with REINFORCE++ on the K&K Logic Puzzle dataset. The training curve is smoothed through exponential moving average with coefficient of 0.95. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Experimental records of Qwen2.5-7B-Instruct-1M trained with REINFORCE++ on the K&K Logic Puzzle dataset. E Limitations One limitation of our study lies in the additional computational overhead introduced by Lopti. As detailed in Appendix C.2, the updating process requ…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

    cs.CL 2026-07 conditional novelty 6.0 of 10

    TACO soft-suppresses positive GRPO credit on high tail-risk tokens (surprisal above local entropy) and consistently beats GRPO-style baselines on three LLMs and eight reasoning benchmarks while stabilizing long training.

  2. STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    STAPO stabilizes RL for LLMs by suppressing gradient updates from rare spurious tokens, yielding 11.49% average gains on math benchmarks over GRPO and similar baselines.

Reference graph

Works this paper leans on

53 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    OpenAI o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. OpenAI o1 system card. arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  2. [2]

    Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [3]

    Kimi k1.5: Scaling reinforcement learning with LLMs

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025

    arXiv 2025

  4. [4]

    Monte carlo tree search boosts reasoning via iterative preference learning

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, 2024

    work page 2024

  5. [5]

    Alphamath almost zero: Process su- pervision without process

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process su- pervision without process. In Advances in Neural Information Processing Systems, volume 38, 2024

    work page 2024

  6. [6]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024

    2024

  7. [7]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024

    work page 2024

  8. [8]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    DAPO: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  2. [10]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  3. [11]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343, 2025

    2025 arXiv

  4. [12]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 31, 2017. 10

    2017

  5. [13]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  6. [14]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 34, 2020

    2020

  7. [15]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volu...

    2022

  8. [16]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  9. [17]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 34, 2020

    2020

  10. [18]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  11. [19]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  12. [20]

    LLaMA 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  13. [21]

    The LLaMA 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  15. [23]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  16. [24]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  17. [25]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  18. [26]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  19. [27]

    The Claude 3 model family: Opus, sonnet, haiku

    Anthropic. The Claude 3 model family: Opus, sonnet, haiku. Technical Report, 2024

    2024

  20. [28]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 37, 2023

    2023

  21. [29]

    From r to Q*: Your language model is secretly a Q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q*: Your language model is secretly a Q-function. In First Conference on Language Modeling, 2024. 11

    2024

  22. [30]

    Is DPO superior to PPO for LLM alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. In International Conference on Machine Learning, volume 41, 2024

    2024

  23. [31]

    DPO meets PPO: Reinforced token optimization for RLHF

    Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. DPO meets PPO: Reinforced token optimization for RLHF. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  24. [32]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  25. [33]

    ORPO: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  26. [34]

    Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. In International Conference on Machine Learning, volume...

    2024

  27. [35]

    SimPO: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems, volume 38, 2024

    2024

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 36, 2022

    2022

  29. [37]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog, 2025

    2025

  30. [38]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025

    2025 arXiv

  31. [39]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  32. [40]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  33. [41]

    V APO: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. V APO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025

    2025 arXiv

  34. [42]

    Efficient reinforcement finetuning via adaptive curriculum learning

    Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025

    2025 arXiv

  35. [43]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  36. [44]

    Hybridflow: A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. Proceedings of the Twentieth European Conference on Computer Systems, 2025

    2025

  37. [45]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024. 12

    2024 arXiv

  38. [46]

    Reinforce++: A simple and efficient approach for aligning large language models

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  39. [47]

    What is the name of this book? Touchstone Books Guildford, UK, 1986

    Raymond Smullyan. What is the name of this book? Touchstone Books Guildford, UK, 1986

    1986

  40. [48]

    Meta-logical problems: Knights, knaves, and rips

    Philip N Johnson-Laird and Ruth MJ Byrne. Meta-logical problems: Knights, knaves, and rips. Cognition, 36(1):69–84, 1990

    1990

  41. [49]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. Proceedings of the 62nd Annual ...

    2024

  42. [50]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems,...

    2022

  43. [51]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  44. [52]

    Approximating KL divergence

    John Schulman. Approximating KL divergence. Technical Blog, 2020

    2020

  45. [53]

    Lily is a knave or Lily is a knight

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229–256, 1992. 13 A Theoretical Interpretations A.1 Gradient Derivation for the GRPO Objective For clarity, we re-state the objective function of...

    1992

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.