Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Stable Reinforcement Learning for Efficient Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GRPO-λ claims that applying length penalties only to groups with high correctness ratios prevents the accuracy collapse that length-penalized reinforcement learning otherwise suffers.

desk verdict A sensible, cheap GRPO tweak that deserves referee time, but the abstract overstates the adaptive mechanism and the current evidence is one run with no seeds or ablations. read the letter →

arxiv 2505.18086 v1 pith:TOY7SBLO submitted 2025-05-23 cs.AI cs.LG

Muzhi Dai , Shixuan Liu , Qingyi Si This is my paper
classification cs.AIcs.LG
keywords reinforcementlearningGRPOchain-of-thoughtlengthpenaltytrainingstabilityreasoningefficiencyrewardshapingLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning that rewards shorter chain-of-thought responses tends to collapse: as responses get shorter, accuracy suddenly drops early in training. GRPO-$\lambda$ claims this collapse is avoidable by applying the length penalty selectively. Within each training batch, it measures the correctness ratio of each query's sampled answer group, applies the length penalty only to the top 20% of groups by correctness, and falls back to plain 0/1 success rewards for the rest. If the claim holds, reward design can push reasoning models toward shorter outputs without sacrificing accuracy. Across five benchmarks it reports a 1.48% average accuracy gain and a 47.3% reduction in chain-of-thought length.

What carries the argument

The load-bearing mechanism is the batch-wise top-$\lambda$ selection rule. For each query, the model samples $m$ completions, each scored 0/1 correct or wrong with a length in tokens; for each group the correctness ratio is computed, groups are ranked within the batch, and the top $\lambda$ (here 20%) are switched to efficiency mode while the rest stay in accuracy mode. Efficiency mode uses the reward $1 - \alpha \cdot \sigma((L_i^k - \mu_k^c)/s_k^c)$ for correct completions and 0 otherwise, where $\mu_k^c$ and $s_k^c$ are the mean and standard deviation of lengths among correct completions in the group; accuracy mode is the standard 0/1 outcome reward. Advantages are then normalized per group as in GRPO. The scheduling rule is what the paper claims prevents premature collapse.

What would settle it

Run GRPO-$\lambda$ on the same setup but rank groups by the opposite criterion, applying length penalties to the bottom 20% of correctness ratios; if accuracy still holds and training stays stable, group correctness is not the cause of the collapse it prevents. Alternatively, hold out hard-problem queries and check pass@1 after training on batches whose top-$\lambda$ groups are all easy questions.

Watch

Extended reading notes

Core claim

The central claim is that length-penalty rewards destabilize GRPO training not because compression is inherently harmful, but because penalties are applied when the model still needs accuracy reinforcement. GRPO-$\lambda$ ranks query-completion groups within a batch by correctness ratio; the top $\lambda$ fraction receives a length-penalty reward that scores correct completions by how far their length deviates from the group's correct-completion mean, and the remaining groups receive standard 0/1 rewards. This batch-wise switch keeps training stable past 100 steps where the uniform-penalty baseline collapses around step 40, and produces the best accuracy-efficiency trade-off in the paper's experiments: average accuracy up 1.48% and chain-of-thought length down 47.3% on GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024, with larger gains on harder math benchmarks.

Load-bearing premise

The method assumes that a group's correctness ratio measures whether the model already reasons well enough that compressing that group's outputs is safe; if high-accuracy groups contain easy questions whose shortened solutions still weaken hard-task reasoning, the stability benefit would not generalize.

Editorial extensions

If this is right

  • Uniform length penalties in GRPO are not safe: without adaptive scheduling, reported accuracy collapses early, with a 13.42% average accuracy drop at matched training steps.
  • With GRPO-$\lambda$, effective training steps extend at least 2.5 times before the collapse seen in the baseline.
  • On harder benchmarks such as AMC 2023 and AIME 2024, the accuracy-efficiency gain is larger than on easier tasks such as GSM8K and GPQA.
  • The method converges to a minimal safe length: in the AMC 2023 study, accuracy is preserved at around 5,000 tokens, and further compression degrades reasoning.
  • Overly aggressive length reduction during training prematurely cuts reasoning paths and impairs exploration, so the compression rate, not just the final length, must be controlled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make the selection threshold adaptive per batch or per difficulty tier instead of a fixed 20%, which would test whether the correctness-ratio rank, not the fraction, is the active ingredient.
  • The same switch could be applied to other reward-shaping terms, not just length penalties, whenever a shaping term and a competence signal interact.
  • One testable consequence is that groups with high correctness but already short responses sit near the safe compression frontier, so forcing further compression on them should reproduce the collapse the method avoids; checking this would isolate why the top-$\lambda$ rule works.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GRPO-lambda, a modification of GRPO for LLM reasoning RL. The method samples 16 completions per query, computes group-level correctness ratios, and within each training batch applies a length-penalty reward to the top-20% of groups ranked by correctness ratio while giving the remaining groups the standard 0/1 outcome reward. The authors report that on Qwen3-8B over GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024, GRPO-lambda improves average accuracy by 1.48% and reduces chain-of-thought length by 47.3% relative to GRPO with a uniform length penalty, while avoiding the training collapse they observe for that baseline.

Significance. If the central claim holds, the proposed reward-scheduling rule would be a simple and practically useful way to stabilize RL for reasoning efficiency, and the paper would provide a useful empirical datapoint connecting reward design, training stability, and overthinking. The use of five external benchmarks is a strength, as is the simplicity of the proposed modification, which should make it easy to reproduce. However, the paper currently provides only one base model, one run per condition, and no error bars or ablations, and the implemented selection rule does not match the correctness-adaptive mechanism described in the abstract. The headline differences are therefore not yet established beyond run-to-run variance, and the interpretation offered for the results is not supported by the algorithm as written.

major comments (4)
  1. [Abstract and Section 3 (Batch-Wise Top-lambda Selection)] The abstract and Section 1 state that a low correctness ratio triggers a switch to length-agnostic 0/1 rewards, implying an absolute correctness threshold. The implemented rule in Section 3 instead ranks groups within each batch and always penalizes the top 20% by relative correctness rank, regardless of absolute correctness. If an entire batch is weak, the selected groups can still have low correctness and receive the length penalty; if the batch is strong, 80% of groups are withheld from efficiency optimization. The claimed correctness-adaptive mechanism is therefore not what is implemented, and the stability benefit could instead come from the fact that only 20% of groups are ever penalized. The paper must either implement an absolute threshold, or reframe the contribution and all abstract/conclusion claims to describe top-lambda selection.
  2. [Section 3, Eq. (1)] Equation (1) uses std(L_k)_correct in the denominator, but this is zero whenever a group has exactly one correct completion, making the reward undefined. With 16 samples per query, such groups are common in early training. No epsilon, minimum-correct-count safeguard, or tie-breaking rule is specified, despite the fact that correctness ratios are quantized in increments of 1/16 and many groups will tie at the selection boundary. The paper should specify a well-defined reward for all possible group configurations and state how ties in the top-20% selection are broken.
  3. [Section 4.2 and Table 1] The empirical evidence is thin: all results use one base model (Qwen3-8B) and one training run per condition, with no standard errors or multiple seeds. The reported headline accuracy gain of 1.48% average is small relative to plausible run-to-run variance for RL training. The paper should provide multiple seeds with error bars, and ideally ablations over lambda, alpha, the training-data correctness filter (Section 4.1), and the group-size choice, to support the claim that the stability and accuracy-efficiency trade-off come from the proposed selection rule rather than from hyperparameter luck.
  4. [Section 4.2, Table 1 (+LP* row)] The comparison to +LP* is reported at the same step count as GRPO-lambda after +LP* has already collapsed, so it does not show the best achievable accuracy-length trade-off for the length-penalty baseline. A standard practice would be to compare with the best checkpoint of each method according to a held-out criterion, or to report the full training curves for all conditions. The paper should also state the number of training steps used for each row, and clarify how the 'at least 2.5x more viable iterations' claim is measured, since Figure 1 shows 100 steps for GRPO-lambda but the inference curves do not specify the exact collapse criterion.
minor comments (6)
  1. [Section 3 (Advantage Computation)] The sentence 'After obtaining the decaying rewards' is inaccurate: the accuracy-priority branch uses a 0/1 reward, not a decaying reward, and the length-penalty branch is not described as decaying either. Please reword to 'After obtaining the rewards'.
  2. [Section 3, Eq. (1)] The symbol sigma is used but never defined; if it denotes the logistic sigmoid, this should be stated explicitly.
  3. [Figure 4] The correctness labels in Figure 4 are confusing: the displayed GRPO-lambda final answer is also marked wrong in the annotations, which contradicts the text's claim that 'our method achieves correct answers' in this example. Please fix the labels or choose an example where the comparison is unambiguous.
  4. [Section 4.1] The training-data filter (sampling 8 completions per query and keeping queries answered correctly 2-6 times) is a nonstandard selection step that could materially affect difficulty and should be motivated or ablated, not just stated as a setting.
  5. [References] References [18] and [32] are the same work (Arora and Zanette) and should be merged or cross-referenced consistently.
  6. [Section 5] The claim of being 'the first systematic study' on how length-penalty reward design impacts RL training stability is overstated given the single-model, single-run experimental setup; please temper the claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: GRPO-λ's mechanism and headline numbers are not constructed from the target result, and the only self-overlap (citation [19]) is descriptive related work.

full rationale

The paper's central claim is that batch-wise top-λ selection of groups for length penalty avoids training collapse while preserving accuracy. The mechanism is defined by Eqs. (1)-(2) and the selection rule in Section 3, and the headline numbers (1.48% accuracy gain, 47.3% length reduction) are measured on external benchmarks (GSM8K, GPQA, MATH-500, AMC 2023, AIME 2024) against a Qwen3-8B baseline. The hyperparameters λ=20% and α=0.2 are stated as fixed constants, not fit to the evaluation results, so the accuracy-efficiency trade-off is not a fitted input renamed as a prediction. The only self-overlap is citation [19] (S-GRPO, sharing first author Muzhi Dai), which appears in Related Work as one of several length-penalty mechanisms; it is not invoked as a justification, uniqueness theorem, or source of the GRPO-λ ansatz, and therefore is not load-bearing. Concerns that the top-λ selection is relative rather than absolute, and that Eq. (1) is undefined when a selected group has exactly one correct completion, are implementation and correctness risks rather than circularity: they do not make any derived quantity equal to its input by construction. The paper also acknowledges in Section 5 that systematic experiments on the λ proportion are left to future work; this is an explicit limitation, not a circular step. The paper does not rename a known result under new coordinates, and no self-citation chain forces its conclusions.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of explicit hyperparameters (lambda and alpha) and on the domain assumption that group correctness ratio is a reliable proxy for when length penalties are safe. No new entities are introduced. The method is an empirical heuristic layered on standard GRPO.

free parameters (3)
  • lambda (top fraction of groups receiving length penalty) = 0.2
    Chosen without ablation; the paper's conclusion notes that 'the proportion of length-penalty groups in each batch (lambda value) significantly impacts performance' but provides no sensitivity study.
  • alpha (length penalty coefficient) = 0.2
    Chosen without ablation; appears in Eq. (1) as the scale of the sigmoid penalty.
  • training data correctness filter = 2-6 correct answers out of 8
    Section 4.1 filters DeepMath-103K queries to those answered correctly 2 to 6 times by Qwen3-8B; this selection criterion may affect the difficulty mix and the measured trade-off.
assumptions (3)
  • domain assumption A high group-level correctness ratio identifies groups where the model's reasoning is mature enough to apply length penalties safely.
    Section 3 (Batch-Wise Top-lambda Selection) states that top-lambda groups 'demonstrate sufficient reasoning capability to focus on length reduction'. This is the core intuition; no evidence beyond the reported runs is provided.
  • domain assumption Uniform length-penalty rewards cause premature, reproducible training collapse for GRPO.
    Figure 1 (left) and the +LP* row in Table 1 show one collapsed run; the paper generalizes this to 'length-aware reward or penalty functions lead to premature RL training collapse' in the introduction.
  • standard math GRPO-style group advantage normalization remains a valid policy-gradient update when rewards are a mixture of 0/1 and length-penalty rewards.
    Section 3 (Advantage Computation) reuses the standard GRPO advantage formula without modification; this is a background assumption of the RL framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stable Reinforcement Learning for Efficient Reasoning." pith.science (2026). https://pith.science/paper/TOY7SBLO

@misc{pith2026250518086,
  author       = {Pith},
  title        = {Pith review of: Stable Reinforcement Learning for Efficient Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOY7SBLO}},
  note         = {Machine review of arXiv:2505.18086}
}
abstract

The success of Deepseek-R1 has drawn the LLM community's attention to reinforcement learning (RL) methods like GRPO. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies have designed reward functions to reinforce models' behaviors in producing shorter yet correct completions. Nevertheless, we observe that these length-penalty reward functions exacerbate RL training instability: as the completion length decreases, model accuracy abruptly collapses, often occurring early in training. To address this issue, we propose a simple yet effective solution GRPO-$\lambda$, an efficient and stabilized variant of GRPO, which dynamically adjusts the reward strategy by monitoring the correctness ratio among completions within each query-sampled group. A low correctness ratio indicates the need to avoid length penalty that compromises CoT quality, triggering a switch to length-agnostic 0/1 rewards that prioritize reasoning capability. A high ratio maintains length penalties to boost efficiency. Experimental results show that our approach avoids training instability caused by length penalty while maintaining the optimal accuracy-efficiency trade-off. On the GSM8K, GPQA, MATH-500, AMC 2023, and AIME 2024 benchmarks, it improves average accuracy by 1.48% while reducing CoT sequence length by 47.3%.

Figures

Figures reproduced from arXiv: 2505.18086 by the authors.

Figure 1
Figure 1. Training process of GRPO+length penalty and our GRPO-λ. Abstract The success of Deepseek-R1 has drawn the LLM community’s attention to re￾inforcement learning (RL) methods like GRPO. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies hav… view at source ↗
Figure 2
Figure 2. Framework of GRPO-λ. from the group distribution. This maps rewards to the interval (0,1), where shorter and correct responses receive values closer to 1, while incorrect responses are assigned zero reward. S-GRPO [19] adopts a dual-rollout strategy, performing early-exit interventions at different positions within the first rollout response to construct a serial group, and allocating exponentially decaying rewards … view at source ↗
Figure 3
Figure 3. Relationship between performance and response [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of a generated content sample on GSM8K. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Length Penalties Make Chain-of-Thought Less Monitorable

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Length-penalized RL shortens chain-of-thought while preserving accuracy and hint influence, but preferentially removes the cues that let a monitor detect that influence.

  2. Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security

    cs.CR 2025-07 conditional novelty 6.0 of 10

    An iterative attacker-defender reinforcement learning method that makes a multimodal LLM refuse more jailbreak prompts without over-refusing ordinary queries.

Reference graph

Works this paper leans on

35 extracted references · 4 canonical work pages · cited by 2 Pith papers

  1. [1]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URLhttps://arxiv. org/abs/2408.03314

    arXiv 2024

  2. [2]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    arXiv 2020

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  4. [4]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  5. [5]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    arXiv 2024

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  7. [7]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022

    2022

  8. [8]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    arXiv 2017

Show all 35 references
  1. [9]

    Group robust preference optimization in reward-free RLHF

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free RLHF. In The Thirty-eighth Annual Conference on Neural Information Processing Systems ,

  2. [10]

    Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning m...

    2025 arXiv

  3. [11]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  4. [12]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong 7 Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025. URL https://arxiv.org/abs...

    2025 arXiv

  5. [14]

    The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025

    2025 arXiv

  6. [15]

    When more is less: Understanding chain-of-thought length in llms

    Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 2025

    2025 arXiv

  7. [16]

    Dynamic early exit in reasoning models, 2025

    Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models, 2025. URLhttps://arxiv.org/ abs/2504.15895

    2025

  8. [17]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025 arXiv

  9. [19]

    S-grpo: Early exit via reinforcement learning in reasoning models, 2025

    Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models, 2025. URL https://arxiv.org/abs/2505.07686

    2025 arXiv

  10. [20]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168

    2021 arXiv

  11. [21]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  12. [22]

    Aime problems and solutions

    MAA Committees . Aime problems and solutions. https://artofproblemsolving. com/wiki/index.php/AIME_Problems_and_Solutions

  13. [23]

    Amc 2023, 2024

    AI-MO. Amc 2023, 2024. URL https://huggingface.co/datasets/AI-MO/ aimo-validation-amc

    2023

  14. [24]

    Measuring mathematical problem solving with the math dataset,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset,

  15. [25]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/research/ learning-to-reason-with-llms , 2025. Accessed: 15 March 2025. 8

    2025

  16. [26]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  17. [27]

    On designing effective rl reward at training time for llm reasoning, 2024

    Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning, 2024. URL https://arxiv.org/abs/2410.15115

    2024 arXiv

  18. [28]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2025 arXiv

  19. [29]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https://arxiv.org/abs/2503.18892

    2025 arXiv

  20. [30]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025. URL https://arxiv.org...

    2025 arXiv

  21. [31]

    Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models, 2025

    Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models, 2025. URL https://arxiv.org/abs/2503.17287

    2025

  22. [32]

    Training language models to reason efficiently

    Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04463, 2025

    2025

  23. [33]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical datas...

    2025 arXiv

  24. [34]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  25. [35]

    Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning, 2025

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2504.01296. 9

    2025 arXiv

  26. [2021]

    URL https://arxiv.org/abs/2103.03874

    arXiv

  27. [2024]

    URL https://openreview.net/forum?id=PRAsjrmXXK

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.