G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance

Wenbo Deng; Xiaoying Tang; Yongxin Guo; Zhenglin Cheng

REVIEW 4 major objections 5 minor 1 cited by

G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read G2RPO-A claims that adaptively shortening and lengthening injected reasoning guidance as rewards rise and fall lets small language models beat vanilla GRPO on math and code benchmarks.

desk verdict A genuinely new adaptive guidance controller with a solid ablation study, but test-set-tuned α and overclaimed consistency make the central comparison weaker than the paper admits. read the letter →

arxiv 2508.13023 v1 pith:VJBC57L2 submitted 2025-08-18 cs.AI

Yongxin Guo , Wenbo Deng , Zhenglin Cheng , Xiaoying Tang This is my paper · ORCID

classification cs.AI

keywords adaptiveguidanceGRPOreinforcementlearningwithverifiablerewardssmalllanguagemodelscurriculummathematicalreasoningcodegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that small language models can get substantially more out of reinforcement learning with verifiable rewards if their roll-outs are seeded with ground-truth reasoning traces whose length is adjusted automatically during training. It first shows that naively adding fixed-length guidance does not help: the reward curve quickly matches vanilla GRPO and the advantage signal becomes too flat. It then proposes G2RPO-A, which shrinks guidance length when recent rewards rise and grows it when they fall, and pairs this controller with an easy-to-hard curriculum. Across Qwen3 0.6B, 1.7B, and 8B, DeepSeek-Math-7B, and DeepSeek-Coder-6.7B, the paper reports consistent accuracy gains over vanilla GRPO on MATH500, Minerva, GPQA, AIME, HumanEval, and LiveCodeBench. If correct, the result is a cheap, training-time lever for improving small-model reasoning without a larger base model or a critic network.

What carries the argument

The central mechanism is the adaptive guidance-length controller in Eq. (5), a one-line ratio update that compares the current average reward $r_k$ with the average over the previous $\min(T,k)$ steps and scales the guidance length $\ell_k$ accordingly. Guidance itself is a ground-truth reasoning trajectory prepended to the prompt so the small model continues from a correct line of thought; the guidance ratio $\alpha$ decides what fraction of each GRPO group of size $G$ receives guidance while the rest generate freely. The controller is protected from batch-difficulty noise by a curriculum that orders samples into ascending difficulty tiers (for math: cn_contest, aops_forum, amc_aime, olympiads, olympiads_ref). Together these pieces convert reward movement into a difficulty dial for the next training batch.

What would settle it

Train G2RPO-A on a fixed dataset with batches shuffled randomly instead of curriculum-ordered, and compare its guidance-length trajectory and final accuracy against vanilla GRPO: if batch-difficulty fluctuations dominate reward changes, the length will oscillate and the reported accuracy gains should shrink or reverse. A more direct check is to correlate the sign of $\ell_{k+1}-\ell_k$ with held-out accuracy measured periodically during training; if the correlation is near zero or negative, the controller is not tracking competence.

Watch

Extended reading notes

Core claim

On the paper's own terms, G2RPO-A 'markedly surpasses vanilla GRPO on nearly every benchmark' and 'consistently boosts accuracy.' With a per-model guidance ratio $\alpha$ and an initial guidance budget of 3,072 tokens, the controller updates guidance length as $\ell_{k+1} = \ell_k \cdot \frac{\min(T,k)\, r_k}{\sum_{\tau=1}^{\min(T,k)} r_{k-\tau}}$ with $T=2$, so rising average reward shortens guidance and falling reward lengthens it. Combined with a curriculum ordering of training samples into ascending difficulty tiers, this produces reported improvements such as Qwen3-1.7B MATH500 rising from 63.74 to 67.21, Qwen3-8B MATH500 rising from 79.49 to 82.08, Qwen3-1.7B HumanEval rising from 67.65 to 75.93, and similar gains on Minerva, GPQA, AIME24/25, and LiveCodeBench. The paper also claims that the optimal guidance ratio depends on task and model size, with code tasks and smaller models needing stronger guidance, and that keeping hard samples with adaptive guidance beats filtering them out.

Load-bearing premise

The load-bearing premise is that a rise or fall in the current average reward is a faithful signal of the model's learning state, so shrinking guidance when rewards rise and growing it when they fall always tracks true competence; the paper itself concedes that when consecutive batches differ markedly in difficulty, reward changes no longer reflect learning progress, and it patches this with a hand-chosen curriculum order.

Editorial extensions

If this is right

Small models can profit from RLVR on hard, sparse-reward tasks by keeping hard examples and attaching adaptive guidance, rather than filtering them out as too easy or too hard.
Because the controller needs only the last few average-reward values, it adds almost no overhead to a GRPO-style trainer and can replace manual guidance-length schedules.
The optimal guidance ratio still varies by model and task, so a practitioner would tune $\alpha$ but can leave guidance length self-adjusting.
Curriculum ordering helps both vanilla GRPO and G2RPO-A, suggesting that presentation order matters independently of the adaptive controller.
If the reported gains hold, the approach is a practical alternative to distillation or much larger base models for improving small-model math and code reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The controller uses a ratio of rewards rather than absolute reward values, so it may transfer across datasets with different reward scales; a direct test would be to normalize rewards and check whether the guidance-length trajectory stays the same.
A natural extension the paper does not pursue is applying the same reward-ratio controller to other RLVR variants, such as step-wise advantage methods, where guidance length could adapt to the model's competence at the token level.
The hand-chosen curriculum tiers are a patch for the controller's sensitivity to batch difficulty; an online difficulty estimator could replace them and make the method fully autonomous.
Under noisy or group-composition-driven reward fluctuations, the length update will oscillate; clamping, momentum, or a dead-zone on $\ell_k$ would be a robustness fix the paper does not explore.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read on G2RPO-A. The adaptive guidance-length controller in Eq. (5) is a real idea — scaling guidance by the ratio of current reward to a short history is simple and not in the cited guided-GRPO baselines. The systematic ablation of guidance ratio versus length is also useful, and the finding that naive fixed guidance fails while partial guidance helps is a concrete, reproducible insight. Credit where due: the experiments cover multiple model families and two task domains, and the code and models are released.

The soft spots are real but not fatal. The biggest is the α selection: Tables 8 and 9 pick the per-model, per-task guidance ratio by highest accuracy on the same benchmarks (MATH500, HumanEval, LiveCodeBench) that later appear in the headline Tables 3–5. That means the reported gains over vanilla GRPO are conditional on having tuned α to the test set. The paper never shows an untuned, α-default version still beats GRPO, so the "adaptive guidance helps" conclusion is confounded with "the right fixed ratio helps." This is the load-bearing issue, and the stress-test note is correct.

Second, the claim that G2RPO-A "consistently boosts accuracy" is contradicted by the paper's own tables: Qwen3-0.6B on MATH500 drops from 54.26 (GRPO) to 51.77 (G2RPO-A), Qwen3-8B on HumanEval drops from 81.48 to 80.33, and Qwen3-1.7B on LiveCodeBench drops from 53.14 to 51.96. The ablation tables also show cases where fixed guidance beats G2RPO-A (e.g., Qwen3-1.7B on MATH500 with ℓ=2048). With single-seed runs and no variance estimates, those reversals could be noise, but they undercut the confident prose.

Third, the paper cites Nath et al. and Park et al. as concurrent guided-GRPO work, but doesn't compare against Nath's "adaptive guidance" method. That's the one missing baseline that matters most. The paper also acknowledges that Eq. (5) breaks when batch difficulty varies, and patches it with curriculum ordering — fair, but it means the controller's premise (reward ratio ≈ learning progress) is untested in non-curriculum settings.

Bottom line: this is a worthwhile empirical contribution that deserves serious peer review, but it needs held-out α selection, multiple seeds, and a more honest characterization of when the method helps. I'd send it to review, expect a major revision, and would cite it as related work if I were working on small-model RLVR.

Referee Report

4 major / 5 minor

Summary. The paper studies Guided GRPO, which injects ground-truth reasoning traces into a subset of rollouts for small language models, and identifies two design axes: the fraction of guided candidates (guidance ratio α) and the guidance length (ℓ). It reports that naive fixed-length guidance is not helpful, and that the optimal guidance length is context-dependent. The proposed method, G2RPO-A, adjusts the guidance length at each training step via Eq. (5), which increases ℓ when recent rewards fall and decreases it when rewards rise, while a curriculum-learning ordering of the training data is used to stabilize the reward signal. Experiments on math (MATH500, Minerva, GPQA, AIME24/25) and code (HumanEval, LiveCodeBench) benchmarks with Qwen3-0.6B/1.7B/8B, DeepSeek-Math-7B, and DeepSeek-Coder-6.7B claim that G2RPO-A markedly and consistently outperforms vanilla GRPO. The paper also includes ablations over α, fixed guidance lengths, decay policies, and curriculum ordering.

Significance. If the reported gains are genuine, the paper makes a useful practical contribution: an adaptive guidance-length controller that mitigates the small-model RLVR reward-sparsity problem, together with an open-source implementation. The systematic ablation of guidance ratio and guidance length is a strength, as is the attempt to isolate the effect of curriculum ordering. The central claim, however, is currently not established to the advertised strength. The main evidence is an empirical comparison in which the guidance ratio is selected on the evaluation benchmarks themselves, and in which several benchmark–model pairs show G2RPO-A underperforming vanilla GRPO. Multiple seeds and a held-out or default-α comparison are needed before the claim of consistent and substantial gains can be accepted. The adaptive controller's premise that reward trends reflect competence is also not independently validated. The novelty relative to concurrent adaptive-guidance work (e.g., Nath et al., 2025) is not demonstrated experimentally.

major comments (4)

[Section 5.1, Tables 3–5 and Tables 8–9] The guidance ratio α is tuned on the evaluation benchmarks themselves. Tables 8 and 9 report accuracy on MATH500, Minerva, GPQA, HumanEval, and LiveCodeBench for several α values, and the paper then uses the best-performing α per model and task family in the headline results of Tables 3–5. This is test-set tuning: the reported gains of G2RPO-A over vanilla GRPO are confounded with the effect of choosing a favorable fixed intervention ratio. To support the claim that adaptive guidance improves performance, the authors should either (a) report results with a fixed default α (e.g., α=1/4 or α=1/2) without per-benchmark selection, or (b) split the benchmarks into a tuning set and a held-out evaluation set and show that the selected α transfers.
[Section 5.2, Tables 3, 4, 5, and Conclusion] The claim that G2RPO-A 'markedly surpasses vanilla GRPO on nearly every benchmark' and 'consistently boosts accuracy' is contradicted by several reported pairs. For example, Table 3 shows Qwen3-0.6B on MATH500 at 51.77 vs. GRPO 54.26; Qwen3-1.7B on Minerva at 15.10 vs. 16.19; Table 4 shows Qwen3-1.7B on MATH500 at 91.69 vs. 94.52; and Table 5 shows Qwen3-8B on HumanEval at 80.33 vs. 81.48 and Qwen3-1.7B on LiveCodeBench at 51.96 vs. 53.14. The paper should report all benchmark–model pairs with the sign of the difference and quantify the fraction of pairs where G2RPO-A wins, rather than relying on 'nearly every' and 'consistently.'
[Section 4.3, Eq. (5)] The adaptive controller assumes that a rise or fall in the current batch reward r_k is a faithful signal of the model's learning state. The paper itself flags in Section 4.3 that when consecutive batches differ in difficulty, reward variation no longer reflects true learning progress, and it mitigates this with a hand-chosen curriculum tier order. This premise is load-bearing but not independently verified. I suggest an additional control experiment: replace the reward-driven controller in Eq. (5) with (i) a random schedule constrained to the same guidance-length range and (ii) a schedule driven by rewards from a small held-out set; if the advantage of G2RPO-A over vanilla GRPO disappears against control (ii), the mechanism is not identifiable from the effect of a generally helpful intervention schedule.
[Section 5.1 and all result tables] No seed information, variance estimates, or confidence intervals are reported for any experiment. With training sets of 1,000 problems for 5 epochs and the small absolute differences seen in several benchmark pairs (e.g., Table 3 GPQA for Qwen3-0.6B: 30.39 vs. GRPO 24.51; Table 5 LiveCodeBench for Qwen3-1.7B: 51.96 vs. 53.14), a single run is insufficient to support the 'markedly surpasses' wording. At minimum, the authors should report the number of seeds and either standard deviations or per-run numbers.

minor comments (5)

[Section 4.1, guided objective] The displayed objective for Guided GRPO has unmatched delimiters: the sum over i contains a $\frac{1}{|o_i|+|g_i|}$ factor whose braces are not closed, making the equation difficult to parse. Please rewrite it with clear brackets for the guidance-token and output-token sums.
[Section 4.2, Eq. (4)] The parameters of the decay policies are not specified: the values of β, γ, and s used in the Table 2 experiments are not given, so the reader cannot reproduce the concave, linear, and stepwise decay curves.
[Tables 10 and 11] The abbreviation RDP is used in Tables 10 and 11 but is not defined in the main text; also, the text says that T=2 is 'already sufficient' and cites Tables 10 and 11, but those tables show guidance-length ablations rather than a sweep over history-window T. Either add a T-sweep table or correct the reference.
[Table 6] The table contains typos ('Minarva' for Minerva) and labels 'Random' and 'CL' without explaining how the random order was generated or whether it is matched to the CL order in content. Please clarify.
[General (appendix references)] The manuscript references an appendix with 'Detailed equations' and 'Appendix Figure 6,' but no appendix is present in the provided text. The full version must include the appendix or remove these references.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical comparison, and the adaptive guidance controller uses reward as a training feedback signal rather than as a definitional input.

full rationale

The paper's central claim is an empirical result: G2RPO-A, an adaptive guidance variant of GRPO, is reported to outperform vanilla GRPO on mathematical reasoning and code-generation benchmarks. No formal derivation is presented that would allow a result to reduce to its inputs by construction. Equation (5) updates the guidance length l_{k+1} using the current and past average rewards r_k, but this is a feedback control rule within training, not a definitional equivalence: the final benchmark scores are not defined by Eq. (5), and the update rule does not guarantee any particular evaluation outcome. The guidance length and ratio are hyperparameters that are selected via ablations; for example, Tables 8 and 9 report accuracy for different alpha values on the benchmark sets, and the caption of Table 3 states that 'guidance ratios are selected based on the best settings obtained from Table 9.' This is a test-set hyperparameter selection concern, which falls under tuning and selection bias, not circularity: the paper does not claim to derive the benchmarks from the alpha values, and the adaptive mechanism is still independently evaluated against fixed-guidance and rule-based-decay baselines in Tables 10 and 11. The paper also cites prior work for GRPO, curriculum learning, and filtering methods, but none of these citations is used to define G2RPO-A's contribution or to forbid alternatives; the cited works supply standard components and baselines. The self-identified limitation in Section 4.3, that reward variations may not reflect learning progress when consecutive batches differ in difficulty, is an acknowledged robustness caveat about the controller, not a circular step. Overall, the derivation chain is not circular; the main risks are empirical tuning and generalization, which are outside the scope of circularity analysis.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several empirical design choices rather than a mathematical derivation. The per-model guidance ratio alpha is selected from ablations on the same evaluation benchmarks, and initial guidance length and history window T are hand-set. The reward-trend controller is an ad hoc assumption. No new physical or formal entities are postulated.

free parameters (4)

guidance ratio alpha = varies per model/task: 0.75, 0.25, 0.14 (math); 0.75, 1.0, 0.57 (code)
Selected from Tables 8/9 based on best performance on the same evaluation benchmarks, then used in main Tables 3/5.
initial guidance length l0 = 3072 tokens
Set by authors; ablations in Tables 1 and 10-11 show large sensitivity to guidance length, so this choice is load-bearing.
history window T = 2
Reported as sufficient from Tables 10/11; no systematic search.
curriculum difficulty tiers = five ordered tiers: cn_contest, aops_forum, amc_aime, olympiads, olympiads_ref
Chosen by source metadata rather than measured difficulty; alternative ADARFT ordering rejected as uninformative.

assumptions (5)

domain assumption GRPO loss with importance weighting, clipping, and KL penalty (Eq. 1) is a valid optimization objective for policy training.
Imported from Shao et al. 2024 and used as the foundation; no re-derivation in this paper.
domain assumption Verifiable rewards (exact-match for math, tests for code) are sufficient training signals for reasoning.
Standard RLVR assumption inherited from cited literature.
domain assumption Injecting ground-truth reasoning tokens into a subset of rollouts improves sampling without collapsing the policy.
Core premise of Guided GRPO; supported empirically in this paper, not proven.
ad hoc to paper The running reward trend r_k is a faithful proxy for model competence, so Eq. (5) can control difficulty.
Ad hoc heuristic; the paper itself notes it fails when batch difficulty varies, and CL is used to compensate.
domain assumption Source-based tier order approximates problem difficulty for curriculum learning.
Uses dataset provenance (cn_contest, aops_forum, etc.) as difficulty proxy; alternative ADARFT ordering was uninformative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance." pith.science (2026). https://pith.science/paper/VJBC57L2

@misc{pith2026250813023,
  author       = {Pith},
  title        = {Pith review of: G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJBC57L2}},
  note         = {Machine review of arXiv:2508.13023}
}

abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has markedly enhanced the reasoning abilities of large language models (LLMs). Its success, however, largely depends on strong base models with rich world knowledge, yielding only modest improvements for small-size language models (SLMs). To address this limitation, we investigate Guided GRPO, which injects ground-truth reasoning steps into roll-out trajectories to compensate for SLMs' inherent weaknesses. Through a comprehensive study of various guidance configurations, we find that naively adding guidance delivers limited gains. These insights motivate G$^2$RPO-A, an adaptive algorithm that automatically adjusts guidance strength in response to the model's evolving training dynamics. Experiments on mathematical reasoning and code-generation benchmarks confirm that G$^2$RPO-A substantially outperforms vanilla GRPO. Our code and models are available at https://github.com/T-Lab-CUHKSZ/G2RPO-A.

Figures

Figures reproduced from arXiv: 2508.13023 by the authors.

**Figure 1.** Naive guidance does not help. Using Qwen2.5-Math-7B as the base model, we train it on the s1K-1.1 dataset for a single epoch with a simple, fixed-length guidance (naive guidance). The naive guidance method shows a temporary increase in the accuracy reward during the early training stages, but it quickly becomes indistinguishable from the vanilla GRPO curve. For the sequence of positive numbers: {𝑎𝑎𝑛𝑛} , the sum of t… view at source ↗

**Figure 2.** Illustration of roll-outs with guidance. An example of using high-quality thinking trajectories to guide models. [2025]. However, these methods precede RLVR or suffer performance degradation in complex problems ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of G2RPO-A. Each step we split roll-outs into a guided set and an unguided set. We then compare the current rewards with those from the previous steps; the resulting ratio determines the future guidance length. • Drawing on these findings, we introduce G2RPO-A, an adaptive algorithm that automatically adjusts guidance length in response to the evolving training state. Our experimental results demonstrate th… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Reward of Guided GRPO. We fine-tuned Qwen3-1.7B on coding tasks, using 10 roll-outs and generating 280 candidates per batch. The candidates’ rewards form a 20x14 matrix. We then applied 2x2 average pooling, reducing it to a 10x7 matrix for clearer visualization. The re…

**Figure 5.** Figure 5: Pitfalls of naive Guided GRPO. We trained Qwen3-1.7B-Base on a curriculum-ordered subset of Math-220K Wang et al. [2024]: problems are presented from easy to hard. Because the curriculum continually increases task difficulty, the accuracy reward does not plateau at a h…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
cs.LG 2026-07 conditional novelty 6.0 of 10

OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

Reference graph

Works this paper leans on

53 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

Online difficulty filtering for reasoning oriented reinforcement learning, 2025

Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning, 2025. URL https://arxiv.org/abs/2504.03380

arXiv 2025
[2]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021
[3]

SFT memorizes, RL generalizes: A comparative study of foundation model post-training

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In The Second Conference on Parsimony and Learning (Recent Spotlight Track), 2025. URL https://openreview.net/forum?id=d3E3LWmTar

work page 2025
[4]

Process reinforcement through implicit rewards

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. CoRR, 2025

work page 2025
[5]

Reinforcement learning for reasoning in small llms: What works and what doesn't, 2025

Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn't, 2025. URL https://arxiv.org/abs/2503.16219

arXiv 2025
[6]

Open r1: A fully open reproduction of deepseek-r1, January 2025

Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

2025
[7]

rstar-math: Small LLM s can master math reasoning with self-evolved deep thinking

Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small LLM s can master math reasoning with self-evolved deep thinking. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=5zwF1GizFa

work page 2025
[8]

Deepseek-coder: When the large language model meets programming-the rise of code intelligence

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. CoRR, 2024

work page 2024

Show all 53 references

[9]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[10]

Measuring mathematical problem solving with the MATH dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,...

2021
[11]

Efficoder: Enhancing code generation in large language models through efficiency-aware fine-tuning

Dong HUANG, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao QING, Heming Cui, Zhijiang Guo, and Jie Zhang. Efficoder: Enhancing code generation in large language models through efficiency-aware fine-tuning. In Forty-second International Conference on Machine Learning, 202...

2025
[12]

Boosting mllm reasoning with text-debiased hint-grpo, 2025

Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song. Boosting mllm reasoning with text-debiased hint-grpo, 2025. URL https://arxiv.org/abs/2503.23905

2025 arXiv
[13]

Openai o1 system card

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. CoRR, 2024

2024
[14]

Livecodebench: Holistic and contamination free evaluation of large language models for code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, 2024

2024
[15]

Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025

Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025. URL https://arxiv.org/abs/2506.00103

2025 arXiv
[16]

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

2022
[17]

Token-supervised value models for enhancing mathematical reasoning capabilities of large language models

Jung Hyun Lee, June Yong Yang, Byeongho Heo, Dongyoon Han, and Kang Min Yoo. Token-supervised value models for enhancing mathematical reasoning capabilities of large language models. CoRR, 2024

2024
[18]

Solving quantitative reasoning problems with language models

Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with l...

2022
[19]

Adaptive group policy optimization: Towards stable training and token-efficient reasoning

Chen Li, Nazhou Liu, and Kai Yang. Adaptive group policy optimization: Towards stable training and token-efficient reasoning. arXiv preprint arXiv:2503.15952, 2025 a

2025
[20]

Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions, 2024

Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions, 2024

2024
[21]

Torl: Scaling tool-integrated rl

Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025 b

2025 arXiv
[22]

Cppo: Accelerating the training of group relative policy optimization-based reasoning models, 2025

Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models, 2025. URL https://arxiv.org/abs/2503.22342

2025
[23]

Understanding r1-zero-like training: A critical perspective, 2025

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL https://arxiv.org/abs/2503.20783

2025 arXiv
[24]

Small language models: Survey, measurements, and insights

Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. CoRR, 2024

2024
[25]

s1: Simple test-time scaling

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models, 2025 a . URL htt...

2025
[26]

s1: Simple test-time scaling, 2025 b

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025 b . URL https://arxiv.org/abs/2501.19393

2025 arXiv
[27]

Adaptive guidance accelerates reinforcement learning of reasoning models, 2025

Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx. Adaptive guidance accelerates reinforcement learning of reasoning models, 2025. URL https://arxiv.org/abs/2506.13923

2025 arXiv
[28]

Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul Kim, Hanieh Deilamsalehy, Namyong Park, Mike Rimer, Zhehao Zhang, Huanrui Yang, Ryan A

Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, Junda Wu, Ashish Singh, Yu Wang, Jiuxiang Gu, Franck Dernoncourt, Nesreen K. Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul K...

2024 arXiv
[29]

Curriculum reinforcement learning from easy to hard tasks improves llm reasoning, 2025

Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, and Shuiwang Ji. Curriculum reinforcement learning from easy to hard tasks improves llm reasoning, 2025. URL https://arxiv.org/abs/2506.06632

2025
[30]

Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J. Kim. Deepvideo-r1: Video reinforcement fine-tuning via difficulty-aware regressive grpo, 2025. URL https://arxiv.org/abs/2506.07464

2025
[31]

Gpqa: A graduate-level google-proof q&a benchmark

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

2024
[32]

Proximal policy optimization algorithms, 2017

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

2017 arXiv
[33]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

2024 arXiv
[34]

Efficient reinforcement finetuning via adaptive curriculum learning, 2025

Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning, 2025. URL https://arxiv.org/abs/2504.05520

2025 arXiv
[35]

Code generation with small language models: A deep evaluation on codeforces, 2025

Débora Souza, Rohit Gheyi, Lucas Albuquerque, Gustavo Soares, and Márcio Ribeiro. Code generation with small language models: A deep evaluation on codeforces, 2025. URL https://arxiv.org/abs/2504.07343

2025
[36]

Qwq: Reflect deeply on the boundaries of the unknown

Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024

2024
[37]

Ni, Linyi Yang, Ying Wen, and Weinan Zhang

Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen, and Weinan Zhang. Openr: An open source framework for advanced reasoning with large language models, 2024. URL https://arxiv.org/abs/2410.09671

2024 arXiv
[38]

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

2022
[39]

Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

2025 arXiv
[40]

Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025

Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms, 2025. URL https://arxiv.org/ab...

2025 arXiv
[41]

Rlvr-world: Training world models with reinforcement learning, 2025

Jialong Wu, Shaofeng Yin, Ningya Feng, and Mingsheng Long. Rlvr-world: Training world models with reinforcement learning, 2025. URL https://arxiv.org/abs/2505.13934

2025
[42]

A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025. URL https://arxiv.org/abs/2504.11343

2025 arXiv
[43]

Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math, 2025

Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, Shuohang Wang, Weijian Xu, Jianfeng Gao, and Weizhu Chen. Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math, ...

2025 arXiv
[44]

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. CoRR, 2024

2024
[45]

Qwen3 technical report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

2025 arXiv
[46]

Treerpo: Tree relative policy optimization, 2025 b

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. Treerpo: Tree relative policy optimization, 2025 b . URL https://arxiv.org/abs/2506.05183

2025
[47]

Limo: Less is more for reasoning, 2025

Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. URL https://arxiv.org/abs/2502.03387

2025 arXiv
[48]

Dapo: An open-source llm reinforcement learning system at scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. CoRR, 2025

2025
[49]

Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837

2025 arXiv
[50]

R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization

Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. CoRR, 2025

2025
[51]

Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen

Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts, 2025. URL https://arxiv.org/abs/2506.02177

2025 arXiv
[52]

Disco balances the scales: Adaptive domain- and difficulty-aware reinforcement learning on imbalanced data, 2025

Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, and Furong Huang. Disco balances the scales: Adaptive domain- and difficulty-aware reinforcement learning on imbalanced data, 2025. URL https://arxiv.org/abs/2505.15074

2025
[53]

A technical study into 0.5b reasoning language models, 2025

Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, and Zheng Cao. A technical study into 0.5b reasoning language models, 2025. URL https://arxiv.org/abs/2506.13404

2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Online difficulty filtering for reasoning oriented reinforcement learning, 2025

Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning, 2025. URL https://arxiv.org/abs/2504.03380

arXiv 2025

[2] [2]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv 2021

[3] [3]

SFT memorizes, RL generalizes: A comparative study of foundation model post-training

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In The Second Conference on Parsimony and Learning (Recent Spotlight Track), 2025. URL https://openreview.net/forum?id=d3E3LWmTar

work page 2025

[4] [4]

Process reinforcement through implicit rewards

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. CoRR, 2025

work page 2025

[5] [5]

Reinforcement learning for reasoning in small llms: What works and what doesn't, 2025

Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn't, 2025. URL https://arxiv.org/abs/2503.16219

arXiv 2025

[6] [6]

Open r1: A fully open reproduction of deepseek-r1, January 2025

Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

2025

[7] [7]

rstar-math: Small LLM s can master math reasoning with self-evolved deep thinking

Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small LLM s can master math reasoning with self-evolved deep thinking. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=5zwF1GizFa

work page 2025

[8] [8]

Deepseek-coder: When the large language model meets programming-the rise of code intelligence

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. CoRR, 2024

work page 2024