Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

rStar2-Agent: Agentic Reasoning Technical Report

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read rStar2-Agent claims that 510 steps of agentic reinforcement learning—a model writing and running Python to check its own reasoning—lift a 14B model to 80.6% pass@1 on AIME24, matching or beating the 671B DeepSeek-R1 with far shorter answers

desk verdict A solid, honest agentic RL recipe with a new resampling method, but the headline 'surpassing DeepSeek-R1' is noise on 30-problem benchmarks and should be tempered. read the letter →

arxiv 2508.20722 v1 pith:J5UDUKID submitted 2025-08-28 cs.CL

Ning Shang , Yifei Liu , Yi Zhu , Li Lyna Zhang , Weijiang Xu , Xinyu Guan , Buze Zhang , Bingcheng Dong
show 7 more authors
Xudong Zhou Bowen Zhang Ying Xin Ziming Miao Scarlett Li Fan Yang Mao Yang
This is my paper
classification cs.CL
keywords agenticreinforcementlearningGRPO-RoCresample-on-correctmathematicalreasoningcode-interpretertooluse14BmodelAIMEbenchmarkstrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that agentic reinforcement learning—training a model to write and run Python code, then reason over the execution results—can by itself turn a 14B pre-trained model into a frontier-level math reasoner, without reasoning-heavy SFT and without huge model scale. The reported result is that rStar2-Agent-14B reaches 80.6% average pass@1 on AIME24 and 69.8% on AIME25 after only 510 RL steps on 64 GPUs within one week, which on average edges past the 671B DeepSeek-R1 (75.2% vs 74.9% across those two benchmarks) while answering in roughly 9–11K tokens versus the 14–17K the paper measures for other reasoning models. The mechanism is GRPO-RoC, a rollout-resampling strategy that filters noisy tool-use trajectories under pure answer-only rewards, plus a Python execution service that keeps thousands of concurrent tool calls fast and isolated, and a three-stage recipe that starts with a non-reasoning cold-start SFT. If correct, the work matters because it maps a concrete path to near-frontier reasoning on commodity hardware and shows that tool-augmented, shorter reasoning can beat longer chain-of-thought at the same accuracy.

What carries the argument

The load-bearing mechanism is GRPO-RoC, an RL algorithm pairing Group Relative Policy Optimization with a Resample-on-Correct rollout strategy. For each question it oversamples 2G multi-turn rollouts in the Python environment, then downsamples to G: negative trajectories are uniformly downsampled to keep diverse failure modes, while positive trajectories are preferentially kept when they have low tool-call error rates and clean formatting. Under an answer-only 0/1 reward, this asymmetric selection keeps the policy from treating buggy tool calls as acceptable; the paper shows it lowers tool-error rates in rewarded trajectories, raises accuracy, and shortens responses relative to vanilla GRPO

What would settle it

Re-run DeepSeek-R1 (and, if accessible, o3-mini and Claude Opus 4.0) on AIME24 and AIME25 using the paper's exact prompt template, 16 samples, temperature 0.6, and boxed-answer extraction: if matched-condition scores come out above 80.6% on AIME24 or above the paper's cross-benchmark average, the 'surpasses DeepSeek-R1' headline fails. A cheaper partial check is to reproduce Stage 1 from the released code and data and confirm AIME24 rises from the ~3.3% SFT baseline toward ~72% within 300 steps.

Watch

Extended reading notes

Core claim

The central discovery is that the bottleneck in agentic RL is the noise the tool environment injects into otherwise-correct trajectories: under naive GRPO, correctly answered rollouts still contain tool-call errors 10–15% of the time, and the model learns to tolerate them. GRPO-RoC oversamples 2G rollouts per question, then resamples down to G—failures kept uniformly to preserve error diversity, successes kept preferentially when clean and error-free. The authors report that this asymmetric selection cuts tool errors, raises AIME accuracy, and shortens responses, in only 510 RL steps, because the cold-start SFT teaches formatting and function-calling rather than reasoning. They also record t

Load-bearing premise

The load-bearing premise is that the published AIME scores for DeepSeek-R1, o3-mini, and Claude Opus 4.0 were produced under the same evaluation protocol the paper uses—16 samples at temperature 0.6 with the same prompt and answer extraction; if those baselines were scored differently, the claim of surpassing them does not follow.

Editorial extensions

If this is right

  • Frontier AIME-class math scores become reachable at 14B scale with one week of training on 64 GPUs, cutting the cost of a top-tier math reasoner by orders of magnitude versus 671B-scale training.
  • Tool-checked reasoning can be shorter than long CoT: at similar or better accuracy the model uses roughly 9–11K tokens per AIME response versus the 14–17K the paper measures for other reasoning models, lowering per-query inference cost.
  • Reasoning-heavy SFT is not needed: a cold-start SFT that only teaches formatting, function calling, and instruction following, followed by agentic RL, takes the model from 3.3%/0% to 80.6%/69.8% on AIME24/AIME25.
  • Math-only agentic RL transfers: GPQA-Diamond science reasoning rises from 42.1% to 60.9%, above DeepSeek-V3, implying the induced behaviors are general reasoning skills rather than memorized math templates.
  • There is a per-model ceiling: training past the 510-step peak collapses reward and accuracy despite multiple attempted fixes, so efficient recipes should aim to reach a base model's ceiling quickly rather than train long.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The RoC principle is reward-design-agnostic: oversampling rollouts, then preferring clean successes while retaining diverse failures, should reduce 'correct-but-sloppy' policy drift in any tool-augmented RL setting with sparse rewards—code generation, formal proof, or web-search agents—not just Python math.
  • A matched-protocol replay is the decisive test of the headline: re-running DeepSeek-R1 and other baselines under the paper's exact prompt, 16 samples, and temperature 0.6 would confirm or overturn the 'surpasses DeepSeek-R1' claim, which currently rests on comparing against published numbers.
  • The reported post-peak collapse suggests the efficient frontier for this recipe is a step budget near saturation; teams with limited GPU time should stop at the accuracy knee rather than continue training, since extra steps measurably hurt.
  • Because the recipe only needs verifiable integer answers, the same pipeline should transfer to any domain with cheap correctness checks where a model can run tools to verify its own work, such as code correctness or data-cleaning tasks with checksums.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning in a Python code environment. The core contributions are: (1) an efficient multi-turn RL infrastructure on 64 MI300X GPUs; (2) GRPO-RoC, a GRPO variant that oversamples 2G rollouts and then resamples positive trajectories toward high-quality, low tool-error traces while downsampling failures; and (3) a three-stage training recipe that starts from non-reasoning SFT and progressively increases response length and data difficulty. The authors claim 510 RL steps produce state-of-the-art AIME24/AIME25 pass@1 scores (80.6/69.8), surpassing DeepSeek-R1 (671B) with shorter responses, plus strong generalization to science, alignment, and tool-use benchmarks. The paper also reports unsuccessful attempts and a previously unreported training-collapse failure mode at the model's reasoning ceiling.

Significance. If the central claims hold, the paper is significant: it demonstrates that small models can reach frontier-level math reasoning with substantially less compute than large reasoning models, and it provides a concrete infrastructure and algorithm recipe. The paper is unusually honest in reporting failed interventions and the observed collapse under continued RL, which is valuable for the community. The main algorithmic result—GRPO-RoC improves tool-use quality and training efficiency over vanilla agentic GRPO—is broadly credible from the training curves, though the specific ablation is confounded (see major comments). The headline cross-model comparison, however, is not currently supported by the evidence presented.

major comments (4)
  1. [§5.2, Table 3] The abstract's central claim that rStar2-Agent-14B 'surpasses DeepSeek-R1 (671B)' is not supported by the evidence as presented. AIME24 and AIME25 each contain only 30 problems, and the pass@1 numbers are averaged over 16 samples per problem. The 0.8-point lead on AIME24 and the 0.2-point deficit on AIME25 are both within sampling noise. The paper reports no confidence intervals, per-problem scores, or multiple seeds. Moreover, baseline numbers for DeepSeek-R1, o3-mini, Claude-Opus-4.0, and QWQ-32B are taken from external sources whose sampling temperature, number of samples, prompt template, and tool-use policy are not shown to match the authors' protocol. The 'surpassing' statement should either be replaced by 'competitive with' or supported by a matched-protocol head-to-head evaluation with uncertainty quantification.
  2. [§4.3, Table 2, Abstract] The total number of RL steps is internally inconsistent. Stage 1 is 300 steps and Stage 3 is 125 steps; Stage 2 is described as using the latest policy 'from the final 385 steps of Stage 2', which implies a Stage 2 length of at least 385 steps, for a total of at least 810. The abstract, Table 2, and §4.3 state 510 RL steps. If '385' is a typo for '85', this must be corrected; if Stage 2 actually ran 385 steps, the 510-step claim and the compute-efficiency comparison in Table 2 need to be revised.
  3. [§5.3, Fig. 9] The ablation of Resample-on-Correct is confounded by the oversampling factor. GRPO-RoC generates 2G=32 rollouts per problem and then selects G=16, while the 'GRPO with Tool' baseline generates only G=16 rollouts and uses all of them. The observed gains could therefore be due to the larger candidate pool rather than the asymmetric quality filtering. An additional baseline that generates 32 rollouts and uses all of them—or otherwise controls for candidate count—is needed to attribute the improvement specifically to the RoC selection mechanism.
  4. [§4.3, §5.2] The final checkpoint is selected using the same AIME24/AIME25 benchmarks on which the headline numbers are reported. Fig. 8 shows evaluation scores being used to decide when to stop, and §4.3 states that 'performance begins to saturate and can even decline, so we stop at 125 steps.' This early stopping on the test set inflates the expected peak relative to a fixed training budget. Please report scores at a pre-specified step or use a held-out validation split for checkpoint selection, and acknowledge the selection effect in the reported pass@1 numbers.
minor comments (5)
  1. [Fig. 6 and §3.1] The text says the environment service achieves '45 calls per step' while also reporting 45K concurrent tool calls per step. This should be 45K; the inconsistency is confusing.
  2. [Table 6] 'V APO' is misspelled; should be 'VAPO'.
  3. [Eq. (6)] The normalization in the GRPO-RoC objective is unclear: after length-normalizing each trajectory inside the sum, the outer denominator appears to be the total selected token count. Please clarify the exact normalization and how it relates to the standard GRPO objective.
  4. [References] Several references are incomplete: [An et al.] lacks a year and venue, [Kimi] is a URL-only citation, and [Tulu3], [AIME], and [Huggingface] are datasets/websites without access dates or formal bibliographic information.
  5. [Fig. 10 and Fig. 11] The figures are described as highlighting top 20% high-entropy tokens in green, but the text-only rendering loses this color cue. Consider adding a textual marker or a separate annotation for accessibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL objective and RoC filter use training-set ground truths and tool-call quality; AIME/HMMT scores are independent measurements.

full rationale

The paper's derivation chain is: Qwen3-14B-base -> non-reasoning SFT -> agentic RL with GRPO-RoC -> evaluation on AIME24/AIME25/HMMT25. The training reward (Eq. 3) is binary answer correctness against ground-truth answers in its own curated training set; RoC (Eqs. 4-5) filters positive rollouts using tool-call error counts and formatting violations. Neither uses AIME24/AIME25 labels or external baseline scores. The reported 80.6/69.8/52.7 are 16-sample pass@1 measurements under a fixed prompt and sampling temperature, not quantities recovered from training inputs. Hyperparameters and the 510-step stopping point were selected from eval curves (Sec. 4.3 and Sec. 5.3), which is model selection rather than definitional circularity; it may inflate expected peaks but does not make the benchmark result equal to a fitted parameter by construction. The only overlapping-author citation (Guan et al. 2025, rStar-Math) appears in a general sentence on test-time scaling ('Test-time scaling has recently driven substantial advances in complex reasoning [Guan et al., 2025, Team et al., 2025]') and is not used to justify GRPO-RoC, the infrastructure, or the training recipe. The paper also discloses unsuccessful attempts and a failure mode (Sec. 4.3.1; Sec. 5.3: 'To our knowledge, this failure mode has not been reported publicly. We hypothesize the root cause is model capacity'), which are limitations relevant to external validity, not circular reasoning. Therefore there is no circular step.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a modest set of hand-chosen training constants and domain assumptions. The RoC scoring weights (p_err, p_format) are the most ad hoc elements; they shape which positive trajectories are used for policy updates. The integer-only data restriction and rule-based verifier are necessary to make rewards reliable. The ceiling hypothesis is an assumption explaining the observed collapse.

free parameters (5)
  • positive-trajectory quality weights (p_err, p_format) = p_err = 0.5 if no tool calls; p_format = min(1, (num_answer_tags-1)/num_turns)
    Hand-designed scoring in RoC biases positive sampling toward tool-using, clean trajectories; central to GRPO-RoC behavior.
  • oversample factor 2G (32 rollouts to 16 selected) = 2G=32, G=16
    RoC oversampling ratio set by hand; affects training cost and signal diversity.
  • Clip-Higher epsilon_high = 0.28
    From prior work (Yu et al. 2025), set to allow more exploration; impacts stability.
  • maximum response lengths per stage = 8K -> 12K -> 12K tokens
    Chosen to balance training cost and capacity; shorter than prior 16K-80K recipes.
  • number of RL steps per stage = 300 + 85 + 125 = 510
    Stopping points set by observed plateaus and eval scores; the final stage explicitly stops when AIME accuracy peaks before collapse.
assumptions (5)
  • domain assumption Integer-answer restriction: only math problems with integer answers are used for RL.
    Section 4.2. Ensures rule-based verifier can judge equivalence reliably; limits data diversity and generalizability.
  • domain assumption Rule-based verifier reliably judges integer answer equivalence.
    Section 4.2/2.2.1. Training rewards depend entirely on this verifier.
  • domain assumption Baseline scores in Table 3 are directly comparable despite different sampling protocols.
    Section 5.2. The paper compares its 16-sample pass@1 at temperature 0.6 with published numbers from closed models whose protocols are not stated to be identical.
  • domain assumption The pre-trained base model already contains the reasoning capacity; RL can only approach it, not exceed it.
    Section 5.3 'On the upper limit of RL-improved reasoning'. Explains the observed collapse after step 510; if false, the ceiling claim and early stopping are unjustified.
  • domain assumption Python execution feedback is accurate and non-misleading enough for RL to learn from it.
    Section 2.1. The environment returns output/error/timeout; environment noise is treated as a training signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of rStar2-Agent: Agentic Reasoning Technical Report." pith.science (2026). https://pith.science/paper/J5UDUKID

@misc{pith2026250820722,
  author       = {Pith},
  title        = {Pith review of: rStar2-Agent: Agentic Reasoning Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5UDUKID}},
  note         = {Machine review of arXiv:2508.20722}
}
read the original abstract

We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance. Beyond current long CoT, the model demonstrates advanced cognitive behaviors, such as thinking carefully before using Python coding tools and reflecting on code execution feedback to autonomously explore, verify, and refine intermediate steps in complex problem-solving. This capability is enabled through three key innovations that makes agentic RL effective at scale: (i) an efficient RL infrastructure with a reliable Python code environment that supports high-throughput execution and mitigates the high rollout costs, enabling training on limited GPU resources (64 MI300X GPUs); (ii) GRPO-RoC, an agentic RL algorithm with a Resample-on-Correct rollout strategy that addresses the inherent environment noises from coding tools, allowing the model to reason more effectively in a code environment; (iii) An efficient agent training recipe that starts with non-reasoning SFT and progresses through multi-RL stages, yielding advanced cognitive abilities with minimal compute cost. To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 (671B) with significantly shorter responses. Beyond mathematics, rStar2-Agent-14B also demonstrates strong generalization to alignment, scientific reasoning, and agentic tool-use tasks. Code and training recipes are available at https://github.com/microsoft/rStar.

Figures

Figures reproduced from arXiv: 2508.20722 by the authors.

Figure 1
Figure 1. rStar2-Agent-14B reaches frontier-level math reasoning in just 510 RL training steps. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. rStar2-Agent trains LLMs to natively use Python coding tools within the dedicated execution environment, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Our prompt template. Question will be replaced with the specific question during training. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Proportion of tool calls that contain errors within correctly answered trajectories. Under naive GRPO, the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The overall design of our agentic reinforcement learning infrastructure. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Our code environment demonstrates scalability by reliably handling up tp 45K concurrent tool calls per [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Top: Naively static rollout allocation leads to significant GPU idle time and synchronization delays. Bottom: our dynamic load-balanced scheduler that assigns rollouts based on available KV cache, dispatches tool call execution asynchronously, and balances computation …
Figure 8
Figure 8. Figure 8: AIME24/AIME25 accuracy and average training response lengths throughout multi-stage RL training. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Ablation of the Resample-on-Correct (RoC) rollout strategy. We compare our GRPO-RoC with two [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Example agentic RL trace#1 with coding tool use and self-reflection. (1) The model first invokes the [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Example agentic RL trace#2 with coding tool use and self-reflection. Top 20% high-entropy tokens are [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

    cs.AI 2026-05 reject novelty 7.0 of 10

    Training an LLM as a multi-turn agent that runs and repairs solver code raises verified optimization solve rates, with the 4B PEARL model outperforming DeepSeek-V3.2-685B in aggregate.

  2. AI Can Learn Scientific Taste

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Reinforcement learning on citation-preference pairs teaches a model to predict which papers will be cited more and to propose ideas that LLM judges rate as likely to be cited more—but "taste" here means citation impact.

  3. CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Replacing execution-error segments in RL trajectories with the model's own successful self-corrections improves accuracy on math and code benchmarks while cutting training steps by roughly two-thirds.

  4. Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.

  5. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0 of 10

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.

Reference graph

Works this paper leans on

31 extracted references · 29 linked inside Pith · cited by 5 Pith papers

  1. [1]

    Phi-4-reasoning technical report

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318,

    arXiv

  2. [3]

    Llama-nemotron: Efficient reasoning models

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949,

    arXiv

  3. [4]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    19 Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585,

    arXiv

  4. [5]

    Reasoning with exploration: An entropy perspective

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758,

    arXiv

  5. [6]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    arXiv

  6. [7]

    Retool: Reinforcement learning for strategic tool use in llms

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536,

    arXiv

  7. [8]

    Glaive function calling v2 dataset

    GlaiveAI. Glaive function calling v2 dataset. URL https://huggingface.co/datasets/glaiveai/ glaive-function-calling-v2 . Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519,

    arXiv

  8. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

Show all 31 references
  1. [10]

    Why distillation can outperform zero-rl: The role of flexible reasoning

    Xiao Hu, Xingyu Lu, Liyuan Mao, YiFan Zhang, Tianke Zhang, Bin Wen, Fan Yang, Tingting Gao, and Guorui Zhou. Why distillation can outperform zero-rl: The role of flexible reasoning. arXiv preprint arXiv:2505.21067,

    arXiv

  2. [11]

    Project euler

    Huggingface. Project euler. URL https://projecteuler.net/. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

    arXiv

  3. [13]

    Torl: Scaling tool-integrated rl

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383 ,

    arXiv

  4. [15]

    Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping

    URL https://arxiv.org/abs/ 2409.00920. Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506.13284, 2025b. 20 Michael Luo...

    arXiv

  5. [16]

    Xinji Mai, Haotian Xu, Weinong Wang, Yingying Zhang, Wenqiang Zhang, et al

    Notion Blog. Xinji Mai, Haotian Xu, Weinong Wang, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773,

    arXiv

  6. [17]

    Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with open- mathreasoning dataset

    Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with open- mathreasoning dataset. arXiv preprint arXiv:2504.16891,

    arXiv

  7. [18]

    Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay

    Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay. arXiv preprint arXiv:2504.03601,

    arXiv

  8. [19]

    Toolrl: Reward is all tool learning needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T ¨ur, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958,

    arXiv

  9. [20]

    Magistral.arXiv preprint arXiv:2506.10910,

    Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral.arXiv preprint arXiv:2506.10910,

    arXiv

  10. [21]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al

    URL https://openreview.net/forum?id=Ti67584b98. ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv pre...

  11. [22]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  12. [23]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    arXiv

  13. [24]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419,

    arXiv

  14. [25]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chen- zhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599,

    arXiv

  15. [26]

    URL https://arxiv.org/abs/2505.09388. Tulu3. Tulu3 sft instruction following dataset. URL https://huggingface.co/datasets/allenai/ tulu-3-sft-personas-instruction-following . Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Ya...

    arXiv

  16. [27]

    Magicoder: Empowering code generation with oss-instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120,

    arXiv

  17. [28]

    Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining

    LLM Xiaomi, Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, et al. Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining. arXiv preprint arXiv:2505.07608,

    arXiv

  18. [29]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al

    URL https://gorilla.cs.berkeley.edu/blogs/8_ berkeley_function_calling_leaderboard.html. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at sca...

    arXiv

  19. [30]

    Promoting efficient reasoning with verifiable stepwise reward

    Chuhuai Yue, Chengqi Dong, Yinan Gao, Hang He, Jiajun Chai, Guojun Yin, and Wei Lin. Promoting efficient reasoning with verifiable stepwise reward. arXiv preprint arXiv:2508.10293, 2025a. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does ...

    arXiv

  20. [31]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911,

    arXiv

  21. [2023]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864, 2025a

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864, 2025a. Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, S...

    arXiv

  22. [2024]

    Kimi-researcher

    Kimi. Kimi-researcher. URL https://moonshotai.github.io/Kimi-Researcher/. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. ar...

    arXiv

  23. [2025]

    Aime problems and solutions

    AIME. Aime problems and solutions. URL https://artofproblemsolving.com/wiki/index.php/AIME_ Problems_and_Solutions. Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-train...

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.