Pith. sign in

REVIEW 3 major objections 4 minor 11 cited by

Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Using Pass@k as the RLVR reward instead of Pass@1 improves exploration and later boosts Pass@1 accuracy, letting a 7B model surpass GPT-4o and Claude-3.7 on the Enigmata benchmark.

desk verdict Useful recipe, shaky theory: the analytical Pass@k advantage isn't the bootstrap limit it claims to be, but the two-stage training idea deserves a serious look. read the letter →

arxiv 2508.10751 v1 pith:OEOH5MG6 submitted 2025-08-14 cs.LG cs.AIcs.CL

Zhipeng Chen , Xiaobo Qin , Youbin Wu , Yue Ling , Qinghao Ye , Wayne Xin Zhao , Guang Shi This is my paper
classification cs.LGcs.AIcs.CL
keywords Pass@ktrainingRLVRreinforcementlearningwithverifiablerewardsexplorationvsexploitationlargereasoningmodelsadvantagefunctionGRPOimplicitrewarddesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes training large reasoning models with Pass@k as the reward in reinforcement learning with verifiable rewards (RLVR), instead of the usual Pass@1. The authors claim that Pass@k training encourages the policy to generate diverse candidate answers, raising Pass@k scores without hurting Pass@1, and that a subsequent Pass@1 training stage converts the exploration gains into stronger single-answer performance. On the Enigmata benchmark, a 7B model trained this way surpasses GPT-4o and Claude-3.7. The paper also derives a closed-form advantage function for Pass@k training, removing the sampling variance of bootstrap group construction, and shows that the resulting advantage design can be seen as a form of implicit reward design.

What carries the argument

The paper's central object is the Pass@k reward in RLVR and its associated advantage function. In Pass@1 training (GRPO), each response's advantage is its reward minus the group mean divided by the group standard deviation. Pass@k training instead groups responses into k-sized sets, scores each group as the maximum of its members' rewards, and derives closed-form response-level advantages: a positive response receives $(1-\bar{R}_{\text{group}})/\sigma_{\text{group}}$, while a negative response receives a weighted combination that depends on $N_{\text{neg}}$, $N_{\text{rollout}}$, and $k$. The analytical derivation (Eqs. 14–15) is what carries the argument: it shows that the advantage depend

What would settle it

Compare the gradient updates from bootstrap sampling (Eq. 6) with the analytical formulas (Eqs. 14–15) on a fixed batch: compute the per-response advantages both ways for varying $N_{\text{pos}}$ and measure the difference in the policy-gradient loss. If the analytical advantages differ from the mean bootstrap advantages by more than sampling noise, or if training with the analytical variant fails to reproduce the bootstrap variant's Pass@k gains, the equivalence claim fails.

Watch

Extended reading notes

Core claim

The central claim is that Pass@k, the probability that at least one of k sampled responses is correct, is a better RLVR reward than Pass@1 because it rewards policies for exploring diverse solution regions. The paper shows that replacing the Pass@1 objective with Pass@k in a GRPO/DAPO-style update improves downstream Pass@k performance while leaving Pass@1 intact, and that the exploration gained is later exploitable: adding a Pass@1 training phase on top of Pass@k training yields large Pass@1 gains, letting a 7B model exceed GPT-4o and Claude-3.7 on Enigmata. The analytical derivation of the advantage (Eqs. 14–15) computes the mean and standard deviation of group rewards under the full combi

Load-bearing premise

The analytical advantage formulas assume every response belongs to exactly $\binom{N_{\text{rollout}}-1}{k-1}$ groups, whereas the bootstrap procedure samples only $N_{\text{group}}=N_{\text{rollout}}$ groups with replacement, so the claimed variance reduction assumes these two updates are effectively equivalent.

Editorial extensions

If this is right

  • Pass@k training scales: more training steps or more rollouts continue to improve Pass@k scores, whereas Pass@1 training plateaus.
  • Combining Pass@k training with a subsequent Pass@1 training phase transfers exploration gains into Pass@1 accuracy across model families, sizes, and task types.
  • The analytical derivation makes Pass@k training efficient, removing the variance of bootstrap sampling without the overhead of full sampling.
  • Pass@k training is robust to the choice of k (4, 8, 16) and can be accelerated by increasing the learning rate.
  • The advantage-curve analysis suggests that down-weighting easy problems prevents local optima, opening a design space of implicit reward functions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The analytical advantage formula averages over all $\binom{N_{\text{rollout}}}{k}$ groups, but the bootstrap implementation samples only $N_{\text{group}}=N_{\text{rollout}}$ groups with replacement; the paper does not reconcile this expectation-vs-realization gap, so the practical equivalence of the two variants is an untested assumption.
  • If the benefit of Pass@k training comes from reweighting optimization toward harder problems, the same advantage-shaping idea could be applied to other verifiable tasks (e.g., code generation or theorem proving) without needing a closed-form Pass@k derivation.
  • The paper's Adaptive Training suggests that entropy-based monitoring, rather than entropy regularization, may be the right way to switch between Pass@k and Pass@1 objectives, a direction the paper leaves largely unexplored.
  • A direct falsifier is to compare the per-response advantages from bootstrap sampling (Eq. 6) with the analytical formulas (Eqs. 14–15) on a fixed batch; if they diverge beyond sampling noise, the analytical variant is not the asymptotic limit of the bootstrap update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Pass@k Training for RLVR of large reasoning models, using the Pass@k metric as a reward to encourage exploration during reinforcement learning. Three implementations are presented: full sampling, bootstrap sampling, and an analytical derivation of response-level advantages. The authors report that Pass@k Training improves Pass@k performance without harming Pass@1, increases response diversity and policy entropy, generalizes across tasks and model families, and that a subsequent Pass@1 Training stage can push a 7B model above closed-source baselines on Enigmata. The paper further interprets the analytical Pass@k advantage as a form of implicit reward design and explores several advantage-function variants. The manuscript includes pseudo-code, an appendix derivation, and a public repository.

Significance. If the results hold, the paper addresses a timely and important question: how the choice of reward metric in RLVR affects the exploration-exploitation trade-off. The combinatorial derivation of expected group rewards and advantages under a uniform-subset model is elegant and useful, and the empirical breadth (multiple model families, sizes, modalities, and benchmarks) is a strength. The public repository and pseudo-code support reproducibility. However, the central analytical derivation has a scaling mismatch with the bootstrap update it claims to approximate, and the empirical claims are supported by single-run experiments without error bars. These issues are substantive but appear fixable within the manuscript's scope.

major comments (3)
  1. [Section 2.4 / Appendix B, Eqs. (6), (14)-(15)] The analytical advantages in Eqs. (14)-(15) are conditional expectations of the group advantage for a response, but the bootstrap update in Eq. (6) is a sum over the realized groups containing that response. With N_group=Nrollout, each response appears in an expected N_group*k/Nrollout = k groups, so the expected bootstrap sum is k times Eqs. (14)-(15). The paper calls the analytical form the "theoretical asymptotic form" of bootstrap sampling (Section 2.4 Takeaway), yet no derivation connects Eq. (6) to Eqs. (14)-(15); indeed, as N_group grows, the bootstrap sum diverges unless renormalized. Because DAPO uses a clipped objective, a k-fold global scale is not automatically equivalent to a learning-rate change, and the stability improvement claimed in Figure 5 may be an artifact of this unresolved scaling difference. The authors should either average Eq. (6) over group memberships or mult
  2. [Sections 2-4, Tables 1-3 and 5-7] All training curves and tables appear to be single runs without error bars or multiple seeds. The paper makes strong claims such as "more stable training process" (Section 2.4) and "does not compromise Pass@1 performance" (Section 2.2), but Figure 5 shows a visible dip for bootstrap at 400 steps and no repeated trials are reported. Please report means and variances over at least three seeds for the main experiments (Tables 1-3 and Figures 4-5), or explicitly state that the results are single-run and temper the corresponding conclusions. This is essential for the central empirical claims.
  3. [Table 3, Section 3.5] The claim that Pass@k Training "does not compromise Pass@1 scores" is contradicted by the Seed1.5-VL-Small row: Pass@1 on MathVision decreases from 54.6 to 53.9 after +Pass@k Training, while the Average is essentially unchanged (62.9 -> 63.0). The conclusion in Section 3.5 that Pass@k benefits transfer without harming Pass@1 needs reconciliation with this drop, either by providing error bars and statistical tests or by weakening the claim to "does not substantially harm overall Pass@1."
minor comments (4)
  1. [Appendix B] The symbol K is used in some equations (e.g., Eqs. (20)-(26)) while the main text uses lowercase k; please unify. Also, Eq. (23) contains "N_bootstrap" which should likely be "N_rollout."
  2. [Section 2.4] The term "standard variance" is a misnomer; the quantity in Eq. (12) is a standard deviation. Consider renaming for clarity.
  3. [Throughout] Typos and inconsistent naming: "Engimata" should be "Enigmata," "effectivenss" should be "effectiveness," "LLM crush" should be "collapse," and some parenthetical formatting errors appear in equations, e.g., Eq. (5) has an extra parenthesis.
  4. [Figure 2] The figure uses both \hat{y}_i and \hat{Y}_j for responses and groups; the notation should be clarified in the caption or text to avoid confusion between individual responses and groups.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Pass@k advantage formulas are derived from the reward definition rather than fitted to results, and exploration claims have independent empirical support.

full rationale

The paper's central analytical claim (Section 2.4 and Appendix B) is a direct combinatorial derivation from the Pass@k reward definition (Eq. 5): group rewards, their mean, standard deviation, and response-level advantages are computed in closed form (Eqs. 7-15) from Nrollout, Npos, Nneg, and k. This derivation does not fit any parameter to the empirical outcomes it is used to explain, and it does not cite the authors' prior work as its justification. The reported exploration improvements are supported by independent measurements of answer diversity and policy entropy (Section 3.2) and by controlled comparisons against noise-reward and entropy-regularization baselines (Section 3.1). The reader-identified gap between bootstrap summation (Eq. 6) and the analytical per-group expectation (Eqs. 14-15) is a real statistical-validity concern about whether the analytical formula is the asymptotic form of the bootstrap, but it is not circularity: the analytical expression is not identical to the bootstrap estimator by construction, and the paper does not rename one as the other. Although the paper contains self-citations (e.g., [8], [9], [17]), these are background or infrastructure references and are not load-bearing for the main derivation. Under the stated rules, no quoted step exhibits a fitted parameter being called a prediction or a claimed result that is equivalent to its input by definition.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The method rests on binary verifier rewards, the DAPO advantage estimator, and a uniform-subset group model. Free parameters are mostly training hyperparameters, with Eq. 17 constants hand-tuned for a preliminary variant.

free parameters (4)
  • k in Pass@k = 8 (ablated 4, 8, 16)
    Controls group size; default hand-chosen in Appendix A.2 and varied in Section 3.4.
  • Nrollout = 32 (128 in full-sampling comparison)
    Rollout count chosen in Appendix A.2; changes advantage magnitudes through Eqs. 11-15.
  • Exceeding Pass@k transformation constants = f(Npos) = 0.4 * log(Npos + 0.5)
    Constants 0.4 and 0.5 in Eq. 17 are hand-chosen to shift the advantage peak to Npos=1; used only for the exploratory Exceeding Pass@k variant.
  • DAPO clip and learning rate = eps_low=0.2, eps_high=0.28, LR=1e-6 (2e-6 and 4e-6 ablated)
    Base RLVR hyperparameters from Appendix A.2; LR scaling is used in Section 3.4 to compensate for larger k.
assumptions (5)
  • domain assumption Verifier gives binary rewards Rpos=1 and Rneg=0.
    Section 2.1 states this reward setting; all derivations in Appendix B assume binary rewards.
  • domain assumption DAPO advantage estimation with clip-higher and token-level loss is the correct base for Pass@k Training.
    Section 2.1 and Appendix A.2 build the method on DAPO rather than generic RLVR.
  • domain assumption Pass@k reward is the maximum reward over k sampled responses.
    Eq. 5 defines Pass@k this way; it is an assumed objective, not derived.
  • domain assumption Groups are uniformly random k-subsets, and every response appears in exactly C(Nrollout-1,k-1) groups.
    Appendix B Eqs. 28-33 require this for closed-form advantages, but the bootstrap procedure in Eq. 6 uses a random number of groups per response.
  • domain assumption Gains on synthetic Maze and Enigmata transfer to mathematical and multimodal tasks.
    Sections 3.3-3.5 use cross-task generalization to support the central claims without a separate theoretical argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models." pith.science (2026). https://pith.science/paper/OEOH5MG6

@misc{pith2026250810751,
  author       = {Pith},
  title        = {Pith review of: Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEOH5MG6}},
  note         = {Machine review of arXiv:2508.10751}
}
abstract

Reinforcement learning with verifiable rewards (RLVR), which typically adopts Pass@1 as the reward, has faced the issues in balancing exploration and exploitation, causing policies to prefer conservative actions, converging to a local optimum. Identifying an appropriate reward metric is therefore crucial. Regarding the prior work, although Pass@k has been used in evaluation, its connection to LLM exploration ability in RLVR remains largely overlooked. To investigate this, we first use Pass@k as the reward to train the policy model (i.e., $\textbf{Pass@k Training}$), and observe the improvement on its exploration ability. Next, we derive an analytical solution for the advantage of Pass@k Training, leading to an efficient and effective process. Building on this, our analysis reveals that exploration and exploitation are not inherently conflicting objectives, while they can mutually enhance each other. Moreover, Pass@k Training with analytical derivation essentially involves directly designing the advantage function. Inspired by this, we preliminarily explore the advantage design for RLVR, showing promising results and highlighting a potential future direction.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

    cs.AI 2025-10 conditional novelty 7.0 of 10

    MENTOR selectively mixes expert and policy token distributions at high-entropy decision points, improving RLVR accuracy and diversity on math and out-of-domain benchmarks compared with full-trajectory imitation.

  2. Spectral Rewiring for Exploration, Purification, and Model Merging

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.

  3. Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MMPO jointly minimizes the first T moments of the per-problem failure-probability distribution — equivalent to minimizing expected truncated time to first success — and reports average gains over single-moment RL obje...

  4. Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Training with appended behavioral instructions plus correctness-filtered self-distillation improves held-out math pass@1 over DAPO for a 1.7B model, but not for 4B at 4K context.

  5. Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

    cs.CL 2026-07 unverdicted novelty 6.0 of 10

    Scaling zero-RL training to a 1T-parameter model yields competitive math performance and spontaneous cognitive behaviors such as self-verification and structured formatting.

  6. Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    CEEH selectively applies entropy regularization to hard questions and a shortest-correct-length penalty to easy ones, reducing reasoning length while preserving accuracy across six math benchmarks.

  7. Beyond the Sampled Token: Preserving Candidate Support in RLVR

    cs.AI 2025-10 conditional novelty 6.0 of 10

    SimKO, an asymmetric top-K token-level regularizer for RLVR, improves pass@K on math/logic benchmarks across K=1..256 without lowering pass@1 relative to GRPO.

  8. Outcome-based Exploration for LLM Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

  9. Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.

  10. TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation

    cs.AI 2026-08 conditional novelty 4.0 of 10

    Dense per-turn pass-ratio rewards improve multi-turn code repair (Pass@3) without changing first-attempt accuracy, across six models on LiveCodeBench.

  11. From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    A systematic analysis of LLM exploration in RLVR, introducing capability-boundary metrics and examining entropy-performance exchange across training stages and token levels.

Reference graph

Works this paper leans on

61 extracted references · 15 canonical work pages · cited by 11 Pith papers

  1. [1]

    L1: controlling how long A reasoning model thinks with reinforcement learning

    Pranjal Aggarwal and Sean Welleck. L1: controlling how long A reasoning model thinks with reinforcement learning. CoRR, abs/2503.04697, 2025

    arXiv 2025

  2. [2]

    Aime2024, 2024

    AIME2024. Aime2024, 2024. URL https://huggingface.co/datasets/HuggingFaceH4/aime_2024

    work page 2024

  3. [3]

    Aime2025, 2025

    AIME2025. Aime2025, 2025. URL https://huggingface.co/datasets/opencompass/AIME2025

    work page 2025

  4. [4]

    A survey of exploration methods in reinforcement learning.CoRR, abs/2109.00157, 2021

    Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning.CoRR, abs/2109.00157, 2021

    arXiv 2021

  5. [5]

    Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.CoRR, abs/2407.21787, 2024

    arXiv 2024

  6. [6]

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphaël Ségerie, Micah Carroll, Andi Peng, Phillip J. K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Mic...

    work page 2023

  7. [7]

    Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles.CoRR, abs/2505.19914, 2025

    Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, Hao Zhou, and Mingxuan Wang. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles.CoRR, abs/2505.19914, 2025

    arXiv 2025

  8. [8]

    Improving large language models via fine-grained reinforcement learning with minimum editing constraint

    Zhipeng Chen, Kun Zhou, Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 5694–5711. Association for ...

    work page 2024

Show all 61 references
  1. [9]

    An empirical study on eliciting and improving r1-like reasoning models.CoRR, abs/2503.04548, 2025

    Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving r1-like reasoning models.CoRR, abs/2503.04548, 2025

    2025 arXiv

  2. [10]

    Reasoning with exploration: An entropy perspective.CoRR, abs/2506.14758, 2025

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.CoRR, abs/2506.14758, 2025

    2025 arXiv

  3. [11]

    Thinker: Learning to think fast and slow.CoRR, abs/2505.21097, 2025

    Stephen Chung, Wenyu Du, and Jie Fu. Thinker: Learning to think fast and slow.CoRR, abs/2505.21097, 2025

    2025

  4. [12]

    The entropy mechanism of reinforcement learning for reasoning language models.CoRR, abs/2505.22617, 2025

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language mo...

    2025 arXiv

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  7. [15]

    Stochastic first- and zeroth-order methods for nonconvex stochastic program- ming

    Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic program- ming. SIAM J. Optim., 23(4):2341–2368, 2013

    2013

  8. [16]

    Bootstrap resampling methods: something for nothing?The Annals of thoracic surgery, 77(4):1142–1144, 2004

    Gary L Grunkemeier and YingXing Wu. Bootstrap resampling methods: something for nothing?The Annals of thoracic surgery, 77(4):1142–1144, 2004

    2004

  9. [17]

    Seed1.5-vl technical report.CoRR, abs/2505.07062, 2025

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Sihang Yuan,...

    2025 arXiv

  10. [18]

    Skywork open reasoner 1 technical report.CoRR, abs/2505.22312, 2025

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report.CoRR, abs/2505.22312, 2025

    2025 arXiv

  11. [19]

    Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025

    2025 arXiv

  12. [20]

    Advancing language model reasoning through reinforcement learning and inference scaling.CoRR, abs/2501.11651, 2025

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling.CoRR, abs/2501.11651, 2025

    2025 arXiv

  13. [21]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.CoRR, abs/2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.CoRR, abs/2503.24290, 2025

    2025 arXiv

  14. [22]

    Test-time learning for large language models.CoRR, abs/2505.20633, 2025

    Jinwu Hu, Zhitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models.CoRR, abs/2505.20633, 2025

    2025 arXiv

  15. [23]

    Openai o1 system card.CoRR, abs/2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya K...

    2024 arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  17. [25]

    PAG: multi-turn reinforced LLM self-correction with policy as generative verifier.CoRR, abs/2506.10406, 2025

    Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan. PAG: multi-turn reinforced LLM self-correction with policy as generative verifier.CoRR, abs/2506.10406, 2025

    2025 arXiv

  18. [26]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 arXiv

  19. [27]

    PP-PG: combining parameter perturbation with policy gradient methods for effective and efficient explorations in deep reinforcement learning

    Shilei Li, Meng Li, Jiongming Su, Shaofei Chen, Zhimin Yuan, and Qing Ye. PP-PG: combining parameter perturbation with policy gradient methods for effective and efficient explorations in deep reinforcement learning. ACM Trans. Intell. Syst. Technol., 12(3):35:1–35:21, 2021

    2021

  20. [28]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.CoRR, abs/2505.24864, 2025

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.CoRR, abs/2505.24864, 2025

    2025 arXiv

  21. [29]

    Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards.CoRR, abs/2505.13445, 2025

    Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards.CoRR, abs/2505.13445, 2025

    2025 arXiv

  22. [30]

    Understanding r1-zero-like training: A critical perspective.CoRR, abs/2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.CoRR, abs/2503.20783, 2025

    2025 arXiv

  23. [31]

    Inference-time scaling for generalist reward modeling.CoRR, abs/2504.02495, 2025

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling.CoRR, abs/2504.02495, 2025

    2025

  24. [32]

    Learning from peers in reasoning models.CoRR, abs/2505.07787, 2025

    Tongxu Luo, Wenyu Du, Jiaxi Bi, Stephen Chung, Zhengyang Tang, Hao Yang, Min Zhang, and Benyou Wang. Learning from peers in reasoning models.CoRR, abs/2505.07787, 2025

    2025 arXiv

  25. [33]

    Exploring the limit of outcome reward for learning mathematical reasoning.CoRR, abs/2502.06781, 2025

    Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathemati...

    2025 arXiv

  26. [34]

    Nesterov and Vladimir G

    Yurii E. Nesterov and Vladimir G. Spokoiny. Random gradient-free minimization of convex functions.Found. Comput. Math., 17(2):527–566, 2017

    2017

  27. [35]

    Moss, Alessandro Sordoni, Rishabh Agarwal, and Arian Hosseini

    Kusha Sareen, Morgane M. Moss, Alessandro Sordoni, Rishabh Agarwal, and Arian Hosseini. Putting the value back in RL: better test-time scaling by unifying LLM reasoners with verifiers.CoRR, abs/2505.04842, 2025

    2025 arXiv

  28. [36]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

    2017 arXiv

  29. [37]

    Amrith Setlur, Matthew Y. R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms.CoRR, abs/2506.09026, 2025

    2025 arXiv

  30. [38]

    Spurious rewards: Rethinking training signals in RLVR.CoRR, abs/2506.10947, 2025

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in RLVR.CoRR, abs/250...

    2025 arXiv

  31. [39]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300, 2024

    2024 arXiv

  32. [40]

    Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Ha...

    2024

  33. [41]

    Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models

    Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen. Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. CoRR, abs/2503.21380, 2025

    2025 arXiv

  34. [42]

    Optimizing language models for inference time objectives using reinforcement learning.CoRR, abs/2503.19595, 2025

    Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and Rémi Munos. Optimizing language models for inference time objectives using reinforcement learning.CoRR, abs/2503.19595, 2025

    2025 arXiv

  35. [43]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025 arXiv

  36. [44]

    Reft: Reasoning with reinforced fine-tuning

    Luong Quoc Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August ...

    2024

  37. [45]

    Pass@k policy optimization: Solving harder reinforcement learning problems

    Christian Walder and Deep Karkhanis. Pass@k policy optimization: Solving harder reinforcement learning problems. CoRR, abs/2505.15201, 2025

    2025 arXiv

  38. [46]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu...

    2024

  39. [47]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effe...

    2025 arXiv

  40. [48]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229–256, 1992

    1992

  41. [49]

    ARM: adaptive reasoning model

    Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. ARM: adaptive reasoning model. CoRR, abs/2505.20258, 2025

    2025

  42. [50]

    Logic-rl: Unleashing LLM reasoning with rule-based reinforcement learning.CoRR, abs/2502.14768, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing LLM reasoning with rule-based reinforcement learning.CoRR, abs/2502.14768, 2025

    2025 arXiv

  43. [51]

    Qwen2.5 technical report.CoRR, abs/2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  44. [52]

    DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

    2025 arXiv

  45. [53]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?CoRR, abs/2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?CoRR, abs/2504.13837, 2025

    2025 arXiv

  46. [54]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.CoRR, abs/2503.18892, 2025

    WeihaoZeng, YuzhenHuang, QianLiu, WeiLiu, KeqingHe, ZejunMa, andJunxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.CoRR, abs/2503.18892, 2025

    2025 arXiv

  47. [55]

    Boning, and Dina Katabi

    Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wei Hong, Duane S. Boning, and Dina Katabi. RL tango: Reinforcing generator and verifier together for language reasoning.CoRR, abs/2505.15034, 2025

    2025

  48. [56]

    Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference

    Qining Zhang and Lei Ying. Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025

    2025

  49. [57]

    What, how, where, and how well? A survey on test-time scaling in large language models.CoRR, abs/2503.24235, 2025

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? A survey on test-time scaling in large language models.CoRR, abs/2503.24235, 2025

    2025 arXiv

  50. [58]

    Openrft: Adapting reasoning foundation model for domain-specific tasks with reinforcement fine-tuning.CoRR, abs/2412.16849, 2024

    Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, and Jitao Sang. Openrft: Adapting reasoning foundation model for domain-specific tasks with reinforcement fine-tuning.CoRR, abs/2412.16849, 2024

    2024 arXiv

  51. [59]

    The surprising effectiveness of negative reinforcement in LLM reasoning.CoRR, abs/2506.01347, 2025

    Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in LLM reasoning.CoRR, abs/2506.01347, 2025

    2025

  52. [60]

    Eric R. Ziegel. Resampling methods.Technometrics, 48(4):576, 2006

    2006

  53. [61]

    S”, “E”, “*

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. TTRL: test-time reinforcement learning.CoRR, abs/2504.16084, 2025. 24 Appendix A Experiment Setup A.1 Details of Downstream Tasks In this section, we present ...

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.