Pith. sign in

REVIEW 5 major objections 7 minor 6 cited by

First Return, Entropy-Eliciting Explore

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces FR3E, which targets partial rollouts at high-entropy tokens to stabilize RLVR training and improve math reasoning accuracy.

desk verdict FR3E is a coherent integration of entropy-guided anchors and partial rollouts with modest but consistent gains; the mechanism is under-validated but the paper is worth refereeing. read the letter →

arxiv 2507.07017 v1 pith:X3FYPPVP submitted 2025-07-09 cs.AI

Tianyu Zheng , Tianshun Xing , Qingshui Gu , Taoran Liang , Xingwei Qu , Xin Zhou , Yizhi Li , Zhoufutu Wen
show 5 more authors
Chenghua Lin Wenhao Huang Qian Liu Ge Zhang Zejun Ma
This is my paper
classification cs.AI
keywords reinforcementlearninglargelanguagemodelsRLVRtokenentropycreditassignmentmathematicalreasoningstructuredexplorationpartialrollouts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FR3E (First Return, Entropy-Eliciting Explore), a reinforcement-learning method for large language models that tries to fix credit assignment in verifiable-reward tasks without a critic or step-level labels. The idea is to first generate a base reasoning trajectory, compute the model's next-token entropy at every position, and pick the top-K highest-entropy tokens as forks where the reasoning could plausibly go another way. FR3E then launches several partial rollouts from each such fork, scores them by final correctness, and uses those scores to build an empirical value for each intermediate state and to modulate the advantage used in policy updates. The authors report that, compared with a GRPO-style baseline with rejection sampling and clip-higher, FR3E keeps policy entropy higher during training, produces longer and more coherent responses, and increases the share of trajectories that are fully correct, with AIME24 accuracy improving by up to 6.1 points on a 32B model. A sympathetic reader would care because the method converts the model's own uncertainty into a cheap, dense training signal, potentially making RLVR more stable and more sample-efficient.

What carries the argument

The load-bearing object is the entropy profile of the policy along the base trajectory. At each position k, the model computes H_k = -\sum_v \pi(v \mid q, t_{<k}) \log \pi(v \mid q, t_{<k}) over the vocabulary, selects the K largest H_k values, and uses those positions to split the trajectory into semantic blocks. Each block boundary defines an intermediate state S_j = (q, B_1, \ldots, B_j), from which M rollouts are generated and scored with a binary correctness reward; the empirical mean is V(S_j). This value estimate feeds an adaptive advantage modulation alpha_j = 1 / \exp(V(S_j) - V(S_{j-1})), and the policy is updated with an asymmetric clip-higher objective. Rejection sampling removes prompts whose rollouts all share one reward, so the entropy-finding phase sees varied outcomes. Together these components convert a sparse final reward into localized, model-derived guidance without training a critic.

What would settle it

Run FR3E with the top-K entropy positions replaced by K randomly chosen positions, or by the K lowest-entropy positions, while keeping all rollouts and advantage modulation identical; if the AIME24 gains and the All-Right versus All-Wrong trends persist, then entropy-based selection is not the active ingredient. A complementary check is to annotate a sample of trajectories and measure how often high-entropy tokens coincide with steps a human would call pivots, such as a new equation, a substitution, or a conclusion, rather than with formatting tokens.

Watch

Extended reading notes

Core claim

The central claim is that token-level entropy marks semantically important decision points in a reasoning chain, so starting additional rollouts from those points gives targeted, semantically grounded feedback that is normally unavailable. Concretely, FR3E treats the highest-entropy tokens in a base trajectory as anchors, segments the trajectory into blocks at those anchors, and samples M continuations from each intermediate state S_j. The average correctness of those continuations defines V(S_j), and the advantage for each sampled path is rescaled by alpha_j = 1 / exp(V(S_j) - V(S_{j-1})), which dampens updates when value is improving and amplifies them when progress stalls. The paper argues that this stabilizes training, keeps entropy from collapsing, and increases the fraction of trajectories that are fully correct rather than all-wrong, and it reports gains on AIME24 and seven other math benchmarks, with the largest gains on general-purpose models.

Load-bearing premise

The load-bearing premise is that the tokens where the model's next-token distribution is most spread out are exactly the places where a reasoning trajectory could meaningfully fork; if those high-entropy positions are mostly punctuation, connectives, or restatements, the partial rollouts add compute without providing useful feedback and the reported gains cannot be attributed to the mechanism.

Editorial extensions

If this is right

  • If FR3E works as claimed, verifiable-reward RL for reasoning can get dense feedback from the model's own uncertainty signal, without training a value network or labeling intermediate steps.
  • Training stability should improve in settings where GRPO-style methods suffer entropy collapse, particularly on specialized math models where longer reasoning chains need sustained exploration.
  • Because the framework needs only final correctness and no reward model, it should transfer to other verifiable domains such as code generation and theorem proving.
  • The reported AIME24 improvements on general-purpose 7B and 32B models suggest the largest gains come when a base model has room to learn reasoning, not when it already carries strong math priors.
  • Partial rollouts from intermediate states have a different inference-cost profile than full rollouts, so the method's practical appeal depends on whether the stability gains justify the extra compute at the chosen M.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The premise that top-K entropy marks semantic forks is adopted from prior work without fresh evidence; if high-entropy positions are dominated by formatting, connective, or restatement tokens, a natural test is to replace the entropy selector with random positions or lowest-entropy positions and check whether the gains survive.
  • The empirical state value V(S_j) is estimated from M rollouts, but M is not reported and the variance of these estimates is not analyzed; whether the advantage modulation helps or merely adds noise may depend on M and on how many rollouts land correct.
  • One implicit consequence is a possible repair mechanism: by anchoring exploration at uncertain but already partially correct trajectories, FR3E may be learning to fix near-miss solutions, which would explain the rise in All-Right and drop in All-Wrong trajectories; a direct check would compare rollouts from high-entropy forks against rollouts from the final answer step.
  • Another extension the paper does not explore is using the learned entropy profile as a signal for where to allocate test-time compute, not just training compute, since the same forks that guide exploration could prioritize verification effort during inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes FR3E (First Return, Entropy-Eliciting Explore), an RLVR algorithm that first generates base reasoning trajectories, computes token-level entropy at each position, selects the top-K highest-entropy tokens as anchors, and then performs M partial rollouts from each anchor state. The empirical value V(Sj) of each intermediate state is estimated from these rollouts, and an adaptive advantage modulation factor alpha_j = 1/exp(V(Sj)-V(Sj-1)) scales the per-token advantages inside a PPO-style objective with clip-higher and rejection sampling. The authors report that FR3E improves training stability, produces longer and more coherent responses, and increases the proportion of fully correct trajectories compared to a GRPO++ baseline on AIME24 and seven other benchmarks across Qwen2.5-7B, Qwen2.5-Math-7B, and Qwen2.5-32B.

Significance. If the entropy-guided partial-rollout mechanism is genuinely responsible for the reported gains, FR3E is a valuable and practical contribution to RLVR for LLM reasoning: it avoids a learned critic, uses only outcome rewards, and is simple to implement within existing frameworks such as VeRL. The paper is honest about the computational mismatch with the baseline in Section 5.2, and it makes the full system available through a project page. However, the central attribution claim is not yet supported: there is no ablation isolating entropy-based anchor selection from random anchors or uniform segmentation, and the number of rollouts M and other key hyperparameters are not reported. The theoretical stability argument in Eq. (16) is tautological rather than substantive. As a result, the empirical improvements, while plausible, remain unverified as evidence for the proposed mechanism.

major comments (5)
  1. [§4.1.2–§4.1.4, §5.3.2] The central premise that top-K token entropy marks semantically important reasoning forks is adopted from reference [33] without validation in this paper's setup. The paper provides no ablation comparing entropy-based anchors against random anchor selection, uniform block segmentation, or lowest-entropy anchors. Since partial rollouts from any mid-trajectory position could add useful exploration, the reported gains may stem from extra computation rather than from entropy elicitation. Please add an anchor-selection ablation with matched rollout counts and report the resulting accuracy and training stability.
  2. [Eq. (13), Eq. (14), Eq. (16), §4.3] The claim that the average modulated advantage is exactly zero is a trivial consequence of defining V(Sj) as the empirical mean of the rollouts from that state; Eq. (16) and Eq. (17) show that each block's inner sum vanishes by construction, independent of alpha_j. This does not demonstrate stability. The subsequent statement that the policy gradient estimator is 'approximately unbiased' is unsupported: alpha_j depends on V(Sj) estimated from the same rollouts used to compute advantages, and the exponential form in Eq. (14) can amplify small differences. Please provide an explicit derivation of the bias/variance properties or remove the unbiasedness claim.
  3. [§5.2] The paper acknowledges that matching compute with GRPO++ is infeasible and that partial rollouts have a different computational footprint. Without a compute-matched baseline or a report of wall-clock time and FLOPs, the improvements in Table 1 could be attributed to the additional inference cost of partial rollouts rather than to the FR3E mechanism. Please report the total compute, the values of K and M, and, if possible, a GRPO++ baseline with an equivalent total rollout budget.
  4. [§5.1, Table 1] The evaluation protocol is inconsistent across benchmarks: AIME24 uses avg@32 while all other benchmarks use greedy decoding. The multi-benchmark comparison therefore mixes two different evaluation regimes, and the reported differences are small (often 0.0–3.5%) without error bars or multiple seeds. Please provide confidence intervals or significance tests, and state clearly why AIME24 alone uses sampling.
  5. [§5.3.2, Figures 7 and 8] The 'All-Right' and 'All-Wrong' trajectory counts are shaped by the method's own sampling strategy, including rejection sampling and entropy-targeted rollouts, so these curves are not a neutral diagnostic of trajectory quality. Please clarify the exact definition of these counts (e.g., measured over which rollouts, at which training steps) and provide the equivalent curves for GRPO++ under comparable conditions.
minor comments (7)
  1. [§3.4] The phrase 'First Return, Then Explor' appears to be a typo for 'First Return, Then Explore'.
  2. [Figure 1] The figure caption contains an apparent template artifact ('stage 名字') that should be removed.
  3. [Eq. (6) and Eq. (7)] The symbol K is used both for the hyperparameter (number of selected positions) and for the set of entropy-sensitive positions, which is confusing; please use distinct notation, e.g., K for the set and k for the cardinality.
  4. [§3.3] The specific clip values epsilon_low = 0.22 and epsilon_high = 0.28 are stated without justification or sensitivity analysis; please cite a source or provide an ablation.
  5. [§5.2, Figure 4b] The sentence contains a formatting error: 'AIME24 (Figure 4b' is missing a closing parenthesis. Please fix.
  6. [Appendix A] The rationale for the data setting is given as 'for the sake of experimental consistency and inertia,' which is not a scientific justification; the appendix itself notes that the conclusion may not transfer to other base models. Please provide evidence or temper the claim.
  7. [References] Reference [7] (PRIME) has a placeholder-like arXiv identifier (2504.00052) and appears to be duplicated in content with reference [6]; please verify the citation details.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FR3E's value estimates come from Monte Carlo rollouts and the entropy-selection premise is cited from external work, not derived from the target benchmarks.

full rationale

The paper's derivation chain is self-contained rather than circular. The token-level entropy H_k (Eq. 5) is a standard definition, the top-K selection (Eq. 7) is explicitly adopted from external prior work [33], and the intermediate-state value V(S_j) (Eq. 13) is a Monte Carlo estimate of outcome rewards from partial rollouts, not a parameter fitted to AIME24 or any other evaluation benchmark. The modulation factor alpha_j and advantage A' (Eqs. 14-15) are defined from these estimates, and the zero-mean property (Eqs. 16-17) is explicitly shown to follow by definition of the empirical value, so it is not presented as an empirical prediction. The central empirical claims are external benchmark comparisons against GRPO++. The main caveats are that the paper does not ablate random anchor selection against entropy-based selection, and Section 5.2 acknowledges that compute-matched comparison with GRPO++ is infeasible; these are experimental-control and robustness concerns, not circularity. There are no load-bearing self-citations and no fitted input is renamed as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical or architectural entities; its load-bearing elements are the entropy-selection rule and the Monte Carlo state-value estimates, which are assumptions, not inventions. The adaptive advantage modulation factor alpha_j is a mathematical construction, not a new entity.

free parameters (4)
  • K (number of top-entropy anchors per trajectory)
    Central hyperparameter of the method; selects how many partial-rollout anchor points are used. Chosen by hand and not reported in the paper.
  • M (number of partial rollouts per anchor state)
    Determines the empirical state value V(Sj); not reported. Directly affects the variance of advantage estimates and is central to the method.
  • clip thresholds epsilon_low and epsilon_high = 0.22, 0.28
    Inherited from DAPO and VAPO; not fitted here, but they shape the policy update and are hand-chosen.
  • learning rate and batch size = 1e-6, 512
    Standard training hyperparameters reported in Section 5.1; not fitted to the target result but required to reproduce.
assumptions (4)
  • domain assumption Token-level entropy of the policy marks semantically important decision points in a reasoning trajectory.
    Invoked in Sections 4.1.2 and 4.1.3 to justify selecting top-K entropy tokens as forks for exploration; adopted from [33] without new validation in this paper.
  • domain assumption A rollout from an intermediate state Sj, scored by final answer correctness, yields a reliable estimate V(Sj) of that state's value.
    Used in Sections 4.2 and 4.3 as the baseline and feedback signal; depends on M rollouts being sufficient, which is not reported.
  • standard math The RLVR reward, a binary correct-final-answer signal, is a valid training signal for math reasoning.
    Standard assumption of the RLVR setup, inherited from prior work; not defended in this paper.
  • standard math PPO and GRPO-style policy optimization with clipped ratios is a sound update rule.
    Assumed throughout Sections 3 and 4; standard result from [25] and [29].

how reviews work

0 comments
Cite this review

Pith. "Pith review of First Return, Entropy-Eliciting Explore." pith.science (2026). https://pith.science/paper/X3FYPPVP

@misc{pith2026250707017,
  author       = {Pith},
  title        = {Pith review of: First Return, Entropy-Eliciting Explore},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3FYPPVP}},
  note         = {Machine review of arXiv:2507.07017}
}
read the original abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    RAIL learns rollout interventions from realized recoverability gains via an online contextual bandit and improves GRPO-based agentic post-training under limited rollout budgets.

  2. Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

    cs.LG 2025-10 conditional novelty 6.0 of 10

    HINT boosts LLM reasoning RL by injecting teacher-generated heuristic hints only on all-failed rollouts, keeping hints out of the policy-optimization prompt, and monitoring guidance quality with a new Affinity metric.

  3. Outcome-based Exploration for LLM Reasoning

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Outcome-based exploration bonuses (UCB-Con and Batch) improve pass@1 and pass@32 for LLM math reasoning while slowing diversity collapse, supported by a bandit model with a strong generalization assumption.

  4. Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning

    cs.AI 2025-08 conditional novelty 6.0 of 10

    A new RL reward-shaping method that rewards low confidence on hard problems and high confidence on easy ones improves LLM math reasoning over a GRPO baseline.

  5. Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

    cs.LG 2026-07 conditional novelty 5.0 of 10

    CPO uses the log-ratio of reference-guided to vanilla token probabilities as a correctness signal for per-token advantage shaping in RLVR, beating entropy-based methods on math and generalization benchmarks.

  6. Self-Reflective Generation at Test Time

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.

Reference graph

Works this paper leans on

42 extracted references · 8 canonical work pages · cited by 6 Pith papers

  1. [33]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning,

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning,

    work page

  2. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740

    arXiv 2024

  3. [2]

    Using confidence bounds for exploitation-exploration trade-offs.JournalofMachineLearningResearch, 3(Nov):397–422, 2002

    Peter Auer. Using confidence bounds for exploitation-exploration trade-offs.JournalofMachineLearningResearch, 3(Nov):397–422, 2002

    work page 2002

  4. [3]

    Finite-time analysis of the multiarmed bandit problem

    Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235–256, 2002

    2002

  5. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [5]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025. URLhttps://arxiv.org/abs/2412.21187

    arXiv 2025

  7. [6]

    Process reinforcement through implicit rewards, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URLhttps://arxiv.org/abs/2502.01456

    arXiv 2025

  8. [7]

    Assessing Validity of ICD-10 Administrative Data in Coding Comorbidities

    Yutong Cui, Hongyi Zhang, and et al. Prime: Process reinforcement with implicit model estimation for llms. arXiv preprint arXiv:2504.00052, 2025

    work page Pith review arXiv 2025

Show all 42 references
  1. [8]

    S-grpo: Early exit via reinforcement learning in reasoning models, 2025

    Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models, 2025. URL https://arxiv.org/abs/2505.07686

    2025 arXiv

  2. [9]

    Go-explore: a new approach for hard-exploration problems.arXiv preprint arXiv:1901.10995, 2019

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems.arXiv preprint arXiv:1901.10995, 2019

    1901 arXiv

  3. [10]

    Stanley, and Jeff Clune

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580–586, February 2021. ISSN 1476-4687. doi: 10.1038/s41586-020-03157-9. URL http: //dx.doi.org/10.1038/s41586-020-03157-9

    2021 doi

  4. [11]

    A survey on mathematical reasoning and optimization with large language models.arXiv preprint arXiv:2503.17726, 2025

    Ali Forootani. A survey on mathematical reasoning and optimization with large language models.arXiv preprint arXiv:2503.17726, 2025

    2025 arXiv

  5. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [13]

    Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning, 2025

    Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning, 2025. URLhttps://arxiv.org/abs/2507.00432

    2025 arXiv

  7. [14]

    Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment, 2024

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment, 2024. URL https://arxiv.org/abs/2410.01679

    2024 arXiv

  8. [15]

    Temporal sampling for forgotten reasoning in llms, 2025

    Yuetai Li, Zhangchen Xu, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Xiang Yue, and Radha Poovendran. Temporal sampling for forgotten reasoning in llms, 2025. URLhttps://arxiv.org/ abs/2505.20196

    2025 arXiv

  9. [16]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. URLhttps://arxiv.org/abs/2305. 20050

    2023

  10. [17]

    Intelligent go-explore: Standing on the shoulders of giant foundation models, 2025

    Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models, 2025. URL https://arxiv.org/abs/2405.15143. 14

    2025 arXiv

  11. [18]

    Improve mathematical reasoning in language models by automated process supervision, 2024

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URLhttps://arxiv.org/abs/2406.06592

    2024 arXiv

  12. [19]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Sur- passing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepScaleR-Surpa...

  13. [20]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention, 2025

    MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haiga...

    2025 arXiv

  14. [21]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, and et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022

    2022

  15. [22]

    A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072, 2023

    Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072, 2023

    2023 arXiv

  16. [23]

    Qwen2.5 technical report, 2025

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025 arXiv

  17. [24]

    Sequence level training with recurrent neural networks, 2016

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks, 2016. Publisher Copyright: © ICLR 2016: San Juan, Puerto Rico. All Rights Reserved.; 4th International Conference on Learning Representations, ICLR 2...

    2016

  18. [25]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  19. [26]

    High-dimensional continuous control using generalized advantage estimation, 2018

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URLhttps://arxiv.org/abs/1506.02438

    2018 arXiv

  20. [27]

    Rewarding progress: Scaling automated process verifiers for llm reasoning,

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning,

  21. [28]

    Spurious rewards: Rethinking training signals in rlvr

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr. https://reth...

    2025

  22. [29]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, 15 Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  23. [30]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the TwentiethEuropean ConferenceonComputerSystems, EuroSys ’25, page 1279–1297. ACM, March 2...

    2025

  24. [31]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  25. [32]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial intelligence, 112(1–2):181–211, 1999

    Richard S Sutton, Doina Precup, and Satinder P Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial intelligence, 112(1–2):181–211, 1999

    1999

  26. [34]

    Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    2022

  27. [35]

    A comparative study on reasoning patterns of openai’s o1 model.arXiv preprint arXiv:2410.13639, 2024

    Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. A comparative study on reasoning patterns of openai’s o1 model.arXiv preprint arXiv:2410.13639, 2024

    arXiv

  28. [36]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  29. [37]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, X...

    2025 arXiv

  30. [38]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URLhttps: //arxiv.org/abs/2503.18892

    2025 arXiv

  31. [39]

    Srpo: A cross-domain implementation of large-scale reinforcement learning on llm, 2025

    Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementation of large-scale...

    2025 arXiv

  32. [40]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. 16 Appendix A Data Se...

    2022 arXiv

  33. [2024]

    URL https://arxiv.org/abs/2410.08146

    arXiv

  34. [2025]

    URL https://arxiv.org/abs/2506.01939

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.