Pith. sign in

REVIEW 2 major objections 3 minor 5 cited by

Reinforcing General Reasoning without Verifiers

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VeriFree replaces the external answer verifier with the model's own probability of producing the reference answer, claiming equivalence to verifier-based RL, lower gradient variance, and match-or-better accuracy on general reasoning…

desk verdict A clean verifier-free RL objective with real practical payoff, but the theory only covers the single-answer case and the empirical comparison lacks a fully controlled verifier baseline. read the letter →

arxiv 2505.21493 v1 pith:4H7LDMYF submitted 2025-05-27 cs.LG cs.CL

Xiangxin Zhou , Zichen Liu , Anya Sims , Haonan Wang , Tianyu Pang , Chongxuan Li , Liang Wang , Min Lin
show 1 more author
Chao Du
This is my paper · ORCID
classification cs.LGcs.CL
keywords verifier-freereinforcementlearninglargelanguagemodelsgeneralreasoningchain-of-thoughtRao-BlackwellizationvariancereductionGRPOreference-answerlikelihood
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

R1-Zero-style reinforcement learning has boosted LLM reasoning, but only where a rule-based check can decide whether an answer is right. This paper tries to lift that restriction for general domains — chemistry, law, health, business — by removing the verifier altogether. The proposed method, VeriFree, uses the probability the model itself assigns to the dataset's reference answer, given the question and its generated reasoning trace, as the training reward. The paper claims that when a unique correct answer exists this objective equals the verifier-based one in expectation, that its gradient estimator has provably lower variance, and that empirically it matches or surpasses verifier-based RL on MMLU-Pro, SuperGPQA, GPQA, and math benchmarks while being simpler, cheaper, and less memory-hungry. If right, it would let reasoning RL be applied to any domain that has questions with reference answers, with no verification machinery at all.

What carries the argument

The load-bearing identity is the marginalization step: for a unique correct answer, $\mathbb{E}_{y \sim \pi_\theta(\cdot|x,z)}[\mathbf{1}\{y = y^\star\}] = \pi_\theta(y^\star \mid x, z)$, so the binary verifier reward can be replaced by a number read off a single forward pass of the policy itself. That number plays two roles in the gradient: it rewards the reasoning trace through a policy-gradient term, and it weights a supervised term that teaches the model to emit the reference answer after good traces. Because the answer variable is summed out analytically, the estimator is Rao-Blackwellized relative to verifier-based sampling, which is the source of the variance reduction in Theorem 1. Supporting machinery addresses practical hurdles: the reasoning trace is cut at the '<answer' token rather than '<answer>' so that patching in the reference answer keeps tokenization consistent between sampling and optimization, and the final on-policy estimator combines an RLOO baseline with grouped sampling (Eq. 7).

What would settle it

Build two training sets from the same general-reasoning questions, identical except that one uses common phrasings as the reference answer and the other uses rare, low-probability phrasings; if the rare-reference model's benchmark accuracy falls well below the common-reference model's while a verifier-based model trained on both sets stays flat, then a single arbitrary reference does not provide a sufficient learning signal and the equivalence claim fails to transfer. A cheaper check: on held-out questions, measure how often a reasoning trace with high $\pi_\theta(y^\star \mid x, z)$ is actually semantically correct, and locate where the proxy reward diverges from the verifier's reward.

Watch

Extended reading notes

Core claim

Under the assumption that each question has a single correct answer string, the expected verifier reward for a sampled reasoning trace — the indicator that the model's final answer exactly matches the reference — equals the probability $\pi_\theta(y^\star \mid x, z)$ that the model itself assigns to that reference answer. VeriFree's objective replaces the verifier with this self-probability: $J_{\mathrm{VeriFree}}(\theta; x, y^\star) = \mathbb{E}_{z \sim \pi_\theta(\cdot|x)}[\pi_\theta(y^\star \mid x, z)]$, which is shown to equal the verifier-based objective $J_{\mathrm{Verifier}}$ in expectation (Eq. 4). The resulting gradient estimator marginalizes out the answer $y$ analytically, making it a Rao-Blackwellization of the verifier-based estimator and therefore no higher in variance (Theorem 1). The paper reports that on roughly 61,000 general-reasoning questions, RL with VeriFree improves Qwen3 base models on MMLU-Pro, SuperGPQA, GPQA, and math benchmarks, matching or exceeding both a model-verifier baseline and Qwen3's own instruct models; it also finds the model's self-confidence $\pi_\theta(y^\star \mid x, z)$ tracks evaluation accuracy during training ($\rho = 0.82$).

Load-bearing premise

The formal equivalence to verifier-based RL is proven only when each question has exactly one valid answer string; for the multiple-valid-answer questions that dominate general reasoning, the paper's own defense is empirical, so the method's success in that regime rests on an unproven premise.

Editorial extensions

If this is right

  • R1-Zero-style RL becomes applicable to any reasoning domain that supplies reference answers — chemistry, law, medicine, business — with no rule-based checker and no separate verifier LLM.
  • Training is cheaper and simpler in practice: only one model is kept in memory, no verifier is maintained, and no KL reference model is stored, since the method drops KL penalties.
  • Learning is more sample-efficient: VeriFree reaches higher accuracy with fewer policy-gradient steps than the verifier-based baseline, which the paper attributes to reduced gradient variance from continuous rather than binary rewards.
  • The model's self-confidence $\pi_\theta(y^\star \mid x, z)$ is a usable training-time proxy for emerging reasoning ability, correlating with evaluation accuracy ($\rho = 0.82$).
  • Reasoning learned on non-math data transfers: a VeriFree model trained with math examples removed still improves on math benchmarks, suggesting general reasoning skills rather than per-domain memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The formal equivalence relies on exact-match uniqueness; in multi-answer settings the paper only argues empirically. A natural extension, which the authors flag as future work, would replace the single reference probability with a sum over an equivalence class of valid answers, making the objective track the verifier's semantic-equivalence reward more faithfully.
  • Because the reward is the model's own likelihood, the failure mode shifts from verifier reward hacking to confidence miscalibration: if a base model assigns systematically low probability to correct but rare phrasings, VeriFree would under-reward exactly the traces that are correct — a prediction a dataset with rare-phrasing references could test.
  • The marginalization trick suggests a more general recipe: whenever the verifier reward is a deterministic function of the sampled answer and the policy's conditional answer distribution is tractable, the verifier can be integrated out analytically. VeriFree is the instance where the policy itself supplies that distribution; the paper's comparison with JLB and LaTRO shows neighbouring variational m
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes VeriFree, a reinforcement-learning objective for LLM reasoning that replaces rule-based or model-based verifiers with the model's own probability of generating the reference answer, πθ(y*|x,z), given the question and a generated reasoning trace. The authors show analytically that under a unique correct answer string, this objective equals the verifier-based RLVR objective in expectation (Eq. 4) and that its single-sample gradient estimator has lower variance via Rao-Blackwellization (Theorem 1). They train Qwen3 base models of sizes 1.7B, 4B, and 8B on a filtered WebInstruct-derived dataset and report comparisons against a verifier-based baseline and existing instruct models on MMLU-Pro, GPQA, SuperGPQA, and math benchmarks, claiming that VeriFree matches or surpasses verifier-based methods while being simpler and more compute-efficient. The paper also provides ablations on tokenization-aware trace splitting, RLOO, and equivalence-class handling, plus a discussion of connections to prior variational methods JLB and LaTRO.

Significance. If the central claims hold, the paper makes a practically valuable contribution: it removes the need to maintain and query a separate verifier LLM during R1-Zero-style RL, reduces memory and compute overhead, and offers a clean derivation rooted in Rao-Blackwellization. The mathematical derivation in Section 2.2 and the variance-reduction proof in Appendix A.2 are correct under the stated exact-match assumption, and the empirical evaluation spans multiple model scales and diverse benchmarks, with code released. The paper is also careful to position itself relative to the related JLB and LaTRO estimators. However, the significance is currently gated by a gap between the theory and the deployment regime: the equivalence proof assumes a unique correct answer string, while the paper's motivation is general reasoning with semantic equivalence and multiple valid answers. The empirical evidence for the multiple-answer regime is indirect and largely confined to multiple-choice evaluations. These issues are fixable but need to be addressed before the paper's headline claims are fully supported.

major comments (2)
  1. [Section 2.2, Eq. (4), with Section 2.1 footnote] The derivation of J_VeriFree = J_Verifier is valid only for the exact-match reward 1{y=y*}, but the paper itself defines verifier rewards in Section 2.1 as semantic equivalence 1{y≡y*}. In general reasoning, where multiple phrasings of a correct answer exist, the verifier objective is E_z[Σ_{y∈C} πθ(y|x,z)] over the equivalence class C, whereas VeriFree optimizes E_z[πθ(y*|x,z)] for a single reference answer. These objectives are not equivalent: the latter can be increased by shifting probability mass toward the reference phrasing without increasing the total probability of any correct answer. Consequently, the claimed equivalence, the variance-reduction theorem, and the interpretation of VeriFree as optimizing the same objective as RLVR do not transfer to the free-form general-reasoning setting the paper targets. The Introduction's caveat that 'using just one as a reference provides a sufficient learning signal' is only supported empirically, and the equivalence-class ablation in Fig. 6 Right is performed on MATH-12k with equivalence classes collected from a model already fine-tuned with a rule-based verifier, reporting only a 'slight' improvement. Since the main evaluations use multiple-choice questions where exact match coincides with correctness, the paper does not currently establish the headline general-reasoning claim for free-form answers.
  2. [Section 3.1, Baselines and Eq. (7)] The verifier baseline is trained with Dr. GRPO [24], while the VeriFree estimator in Eq. (7) is a GRPO/RLOO estimator with the 'corrected response-length normalization' from Liu et al. [24]. The text does not state whether VeriFree also uses Dr. GRPO or a different optimizer. If the optimization algorithms differ between the two arms, the reported comparison does not isolate the effect of replacing the verifier with the model's own probability; part of the observed difference could be attributable to the optimizer rather than to the reward signal. Please specify the exact optimizer used for VeriFree, and ideally provide a matched-optimizer ablation to support the claim that VeriFree 'matches and even surpasses' verifier-based methods.
minor comments (3)
  1. [Section 2.2, Theorem 1] The variance inequality in Eq. (6) is presented without specifying that the comparison is for covariance matrices when the gradients are vectors; the proof in Appendix A.2 uses the law of total variance, which holds for covariance matrices, so the statement should be made precise.
  2. [Table 3 and Abstract] The claim that VeriFree 'matches and even surpasses' verifier-based methods is supported on average but not uniformly: for example, on Minerva Math the Qwen3-8B-Base-VeriFree model scores 30.5 versus 39.0 for the Verifier baseline, and similar per-domain gaps appear in Tables 1 and 2. Please qualify the headline claim with the observed per-domain variability and consider reporting variance or significance across runs.
  3. [Section 2.4] The sentence explaining that the pattern ' r>' does not appear in standard tokenizer vocabularies is difficult to parse; please clarify the token-level justification and specify whether this claim applies to all tokenizers used in the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core derivation is an algebraic identity and the variance reduction is standard Rao-Blackwellization; empirical claims are tested against external benchmarks.

full rationale

The paper's central derivation (Eq. 4) replaces E_y[1{y=y*}] with πθ(y*|x,z) by explicitly summing over y, which is a straightforward marginalization under the stated exact-match assumption; the gradient estimator in Eq. (5) and Theorem 1 are derived directly from this identity via the score function and the law of total variance (Rao-Blackwellization), with no fitted constants or benchmark-dependent parameters. The method is not validated by in-sample agreement with its own training signal: the main results compare VeriFree against an external model-based verifier baseline and external benchmarks (MMLU-Pro, SuperGPQA, GPQA, and math suites), and the ablations test design choices against held-out evaluation accuracy. Self-citations to the Oat framework, Dr. GRPO, and length normalization are implementation references, not load-bearing premises of the derivation. The acknowledged gap—that the exact-match assumption in Eq. (4) does not formally cover multiple valid answer phrasings, with only empirical evidence offered for that regime—is a limitation in scope and rigor, not a circular step, because the multiple-answer objective is not defined in terms of the method's own outputs. The correlation analysis in Fig. 4 (right) reports an empirical relationship between training reward and accuracy; it is not a prediction derived from the training signal itself. No circular step matching the enumerated patterns could be quoted and exhibited.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. It relies on the model's own probability as a reward, which is a design choice, and on the unique-answer assumption for theoretical equivalence. Free parameters are standard RL hyperparameters and a data filtering threshold.

free parameters (3)
  • Answer token length threshold = <7 tokens
    Used to filter WebData to short-answer questions; chosen by the authors, not derived.
  • Group size = 8
    Number of responses sampled per prompt for the RLOO baseline; chosen standard value.
  • Training steps = 4000 (1.7B/4B), 3000 (8B)
    Stopping criterion chosen based on compute budget; affects results but not the derivation.
assumptions (3)
  • domain assumption A unique correct answer string exists for each question, so exact-match reward is appropriate.
    Used in Eq. (4) to equate J_Verifier and J_VeriFree; admitted by the authors to not hold generally.
  • domain assumption The tokenizer does not produce the token sequence ' r>' across the split boundary at '<answer'.
    Section 2.4 relies on this to ensure tokenization-consistent extraction of reasoning traces.
  • ad hoc to paper The model's conditional probability πθ(y*|x,z) is a meaningful reward signal for reasoning quality.
    This is the core hypothesis; without it the method is just supervised learning on the reference answer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcing General Reasoning without Verifiers." pith.science (2026). https://pith.science/paper/4H7LDMYF

@misc{pith2026250521493,
  author       = {Pith},
  title        = {Pith review of: Reinforcing General Reasoning without Verifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4H7LDMYF}},
  note         = {Machine review of arXiv:2505.21493}
}
read the original abstract

The recent paradigm shift towards training large language models (LLMs) using DeepSeek-R1-Zero-style reinforcement learning (RL) on verifiable rewards has led to impressive advancements in code and mathematical reasoning. However, this methodology is limited to tasks where rule-based answer verification is possible and does not naturally extend to real-world domains such as chemistry, healthcare, engineering, law, biology, business, and economics. Current practical workarounds use an additional LLM as a model-based verifier; however, this introduces issues such as reliance on a strong verifier LLM, susceptibility to reward hacking, and the practical burden of maintaining the verifier model in memory during training. To address this and extend DeepSeek-R1-Zero-style training to general reasoning domains, we propose a verifier-free method (VeriFree) that bypasses answer verification and instead uses RL to directly maximize the probability of generating the reference answer. We compare VeriFree with verifier-based methods and demonstrate that, in addition to its significant practical benefits and reduced compute requirements, VeriFree matches and even surpasses verifier-based methods on extensive evaluations across MMLU-Pro, GPQA, SuperGPQA, and math-related benchmarks. Moreover, we provide insights into this method from multiple perspectives: as an elegant integration of training both the policy and implicit verifier in a unified model, and as a variational optimization approach. Code is available at https://github.com/sail-sg/VeriFree.

Figures

Figures reproduced from arXiv: 2505.21493 by the authors.

Figure 1
Figure 1. The general reasoning capability is significantly improved when we apply VeriFree to fine-tune Qwen3 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. VeriFree enables R1-Zero-style LLM training without requiring access to a verifier. In the case of a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A pseudocode-like comparison of VeriFree (ours) and the standard R1-Zero approach. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left: MMLU-Pro accuracy of VeriFree and the baseline fine-tuned from Qwen3-8B base model along training steps. The curve is smoothed by a moving average with an interval of 384. Right: The dynamics of MMLU-Pro evaluation accuracy and average model confidence πθ(y ⋆ |x,…
Figure 5
Figure 5. Figure 5: Reasoning transfer to math without math training. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Left: MMLU-Pro Evaluation Accuracy of VeriFree against ablation variants (w/o RLOO, w/o tokenization-aware split strategy) along training steps. Note that all these models are based on Qwen3-1.7B base models. Right: Effects of introducing the equivalent class to VeriFr…
Figure 7
Figure 7. Figure 7: Category distributions in WebData. A breakdown of the “grouped” category (right) shows detailed [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Predictive Divergence Masks for LLM RL

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using the first-order change of KL divergence, not the sampled importance ratio, as the mask direction criterion improves LLM RL stability and performance over DPPO.

  2. Reverse-Engineered Reasoning for Open-Ended Generation

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Given a high-quality output, the authors search for a thinking trace that minimizes that output's perplexity, then fine-tune Qwen3-8B on 20,000 such traces, reporting writing performance near GPT-4o and Claude 3.5.

  3. URPO: A Unified Reward & Policy Optimization Framework for Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A seven-billion-parameter model trained with one unified GRPO loop improves instruction following, reasoning, and reward modeling at the same time.

  4. Reinforcement Pre-Training

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Reinforcement pre-training (RPT) trains an LLM with a verifiable reward for correct next-token prediction after generating a chain of thought, improving token accuracy and downstream reasoning over its base model.

  5. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

60 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [24]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    arXiv 2025

  2. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740, 2024

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  3. [2]

    George Casella and Christian P. Robert. Rao-blackwellisation of sampling schemes.Biometrika, 83(1):81–94, 1996

    work page 1996

  4. [3]

    Bootstrapping language models with dpo implicit rewards

    Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  5. [4]

    Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding.arXiv preprint arXiv:2411.04282, 2024

    Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding.arXiv preprint arXiv:2411.04282, 2024

    arXiv 2024

  6. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [6]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    arXiv 2025

  8. [7]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

Show all 60 references
  1. [8]

    Rlef: Grounding code llms in execution feedback with reinforcement learning.arXiv preprint arXiv:2410.02089, 2024

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning.arXiv preprint arXiv:2410.02089, 2024

    2024 arXiv

  2. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [11]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  5. [12]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  6. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  7. [14]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URLhttps://arxiv.org/abs/2503.24290. 10

    2025 arXiv

  8. [15]

    Self-improvement in language models: The sharpening mechanism.arXiv preprint arXiv:2412.01951, 2024

    Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism.arXiv preprint arXiv:2412.01951, 2024

    2024 arXiv

  9. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  10. [17]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  11. [18]

    Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:3843–3857, 2022

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:...

    2022

  12. [19]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repositor...

    2024

  13. [20]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/fo...

    2024

  14. [21]

    X-reasoner: Towards generalizable reasoning across modalities and domains, 2025

    Qianchu Liu, Sheng Zhang, Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul V ozila, Tristan Naumann, and Hoifung Poon. X-reasoner: Towards generalizable reasoning across modalities and domains, 2025. URL https://arxiv.org/ abs/2505.03981

    2025 arXiv

  15. [22]

    Oat: A research-friendly framework for llm online alignment.https://github.com/sail-sg/oat, 2024

    Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research-friendly framework for llm online alignment.https://github.com/sail-sg/oat, 2024

    2024

  16. [23]

    There may not be aha moment in r1-zero-like training — a pilot study

    Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in r1-zero-like training — a pilot study. https://oatllm.notion.site/ oat-zero, 2025. Notion Blog

    2025

  17. [25]

    Deepcoder: A fully open-source 14b coder at o3-mini level, 2025

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025. Notion Blog

    2025

  18. [26]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

    2025

  19. [27]

    General- reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652, 2025

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General- reasoner: Advancing llm reasoning across all domains.arXiv preprint arXiv:2505.14652, 2025

    2025 arXiv

  20. [28]

    Ng, Daishi Harada, and Stuart J

    A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. InInternational Conference on Machine Learning,

  21. [29]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/. 11

    2024

  22. [30]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  23. [31]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  24. [32]

    Training chain-of- thought via latent-variable inference.Advances in Neural Information Processing Systems, 36: 72819–72841, 2023

    Du Phan, Matthew Douglas Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of- thought via latent-variable inference.Advances in Neural Information Processing Systems, 36: 72819–72841, 2023

    2023

  25. [33]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  26. [34]

    Learning to drive a bicycle using reinforcement learning and shaping

    Jette Randlov and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. pages 463–471, 01 1998

    1998

  27. [35]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  28. [36]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  29. [37]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  30. [38]

    Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    2025 arXiv

  31. [39]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  32. [40]

    Learning to chain-of-thought with jensen’s evidence lower bound.arXiv preprint arXiv:2503.19618, 2025

    Yunhao Tang, Sid Wang, and Rémi Munos. Learning to chain-of-thought with jensen’s evidence lower bound.arXiv preprint arXiv:2503.19618, 2025

    2025 arXiv

  33. [41]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025. URLhttps://qwenlm.github.io/blog/qwen3/

    2025

  34. [42]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  35. [43]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  36. [44]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  37. [45]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025. 12

    2025 arXiv

  38. [46]

    Self-rewarding language models.International Conference on Machine Learning,

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.International Conference on Machine Learning,

  39. [47]

    Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions.arXiv preprint arXiv:2502.13124, 2025

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions.arXiv preprint arXiv:2502.13124, 2025

    2025

  40. [48]

    Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  41. [49]

    Mammoth2: Scaling instructions from the web.Advances in Neural Information Processing Systems, 37:90629–90660, 2024

    Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web.Advances in Neural Information Processing Systems, 37:90629–90660, 2024

    2024

  42. [50]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.https://hkust-nlp.notion.site/simplerl-reason, 2025

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.https://hkust-nlp.notion.site/simplerl-reason, 2025. Notion Blog

    2025

  43. [51]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  44. [52]

    standard candles

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084, 2025. 13 Appendix Table of Contents A Theoretical Analysis 14 A.1 Derivation of Gradie...

    2025 arXiv

  45. [55]

    **Observe the Period**: First, astronomers measure the period of the star’s brightness variation, which is the time it takes for the star to go from brightest to dimmest and back to brightest

  46. [56]

    This relation is usually given in a form where a graph or a formula relates period to absolute magnitude (a measure of intrinsic brightness)

    **Determine Intrinsic Luminosity**: Using the period-luminosity relation, they can then determine the star’s intrinsic luminosity. This relation is usually given in a form where a graph or a formula relates period to absolute magnitude (a measure of intrinsic brightness)

  47. [57]

    **Compare to Apparent Luminosity**: The apparent magnitude (how bright the star appears from Earth) can be measured directly

  48. [58]

    ""everyone else is doing it

    **Calculate Distance**: Using the inverse square law for light, which relates the intrinsic luminosity, apparent brightness, and distance, astronomers can then calculate the distance to the Cepheid variable. This method is crucial for measuring distances in astronomy because i...

  49. [59]

    Their reasoning is fear-based, and they view rules as set by authority figures

    **Stage 1: Obedience and Punishment Orientation** In this stage, children obey rules to avoid punishment. Their reasoning is fear-based, and they view rules as set by authority figures. The focus is on the consequences of following or breaking rules, speci fically the avoidanc...

  50. [60]

    what’s in it for me?

    **Stage 2: Individualism and Exchange** In this stage, which is the second stage of the preconventional level, children recognize that different people have different needs and interests. They obey rules only when it is in their own interest or when they can benefit from doing...

  51. [1999]

    URLhttps://api.semanticscholar.org/CorpusID:5730166

  52. [2024]

    doi: 10.48550/arXiv.2401.10020

    doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.