REVIEW 4 major objections 5 minor 13 cited by
Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Outcome-reward RL, built from best-of-N positive cloning, reshaped negative rewards, and token-level reweighting, reaches 94.0 pass@1 on MATH-500 at 7B scale and 95.0 at 32B scale.
desk verdict OREAL's empirical results are real, but the paper's central theory—positive-only BoN cloning gives the KL-regularized optimum—is wrong for binary rewards. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the best-of-N selection distribution over trajectories under 0/1 rewards, written in the paper as $\pi_{\mathrm{BoN}}(s) = \pi(s)[ R(s)(1-(1-p)^n)/p + (1-R(s))(1-p)^{n-1}]$, together with the claim that its KL divergence from $\pi$ is $\log n - (n-1)/n$. This identity is what lets the authors replace RL with behavior cloning on positive samples; the same distribution is then used to derive the negative-sample reward-shaping factor $1-p$ and the token-level weights $\omega_+, \omega_-$ that enter the total loss $L_{\mathrm{total}}$.
What would settle it
Take a small binary-reward environment, say two correct and two incorrect trajectory types with known probabilities, and compute the exact best-of-N selection distribution for n = 2, 4, 8, ...; if the KL divergence relative to the base policy saturates at a finite value instead of following $\log n - (n-1)/n$, the stated theoretical premise fails and the paper's own ablation data provide the empirical check of how much the additional loss terms matter.
Extended reading notes
Core claim
The central claim is that in an environment where every trajectory is labeled only correct or incorrect, the best-of-N policy is the right optimization target: as the sample size grows, the distribution of the selected trajectory concentrates on the correct-conditioned distribution $\pi(s)/p$, and the paper derives the BoN distribution and its KL divergence to argue that this target coincides with the solution of the KL-regularized policy objective. Consequently the training objective combines positive-example alignment $L_1$, a negative-sample loss $L_2$ whose rewards are reshaped by the factor $1-p$ to keep gradients consistent, and a token-level importance weighting from a reward model $w(s_{\le t})$ trained only on outcome labels. On this foundation the paper reports that the full OREAL pipeline reaches 91.0 pass@1 on MATH-500 with a 7B policy from its own supervised initialization, 94.0 when that initialization is the distilled 7B model, and 95.0 at 32B scale, which the authors present as the current performance limit of outcome-reward RL for math reasoning.
Load-bearing premise
The proof assumes that best-of-N selection over binary 0/1 reward trajectories obeys the continuous-reward formula whose KL divergence, $\log n - (n-1)/n$, grows without bound; with ties among correct trajectories the selected distribution instead converges to the correct-conditioned distribution with bounded KL, and that difference is what the claimed equivalence rests on.
Editorial extensions
If this is right
- Outcome-only RL can serve as a substitute for distillation: OREAL-32B improves on the 32B distilled baseline on MATH-500, LiveMathBench, and OlympiadBench, and OREAL-DSR1-Distill-Qwen-7B improves its own 7B initialization.
- Because positive cloning is claimed sufficient, the framework removes the need for a learned value network or process reward model in binary-feedback math settings.
- Reward shaping on negative trajectories should be based on the estimated failure probability, making the policy gradient consistent between correct and incorrect samples; the ablation attributes steady gains to this component.
- Token-level importance weighting, trained from outcome rewards, lets the model concentrate on decisive steps and errors in long chains of thought, which the ablation shows gives the largest single gain on MATH-500.
- The final performance depends strongly on the initial supervised policy and on the difficulty and scope of the training queries, not just on the RL algorithm.
Reading between the lines
- If the theoretical equivalence were exact, raising the sampling budget in BoN would be a complete training algorithm by itself; the paper's own ablation, in which reward shaping and token weighting add about five points over positive-only cloning, suggests the practical headroom is carried at least partly by the empirical components rather than by the theorem.
- The token-level reward model is a byproduct that could be used at inference time for early stopping or selective search, since its per-token scores localize where a trajectory turns correct or wrong; the paper visualizes these scores but does not test such uses.
- The reliance on the initial policy and on skill-targeted SFT before RL implies that outcome-only RL mostly amplifies knowledge already present in the base model; a controlled comparison on problems outside the training distribution, rather than MATH-500, would be the sharper test of true reasoning gains.
- The MATH-500-focused skill enhancement makes part of the RL gain a data-curation effect; separating the SFT augmentation from the RL loop would clarify how much of the reported jump is due to OREAL itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OREAL, an outcome-reward-based reinforcement learning framework for mathematical reasoning. The method combines three components: behavior cloning on positive trajectories obtained by best-of-N (BoN) sampling, a reward-shaping term for negative samples, and token-level importance weights derived from a learned token reward model. The authors claim a theoretical proof that positive-only behavior cloning is sufficient to recover the KL-regularized optimal policy in binary-reward environments, and they report strong empirical results, including 94.0 pass@1 on MATH-500 for a 7B model and 95.0 pass@1 for a 32B model.
Significance. The empirical contribution is potentially strong: OREAL-7B and OREAL-32B compare favorably with many competitive baselines across multiple math benchmarks, and the paper includes ablations and a concrete training recipe. However, the advertised theoretical foundation is load-bearing. The abstract and Section 2.2 claim a proof that positive-only behavior cloning is sufficient for KL-regularized optimality. That claim is not established: Section 2.2's KL computation uses the continuous-reward BoN formula in a binary-reward setting, where the relevant selected distribution is a conditional distribution with bounded KL. The negative-reward shaping in Section 2.3 further contradicts the positive-only sufficiency claim. The empirical recipe may still be useful as a heuristic, but the central theoretical guarantee and the framing of a 'limit of outcome reward' overstate what is demonstrated.
major comments (4)
- [Section 2.2, Lemma 2.1 and Eq. (4)] The central theoretical claim is not supported. Lemma 2.1 states that BoN selection under binary rewards yields the conditional distribution π(s)/p, which is independent of n. Eq. (4) then imports the continuous-reward formula πBoN(s)=n[P(s)]^{n-1}π(s) with KL = log n − (n−1)/n, and uses it to argue that increasing n can cover the entire positive real axis. Under the binary reward defined in Eq. (3), all positive trajectories are tied at reward 1, so the BoN-selected distribution is π(s|R=1) for every n (assuming at least one positive), and its KL divergence from π is −log p, which is bounded. This equals the extreme α→0 case of the KL-regularized optimum, not an arbitrary KL level. Consequently, the inference that positive-only behavior cloning can realize the full family of KL-regularized policies in Eq. (2) is invalid.
- [Section 2.3, Eq. (6)] The derivation of the negative-sample reward shaping contains an algebraic inconsistency. Starting from the definition of πbon(s) in Eq. (5), the positive-gradient term should carry the coefficient (1 − (1−p)^n)/p times E_{π,D+}[∇ log π(s)], not n(1−p)^{n−1} times that quantity. For example, with p=0.5 and n=4, these coefficients are 1.875 and 0.5, respectively. Similarly, the negative-gradient coefficient under Eq. (5) is (1−p)^n, not n(1−p)^n. In addition, the first line of Eq. (6) includes R(s) as a multiplier, but R(s)=0 on D−, so the subsequent 'similarly' computation that yields a nonzero negative term silently drops that factor. The reshaping factor R*(s)=(1−p)R(s) is therefore not derived correctly.
- [Sections 2.2 and 2.3, Algorithm 1] The framework is internally inconsistent. Section 2.2 claims that behavior cloning on positive BoN samples is sufficient to recover the KL-regularized optimal policy, but Section 2.3 argues that negative-sample rewards must be reshaped to maintain gradient consistency, and Algorithm 1 uses one positive and one negative trajectory per question. If positive samples alone suffice, the negative term is unnecessary; if negative shaping is necessary, then the sufficiency claim is false. The paper does not reconcile these positions, and the ablation in Table 2 only shows cumulative gains of adding components, not a comparison of positive-only BC against the claimed optimal policy.
- [Section 3.2, Eqs. (10)–(12) and Algorithm 1] The token-level reward model is initialized from the policy model, trained on binary rewards of the current policy's own rollouts, and then used to compute importance weights for the loss that updates the same policy. This creates a potential circularity: the reward model has no independent source of labels and is not validated on held-out data, so the importance weights can reflect the policy's current distribution rather than an unbiased estimate of token-level contribution. The paper should provide evidence that this reweighting improves generalization, for example by comparing against a token reward model trained on separate rollouts or against outcome-only training without token-level weights.
minor comments (5)
- [Abstract and Section 4.2] The abstract's claim that 'for the first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL' is misleading: the 94.0 result, OREAL-DSR1-Distill-Qwen-7B, starts from DeepSeek-R1-Distill-Qwen-7B, which is itself produced by distillation. The sentence should distinguish the RL-from-RFT result (OREAL-7B, 91.0) from the result obtained by applying OREAL to a distilled initialization.
- [Table 1] The caption states that baselines taken from prior reports are marked with '*', but several presumably external numbers in Table 1 are not marked, including QwQ-32B-Preview and o1-mini. Please clarify which results were reproduced in-house and which are quoted from the literature.
- [Throughout] There are several typos and notation issues: 'Tabel 1' and 'Tabel 2' should be 'Table 1' and 'Table 2'; 'BoNBoN [31]' is an odd double reference; 'ID+' and 'ID−' are used without explicit definition; and 'logn − n−1/n' in Eq. (4) should be written with clearer parentheses, e.g., log n − (n−1)/n.
- [Section 2.2] The phrase 'reward equivalence principle stated in Eq. 3' is misleading because Eq. (3) is a definition of binary outcome reward, not a principle. Consider rephrasing to 'the binary reward definition in Eq. (3)'.
- [Section 3.3] The skill-based enhancement procedure is described only qualitatively. Please specify how the model's repeated failures are determined, how many questions are augmented per skill, and whether the augmented questions are used in both RFT and RL training.
Circularity Check
No significant circularity: the empirical results are checked against external benchmarks and the policy losses are not fitted predictions; the main theoretical claim has a correctness gap but does not reduce to its own inputs.
full rationale
The claimed derivation chain is not circular in the sense of reducing a predicted quantity to a fitted input. L1 (positive behavior cloning), L2 (negative reward shaping), and Ltotal (token-level reweighting) are standard policy-gradient/BC objectives computed from on-policy rollouts and binary verifier rewards; no reported evaluation number is used to fit a parameter that is then reported as a prediction. The external benchmarks (MATH-500, AIME, OlympiadBench) provide an independent check. The self-citations that exist (LiveMathBench [47], Inverse-Q* [37], OpenDataLab [40], InternLM-Math [5]) are contextual, data-source, or related-work citations and are not used to justify the paper's main theoretical or empirical claims. The main theoretical claim in Section 2.2 is unsupported: Lemma 2.1 states that the binary-reward BoN selection distribution converges to pi(s)/p independent of n, while Eq. (4), with KL = log n - (n-1)/n, is the continuous-reward BoN formula; using Eq. (4) to conclude that arbitrary KL constraints are achievable contradicts Lemma 2.1. This is a mathematical correctness gap, not a circular reduction, so it does not raise the circularity score. Minor self-citations are not load-bearing, yielding a low circularity score.
Assumptions & free parameters
free parameters (3)
- query pass-rate filter threshold =
0.8
- KL coefficient beta =
0.01
- positive/negative loss balance eta =
not reported
assumptions (4)
- standard math Closed-form solution to KL-regularized policy optimization (Eq 2) is valid for the outcome-reward MDP.
- domain assumption Binary outcome reward treats all correct trajectories as equally valid and the verifier supplies correct binary labels.
- ad hoc to paper The Best-of-N distribution under binary rewards follows the continuous-reward formula pi_BoN(s) = n [P(s)]^(n-1) pi(s) with KL = log n - (n-1)/n.
- domain assumption A token-level reward model trained on binary outcome labels provides a faithful per-token credit assignment for reweighting the policy loss.
Cite this review
Pith. "Pith review of Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning." pith.science (2026). https://pith.science/paper/E47KYOG7
@misc{pith2026250206781,
author = {Pith},
title = {Pith review of: Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/E47KYOG7}},
note = {Machine review of arXiv:2502.06781}
}
read the original abstract
Reasoning abilities, especially those for solving complex math problems, are crucial components of general intelligence. Recent advances by proprietary companies, such as o-series models of OpenAI, have made remarkable progress on reasoning tasks. However, the complete technical details remain unrevealed, and the techniques that are believed certainly to be adopted are only reinforcement learning (RL) and the long chain of thoughts. This paper proposes a new RL framework, termed OREAL, to pursue the performance limit that can be achieved through \textbf{O}utcome \textbf{RE}w\textbf{A}rd-based reinforcement \textbf{L}earning for mathematical reasoning tasks, where only binary outcome rewards are easily accessible. We theoretically prove that behavior cloning on positive trajectories from best-of-N (BoN) sampling is sufficient to learn the KL-regularized optimal policy in binary feedback environments. This formulation further implies that the rewards of negative samples should be reshaped to ensure the gradient consistency between positive and negative samples. To alleviate the long-existing difficulties brought by sparse rewards in RL, which are even exacerbated by the partial correctness of the long chain of thought for reasoning tasks, we further apply a token-level reward model to sample important tokens in reasoning trajectories for learning. With OREAL, for the first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL, being on par with 32B models. OREAL-32B also surpasses previous 32B models trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our investigation also indicates the importance of initial policy models and training queries for RL. Code, models, and data will be released to benefit future research\footnote{https://github.com/InternLM/OREAL}.
Figures
Forward citations
Cited by 13 Pith papers
-
RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback
RFTF trains a value model on temporal state orderings to supply dense rewards for reinforcement fine-tuning of vision-language-action models, achieving an average success length of 4.296 on CALVIN ABC-D with the Seer-...
-
Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving
A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.
-
Test-Time Scaling with Reflective Generative Model
MetaStone-S1 combines a shared policy and self-supervised process reward head to select high-quality reasoning traces, reaching o3-mini-level scores at 32B parameters.
-
Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
A 7B LLM trained with sparse completion rewards and a GRPO-style algorithm reaches state-of-the-art on ALFWorld and ScienceWorld.
-
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
CoVo trains LLMs with a self-generated reward based on the consistency and volatility of intermediate reasoning states, matching supervised RL performance without external labels.
-
Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning
A 3B model trained with a small SFT warm-up followed by verifiable-reward RL matches or exceeds far larger models on EHR-based medical calculation, trial matching, and diagnosis tasks.
-
Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLM
Entropy Area Score sums token-level predictive entropy across a reasoning sequence, correlates with answer entropy, and selects SFT training data better than Pass Rate filtering in limited AIME experiments.
-
Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
Using Pass@k as an RLVR reward, with bootstrap sampling and an analytical advantage formula, improves exploration and later Pass@1 performance of reasoning LLMs.
-
VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL
VARD fine-tunes diffusion models by backpropagating through a learned value function that assigns dense, differentiable reward estimates to every intermediate denoising step, with KL regularization keeping the model n...
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
A Technical Survey of Reinforcement Learning Techniques for Large Language Models
A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.
-
Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Reference graph
Works this paper leans on
-
[1]
Towards large reasoning models: A survey of reinforced reasoning with large language models
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686, 2025
arXiv 2025
-
[2]
Evaluation of openai o1: Opportunities and challenges of agi
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486, 2024
arXiv 2024
-
[3]
Mathematical language models: A survey
Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, et al. Mathematical language models: A survey. arXiv preprint arXiv:2312.07622, 2023
arXiv 2023
-
[4]
Nikolaos Matzakos, Spyridon Doukakis, and Maria Moundridou. Learning mathematics with large language models: A comparative study with computer algebra systems and other tools. International Journal of Emerging Technologies in Learning (iJET), 18(20):51–71, 2023
work page 2023
-
[5]
Internlm-math: Open math large language models toward verifiable reasoning
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024
arXiv 2024
-
[6]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
arXiv 2024
-
[7]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[8]
Self-consistency improves chain of thought reasoning in language models
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022
arXiv 2022
Show all 72 references
-
[9]
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022. 12
2022
-
[10]
Learning to reason with llms
OpenAI. Learning to reason with llms. 2024
2024
-
[11]
O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024
2024 arXiv
-
[12]
Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning system...
2024 arXiv
-
[13]
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...
2025
-
[14]
Deepseek-v3 technical report
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024
2024 arXiv
-
[15]
Let’s verify step by step
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023
2023 arXiv
-
[16]
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025
2025 arXiv
-
[17]
Math-shepherd: Verify and reinforce llms step-by-step without human annotations
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...
2024
-
[18]
Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024
-
[19]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[20]
Process reinforcement through implicit rewards
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025
2025 arXiv
-
[21]
Measuring mathematical problem solving with the math dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021
2021 arXiv
-
[22]
Qwq: Reflect deeply on the boundaries of the unknown, November 2024
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024
2024
-
[23]
Scaling relationship on learning mathematical reasoning with large language models
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023
2023 arXiv
-
[24]
Scaling laws for reward model overoptimization
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023
2023
-
[25]
Compositional preference models for aligning lms
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. Compositional preference models for aligning lms. arXiv preprint arXiv:2310.13011, 2023
2023 arXiv
-
[26]
T\" ulu 3: Pushing frontiers in open language model post-training
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024
2024 arXiv
-
[27]
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...
2022
-
[28]
Measuring goodhart’s law
Jacob Hilton and Leo Gao. Measuring goodhart’s law. OpenAI Research Blog, 2022
2022
-
[29]
Training language models with language feedback at scale
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303.16755, 2023
2023 arXiv
-
[30]
Reward model ensembles help mitigate overoptimization
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023
2023 arXiv
-
[31]
Bonbon alignment for large language models and the sweetness of best-of-n sampling
Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.00832, 2024
2024 arXiv
-
[32]
Bond: Aligning llms with best-of-n distillation
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.14622, 2024
2024 arXiv
-
[33]
An invariant form for the prior probability in estimation problems.Proceedings of the Royal Society of London
Harold Jeffreys. An invariant form for the prior probability in estimation problems.Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453– 461, 1946
1946
-
[34]
Inference-aware fine-tuning for best-of-n sampling in large language models
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. arXiv preprint arXiv:2412.15287, 2024
2024
-
[35]
Keinosuke Fukunaga and Donald M. Hummels. Leave-one-out procedures for nonparametric error estimates. IEEE transactions on pattern analysis and machine intelligence, 11(4):421–423, 1989. 14
1989
-
[36]
From r to Q*: Your language model is secretly a q-function
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q*: Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024
2024 arXiv
-
[37]
Inverse-q*: Token level reinforcement learning for aligning large language models without preference data
Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, and Xuanjing Huang. Inverse-q*: Token level reinforcement learning for aligning large language models without preference data. arXiv preprint arXiv:2408.14874, 2024
2024 arXiv
-
[38]
Training verifiers to solve math word problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021
-
[39]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024
2024 arXiv
-
[40]
Opendatalab: Empowering general artificial intelligence with open datasets, 2024
Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. Opendatalab: Empowering general artificial intelligence with open datasets, 2024
2024
-
[41]
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repositor...
2024
-
[42]
Hello GPT-4o, 2024
OpenAI. Hello GPT-4o, 2024
2024
-
[43]
Claude 3.5 sonnet, 2024
Anthropic. Claude 3.5 sonnet, 2024
2024
-
[44]
7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason , 2025. Notion Blog
2025
-
[45]
rstar-math: Small llms can master math reasoning with self-evolved deep thinking
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 2025
2025 arXiv
-
[46]
American invitational mathematics examination - aime
MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME
-
[47]
Are your llms capable of stable reasoning? arXiv preprint arXiv:2412.13147, 2024
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your llms capable of stable reasoning? arXiv preprint arXiv:2412.13147, 2024
2024 arXiv
-
[48]
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....
2024 arXiv
-
[49]
Opencompass: A universal evaluation platform for foundation models
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023
2023
-
[50]
Policy gradient meth- ods for reinforcement learning with function approximation
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999
1999
-
[51]
Tree of thoughts: Deliberate problem solving with large language models
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024
2024
-
[52]
Graph of thoughts: Solving elaborate problems with large language models
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Confe...
2024
-
[53]
Metamath: Bootstrap your own mathematical questions for large language models
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023
2023 arXiv
-
[54]
Augmenting math word problems via iterative question composing
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. arXiv preprint arXiv:2401.09003, 2024
2024 arXiv
-
[55]
Query and response augmentation cannot help out-of-domain math reasoning generalization
Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Query and response augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint arXiv:2310.05506, 2023
-
[56]
Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks
Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201, 2023
2023 arXiv
-
[57]
Mammoth: Building math generalist models through hybrid instruction tuning
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023
2023 arXiv
-
[58]
Sft memorizes, rl generalizes: A comparative study of foundation model post-training
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025
2025 arXiv
-
[59]
Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15–18, 2024
Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15–18, 2024
2024
-
[60]
Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023
2023 arXiv
-
[61]
Beyond human data: Scaling self-training for problem-solving with language models
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023
2023 arXiv
-
[62]
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. 16 A Token Lev...
2024 arXiv
-
[63]
You don’t need to re-generate the answer to the question because the standard answer has been given
Please refer to the given standard answer. You don’t need to re-generate the answer to the question because the standard answer has been given. You only need to judge whether the candidate’s answer is consistent with the standard answer according to the form of the question. D...
-
[64]
Because the candidate’s answer may be different from the standard answer in the form of expression, before making a judgment, please understand the question and the standard answer first, and then judge whether the candidate’s answer is correct, but be careful not to try to an...
-
[65]
As long as the answer is the same as the standard answer, it is enough
Some answers may contain multiple items, such as multiple-choice questions, multiple-select questions, fill-in-the-blank questions, etc. As long as the answer is the same as the standard answer, it is enough. For multiple-select questions and multiple-blank fill-in-the-blank q...
-
[66]
And some formulas are expressed in different ways, but they are equivalent and correct
Some answers may be expressed in different ways, such as some answers may be a mathematical expression, some answers may be a textual description, as long as the meaning expressed is the same. And some formulas are expressed in different ways, but they are equivalent and correct
-
[67]
A" or "B
If the prediction is given with \boxed{}, please ignore the \boxed{} and only judge whether the candidate’s answer is consistent with the standard answer. Please judge whether the following answers are consistent with the standard answer based on the above criteria. Grade the ...
-
[68]
Take time to think thoroughly rather than rushing to an answer
-
[69]
Rigorously prove each key conclusion
-
[70]
Keep an open mind and try different approaches
-
[71]
Summarize valuable problem-solving methods
-
[72]
Maintain healthy skepticism and verify multiple times Your response should reflect deep mathematical understanding and precise logical thinking, making your solution path and reasoning clear to others. When you’re ready, present your complete solution with: - Clear problem und...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.