Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a language model can reinforce its own reasoning with no ground-truth labels or external reward models, using a reward built from how consistently its intermediate steps converge on the final answer, and reports…

desk verdict CoVo's consistency/volatility observation is real on the base model, but the paper never shows the signal survives training, so the central claim remains conditional. read the letter →

arxiv 2506.08745 v1 pith:6XJTB6SV submitted 2025-06-10 cs.AI cs.CL

Kongcheng Zhang , Qi Yao , Shunyu Liu , Yingjie Wang , Baisheng Lai , Jieping Ye , Mingli Song , Dacheng Tao This is my paper
classification cs.AIcs.CL
keywords self-rewardingreinforcementlearningLLMreasoningintrinsicrewardconsistencyandvolatilitycuriositybonustrajectoryanalysislabel-freeRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the reasoning ability of a language model can be improved by its own internal signals alone, with no answer labels, no verifiers, and no trained reward models. Its central observation is that when a model samples many solutions to the same prompt, correct solutions differ measurably from wrong ones in trajectory shape: the intermediate steps of correct answers are consistently closest to their own final answer and stop wandering toward other candidates early, while wrong answers fluctuate and converge late. From this the paper builds CoVo, a reward that scores each sampled trajectory by that consistency and volatility, aggregates the scores robustly across solutions sharing the same answer, adds a curiosity bonus to keep exploration alive, and optimizes with an off-the-shelf policy-gradient algorithm. On math, commonsense, and science benchmarks across three models, CoVo matches or beats supervised RL that uses ground-truth answers, which matters because it points to a scalable path for improving reasoning from unlabeled prompts.

What carries the argument

The load-bearing object is the likelihood distance matrix $D\in\mathbb{R}^{T\times K}$ between the $T$ intermediate reasoning states of one sampled trajectory and the $K$ distinct final answers produced by all sampled trajectories for the same prompt, where each entry is $d(s_i,y_k) = -\frac{1}{|y_k|}\sum_j \log\pi_\theta(y_k[j]\mid s_i, y_k[:j])$ — the per-token negative log-likelihood the policy assigns to candidate answer $y_k$ given prefix $s_i$. Two scalar features are read off this matrix: consistency $\mathrm{Con}(\tau)$, the fraction of intermediate states for which the trajectory's own final answer is the nearest candidate, and volatility $\mathrm{Vol}(\tau)$, the normalized index of the last state that is nearest to a different candidate. These two features are the reward's entire content: each trajectory becomes a vector $\mathrm{Con}(\tau_i)[\cos(\mathrm{Vol}(\tau_i)),\sin(\mathrm{Vol}(\tau_i))]$, vectors are summed within groups that share a final answer, and the intrinsic reward is the group-mean magnitude of the combined vector — a polar-coordinate aggregation that the paper proves is monotone in both features and more robust to outliers than the linear difference $\mathrm{Con}-\mathrm{Vol}$. A curiosity reward, the average token log-probability along a transition penalized by $\ln[\mathrm{KL}(P_{i+1},U)+1]$ against a uniform distribution, is added to keep the sampled solution space diverse. The combined reward feeds a Reinforce++ update with a KL reference term, and the paper's variational bound (Proposition 2) is what connects this reward to a principled objective over latent reasoning paths.

What would settle it

Measure the separation between correct and incorrect trajectories in consistency and volatility (the statistics of Table 2) using the training policy's likelihoods at several checkpoints during a CoVo run; a significant narrowing or inversion of that gap while reported accuracy still rises would show the reward has decoupled from correctness. A complementary test: run CoVo on a prompt set where the model produces confidently wrong, homogeneous solutions, and check whether the intrinsic reward increases while accuracy falls — that pattern would be direct evidence of the reward-hacking the paper claims to avoid.

Watch

Extended reading notes

Core claim

The discovery the paper defends is that intermediate reasoning states carry a reliable, self-supervised signal of answer correctness, and that this signal is strong enough to replace external supervision in RL training for reasoning. Concretely, the paper defines a distance matrix $D\in\mathbb{R}^{T\times K}$ where each entry is the per-token negative log-likelihood from an intermediate state $s_i$ to a candidate final answer $y_k$, computed with the policy's own probabilities; from this matrix, a trajectory is summarized by its consistency $\mathrm{Con}(\tau)$ (the fraction of states nearest to the trajectory's own answer) and volatility $\mathrm{Vol}(\tau)$ (the normalized position of the last state that deviates toward another answer). Correct and incorrect trajectories separate cleanly in these two features across math, commonsense, and science prompts. CoVo converts the separation into a reward: trajectories are grouped by final answer, each trajectory is embedded as a vector $\mathrm{Con}(\tau_i)[\cos(\mathrm{Vol}(\tau_i)),\sin(\mathrm{Vol}(\tau_i))]$, the group vectors are summed, and the magnitude of the sum is the intrinsic reward, with a proof that this geometric aggregation keeps the desired monotonicity while damping outlier influence. A curiosity bonus based on token log-probabilities with a KL-from-uniform penalty is added to counter diversity collapse. The paper further claims a theoretical grounding: optimizing this reward is equivalent to variational inference over latent reasoning trajectories, and it proves that majority-voting rewards push the policy toward its current mode even when that mode is wrong, while the consistency-based reward does not. The empirical claim closes the argument: on MATH-500, GSM8K, AMC-23, Olympiad Bench, MMLU-Pro, GPQA, and CommonsenseQA, CoVo reaches accuracy comparable to or higher than supervised RL baselines on Llama3.2-3B, Qwen2.5-3B, and Qwen2.5-7B.

Load-bearing premise

The whole method rests on the premise that correct answers reliably look more consistent and less volatile than wrong answers in the model's own probabilities, and that this gap survives while the policy is being updated; the paper verifies the gap on the starting model but not on the training policy, so if the model becomes uniformly more confident, wrong answers could come to look "consistent" too and the reward would decouple from correctness.

Editorial extensions

If this is right

  • Reasoning RL no longer needs answer keys: any prompt set that yields multiple distinct sampled solutions becomes usable training data, since the reward comes from the model's own trajectory geometry.
  • The majority-voting failure mode is addressed head-on: the paper proves vote-counting rewards collapse the policy onto the current sample mode even when that mode is wrong, and reports that CoVo's reward accuracy stays stable over 500 training steps where majority voting drifts.
  • Training diversity is preserved: CoVo keeps sampled reasoning paths more diverse than rule-based GRPO on semantic-similarity, entropy, and lexical-overlap metrics, suggesting the curiosity bonus offsets the homogenization RL usually causes.
  • Math-only unlabeled training transfers: improvements appear on commonsense and science benchmarks the model never saw in training, indicating the learned behavior is general reasoning rather than answer-format memorization.
  • The approach is backbone-agnostic and scales in parameters: consistent gains appear on 3B and 7B models with different base checkpoints, which supports the paper's claim of a scalable unsupervised pathway.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test the paper does not run: recompute the consistency–volatility gap between correct and incorrect trajectories using the training policy's own likelihoods at later checkpoints; if the gap narrows as the policy becomes globally more confident, the reward's grounding in correctness weakens and the method would need periodic revalidation.
  • Because the reward needs several distinct final answers per prompt to be informative, CoVo's signal thins out on problems with large, flat answer spaces or small sampling budgets; testing it on code-generation or long-form proof tasks with sparse answer agreement would reveal the boundary of the mechanism.
  • The paper explicitly leaves visual-language reasoning untested; a natural extension is whether consistency of reasoning states still separates correct from incorrect answers when states mix language and perception, or whether perceptual grounding breaks the likelihood-based distance.
  • The vector aggregation is essentially a smoothed measure of how many steps commit to the final answer; a cheaper implementation that counts committed steps directly would test whether the geometric form earns its extra complexity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoVo, a self-rewarding reinforcement learning method for LLM reasoning. It defines two trajectory-level features—consistency (the fraction of intermediate states whose likelihood is closest to the trajectory's own final answer) and volatility (the normalized position of the last state that deviates from the own answer)—computed from a distance matrix based on the policy's token log-likelihoods. Trajectories are grouped by final answer, and an intrinsic reward is computed via linear or vector aggregation of these features; a curiosity bonus based on average token log-probability and a KL penalty is added to encourage exploration. The policy is optimized with Reinforce++. Experiments on three model families (Llama3.2-3B, Qwen2.5-3B, Qwen2.5-7B) across math, commonsense, and science benchmarks report that CoVo matches or exceeds rule-based supervised RL baselines, with additional diversity analyses and ablations. Theoretical sections provide propositions on model collapse of majority voting, a variational interpretation, and a convergence bound.

Significance. If the empirical claims hold, CoVo offers a meaningful step toward label-free RL for reasoning: it uses intermediate reasoning structure rather than only final-answer aggregation, potentially reducing some forms of reward hacking, and it is demonstrated on multiple model sizes with released code. The paper also includes careful ablations of reward components and an empirical analysis of trajectory patterns. However, the current evidence is not yet conclusive: the main comparisons lack statistical uncertainty, the training data selection excludes a potentially substantial subset of prompts, the curiosity reward's KL penalty is ill-specified in pseudocode, and the persistence of the consistency-volatility separation under the evolving policy is not established. These gaps are fixable but require additional empirical analysis and clarification.

major comments (4)
  1. [Table 1 and Section 4.2] Table 1 reports single runs without error bars or multiple seeds. Many differences against supervised RL baselines are within 0.2–0.4 percentage points (e.g., Llama3.2-3B MATH-500: CoVo 51.2 vs GRPO 51.8; Qwen2.5-3B GSM8K: CoVo 88.7 vs RLOO 89.1; Qwen2.5-7B MATH-500: CoVo 78.4 vs Reinforce++ 78.2). The headline claim 'comparable to or even surpassing supervised RL' is therefore not statistically supported. Please report standard deviations or confidence intervals across multiple seeds for CoVo and baselines, or at least demonstrate that the ordering is consistent across seeds.
  2. [Section 2 and Section 3.1] The reward computation and training deliberately exclude prompts for which all sampled answers are identical ('we focus on prompts where πθ generates multiple final answers'). This post-hoc filtering depends on the policy's own sampling and biases the learning signal toward prompts with answer diversity. The paper does not quantify the fraction of discarded prompts or compare against baselines trained on the same filtered subset. Without this control, one cannot attribute the performance gains to the consistency/volatility reward rather than to the selection of easier or more diverse prompts. Please report the number of prompts retained per dataset and include an ablation that applies the same filtering to supervised RL baselines.
  3. [Section 3.3.2, Eq. (10), Algorithm 2] The curiosity reward p_KL is not well-defined as written. In Eq. (10), p_KL = ln[KL(P_{i+1}, U) + 1], where P_{i+1} is described as the token probability distribution of state s_{i+1}; however, Algorithm 2 accumulates P as a list of log-probabilities (line 8) and then computes KL(P, U) at line 11. A KL divergence requires two probability distributions, not a list of scalar log-probabilities. Please specify the exact construction of P_{i+1} and correct the pseudocode so the reward is reproducible.
  4. [Section 4.3, Fig. 5, Proposition 3] The core assumption that correct trajectories have higher consistency and lower volatility than incorrect ones is validated only on the base model (Table 2, Fig. 2, Fig. 4). During RL, the reward is recomputed from the current policy's likelihoods (Eq. (1)), so the separation must be re-verified under distribution shift. Figure 5 reports 'reward accuracy' but never defines how it is computed, shows no checkpoint-wise feature distributions, and covers only one model. Similarly, Proposition 3 assumes that the CoVo reward ranks y_γ highest and has low misclassification probability, but does not prove these properties hold for the trained policy. Please provide an analysis of consistency/volatility stratifications at multiple training checkpoints, and either prove or empirically demonstrate that the reward-correctness association persists under the evolving policy.
minor comments (5)
  1. [Table 2] The table header contains a typo: 'V olatility' should be 'Volatility'. Additionally, some standard deviations are very large (e.g., 0.867±0.785 for GPQA), which suggests heavy-tailed distributions; consider reporting medians or interquartile ranges for clarity.
  2. [Figure 5] The legend contains a typo: 'Groud Truth' should be 'Ground Truth'. Please also clarify what the three reward curves in the left panel represent and how 'Reward Accuracy' is computed, as this is central to the reward-hacking claim.
  3. [Section 3.2, Eq. (5)] The volatility formula uses a max over a set containing a boolean expression (|D[i,0] ≠ min_k D[i,k]|), which is confusing. It would be clearer to define Vol(τ) = (1/T) · max{ i : D[i,0] ≠ min_{0≤k<K} D[i,k] }, with the convention that the maximum is 0 if the set is empty.
  4. [Appendix B.3.1] The monotonicity proof for rV_int with respect to volatility is heuristic: the step 'When Δv causes Vi to deviate from the current group, it tends to be sizable. Therefore (Vi − Vj + Δv/2) > 0, which leads to ΔrV_int(Vi)<0' is not a formal argument. Since the paper states that a formal proof is provided, please tighten this derivation or explicitly state the additional conditions needed for the sign to hold.
  5. [Proposition 2, Eq. (12)] The notation 'r(s,x,y) ∈ [0,1] ∝ log πθ(y|x⊕s)' mixes proportionality with a bounded range; log-probabilities are non-positive, so the stated range is incompatible with a positive proportionality constant. Please clarify the exact relationship, e.g., by defining r as a normalized or shifted version of log πθ.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoVo's reward is self-referential by design, but the paper's central empirical claims rest on held-out benchmarks and conditional theoretical statements, not on a definitional equivalence.

full rationale

The paper computes its intrinsic reward from the policy's own likelihoods (Eqs. 1, 4, 5, 9), but that is the intended mechanism of a self-rewarding method rather than a hidden circular reduction. The key premise that correct trajectories show higher consistency and lower volatility than incorrect ones is empirically checked at initialization on external datasets using ground-truth stratification (Table 2, Figure 2, Section 4.3), and the main performance claim is evaluated with held-out Pass@1 accuracy on MATH-500, GSM8K, AMC-23, Olympiad Bench, MMLU-Pro, GPQA, and CommonsenseQA (Table 1). The theoretical statements are not used to derive the empirical results: Proposition 2 is a standard variational lower bound in which the reward is defined as proportional to log-likelihood, so the inequality holds by construction but does not by itself establish that CoVo's consistency/volatility reward improves correctness; Proposition 3 explicitly assumes that the reward ranks the high-consistency answer highest and has low misclassification probability, making the convergence conclusion conditional rather than a proof that the premise survives training. Self-citations in the paper (e.g., references 23, 60, 58) appear only in related-work or follow-up contexts and are not load-bearing for the derivation. The concern that the consistency-correctness correlation may degrade under distribution shift during optimization is a legitimate correctness or robustness risk, but it is not circularity under the definitions used here, because the paper does not claim to prove that persistence and instead reports external benchmark evaluations. Therefore, no circular step can be exhibited from the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it relies on the unproven assumption that consistency and volatility of intermediate states track correctness across training, and on the validity of model likelihood as a distance measure. The free parameters are standard training hyperparameters rather than fitted constants, but the central reward design choices (grouping by exact answer string, vector-angle encoding of volatility) are made without sensitivity analysis.

free parameters (3)
  • n_samples_per_prompt = 16
    Number of trajectories sampled per prompt; determines the size of the distance matrix and reward grouping. Chosen by hand, not tuned on validation.
  • init_kl_coef = 1e-4
    KL regularization coefficient in the RL objective; a hyperparameter of the training setup.
  • learning_rate = 5e-7
    Optimizer learning rate; affects convergence but not the reward definition.
assumptions (5)
  • domain assumption Correct reasoning trajectories exhibit higher consistency and lower volatility than incorrect ones, and this separation persists under policy training.
    Empirically shown on base models across MATH, MMLU, GPQA, and CommonsenseQA (Table 2), but relied on as a persistent signal during RL; no post-training verification.
  • domain assumption The model's own token likelihoods are a valid measure of distance between intermediate states and final answers.
    Distance d(s_i, y) in Eq. (1) treats negative log-likelihood as a proxy for semantic distance; no calibration against human judgments.
  • domain assumption Grouping responses by exact final-answer string is sufficient to form meaningful comparison groups.
    Trajectories with the same final answer string are averaged together; semantically equivalent but differently formatted answers are treated as distinct groups.
  • standard math Tabular policy approximation and small KL coefficient are valid for the convergence analysis.
    Proposition 3 uses the tabular policy gradient flow from Razin et al. [32], which is a simplification of autoregressive LLM policies.
  • ad hoc to paper The reward function ranks the correct trajectory highest and has low misclassification probability (Proposition 3 assumptions).
    This is assumed to prove convergence; it does not establish that CoVo's reward actually satisfies this condition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning." pith.science (2026). https://pith.science/paper/6XJTB6SV

@misc{pith2026250608745,
  author       = {Pith},
  title        = {Pith review of: Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XJTB6SV}},
  note         = {Machine review of arXiv:2506.08745}
}
read the original abstract

Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo.

Figures

Figures reproduced from arXiv: 2506.08745 by the authors.

Figure 1
Figure 1. (Left) A conceptual illustration of distinct reasoning patterns: Correct trajectories con [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Normalized distance curve of correct and incorrect trajectories with varying state numbers. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of reason￾ing diversity on different methods. performance on par with or even exceeding the supervised RL baselines in most cases. Given the growing demand for large-scale RL in industry, this scalability demonstrates the effectiveness of our approach and its potential for broad real￾world deployment. Domain Generalization. While our training dataset solely consisting of data from the mathematical reas… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The distribution of consistency and volatility for correct and incorrect responses across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Training dynamics with different types of reward signals using Qwen2.5-3B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Token-level confidence arbitration between VL and text teachers during on-policy distillation improves multimodal student reasoning over single-teacher OPD and GRPO.

  2. Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

    cs.LG 2025-09 conditional novelty 5.0 of 10

    EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.

Reference graph

Works this paper leans on

65 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Kakade, Jason D

    Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift.J. Mach. Learn. Res., 22:98:1–98:76, 2021

    work page 2021

  2. [2]

    Open r1: Evaluating llms on uncontaminated math competitions, 2025

    Loubna Ben Allal, Lewis Tunstall, Anton Lozhkov, Elie Bakouch, Guilherme Penedo, and Gabriel Martín Blázquez Hynek Kydlicek. Open r1: Evaluating llms on uncontaminated math competitions, 2025

    work page 2025

  3. [3]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [4]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  5. [5]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  6. [6]

    An empirical study on eliciting and improving r1-like reasoning models.arXiv preprint arXiv:2503.04548, 2025

    Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving r1-like reasoning models.arXiv preprint arXiv:2503.04548, 2025. 11

    arXiv 2025

  7. [7]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 65 references
  1. [9]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [11]

    Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scienti...

    2024

  4. [12]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InConference on Neural Information Processing Systems, 2021

    2021

  5. [13]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  6. [14]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143, 2024

    Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143, 2024

    2024 arXiv

  7. [15]

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  8. [16]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, 2019

    2019

  10. [18]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InACM Symposium on Operating Systems Principles, 2023

    2023

  11. [19]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  12. [20]

    Numinamath, 2024

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024

    2024

  13. [22]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, 2024

    2024

  14. [23]

    A survey of direct preference optimization.arXiv preprint arXiv:2503.11701, 2025

    Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, and Dacheng Tao. A survey of direct preference optimization.arXiv preprint arXiv:2503.11701, 2025

    2025 arXiv

  15. [24]

    Exploring the limit of outcome reward for learning mathematical reasoning

    Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781, 2025. 12

    2025 arXiv

  16. [25]

    Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  17. [26]

    On the global convergence rates of softmax policy gradient methods

    Jincheng Mei, Chenjun Xiao, Csaba Szepesvári, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. InInternational Conference on Machine Learning, 2020

    2020

  18. [27]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  19. [28]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. InConference on Neural Information Processing Systems, 2024

    2024

  20. [29]

    Are NLP models really able to solve simple math word problems? InNorth American Chapter of the Association for Computational Linguistics, 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? InNorth American Chapter of the Association for Computational Linguistics, 2021

    2021

  21. [30]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InConference on Neural Information Processing Systems, 2023

    2023

  22. [31]

    Semantic cosine similarity

    Faisal Rahutomo, Teruaki Kitasuka, Masayoshi Aritsugi, et al. Semantic cosine similarity. InInternational student conference on advanced science and technology, 2012

    2012

  23. [32]

    What makes a reward model a good teacher? an optimization perspective.arXiv preprint arXiv:2503.15477, 2025

    Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective.arXiv preprint arXiv:2503.15477, 2025

    2025

  24. [33]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing, 2019

    2019

  25. [34]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In Conference on Language Modeling, 2024

    2024

  26. [35]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  27. [36]

    A mathematical theory of communication.The Bell system technical journal, 1948

    Claude E Shannon. A mathematical theory of communication.The Bell system technical journal, 1948

    1948

  28. [37]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  29. [38]

    Cross- ing the reward bridge: Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Cross- ing the reward bridge: Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    2025 arXiv

  30. [39]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  31. [40]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InNorth American Chapter of the Association for Computational Linguistics, 2019

    2019

  32. [41]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023

    2023

  33. [43]

    Latent space chain-of-embedding enables output-free llm self-evaluation.arXiv preprint arXiv:2410.13640, 2024

    Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. Latent space chain-of-embedding enables output-free llm self-evaluation.arXiv preprint arXiv:2410.13640, 2024

    2024 arXiv

  34. [44]

    Wong, Zhuosheng Zhang, and Rui Wang

    Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, Zhuosheng Zhang, and Rui Wang. Embedding trajectory for out-of-distribution detection in mathematical reasoning. InConference on Neural Information Processing Systems, 2024. 13

    2024

  35. [45]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024

  36. [46]

    Cream: Consistency regularized self-rewarding language models.arXiv preprint arXiv:2410.12735, 2024

    Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models.arXiv preprint arXiv:2410.12735, 2024

    2024 arXiv

  37. [47]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InConference on Neural Information Processing Systems, 2022

    2022

  38. [48]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge.arXiv preprint arXiv:2407.19594, 2024

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge.arXiv preprint arXiv:2407.19594, 2024

    2024 arXiv

  39. [49]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2502.14768, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2502.14768, 2025

    2025 arXiv

  40. [50]

    Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613, 2025

    Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613, 2025

    2025 arXiv

  41. [51]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  42. [53]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  43. [54]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. InInternational Conference on Machine Learning, 2024

    2024

  44. [55]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  45. [56]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    2025 arXiv

  46. [57]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  47. [58]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  48. [59]

    Sample efficient reinforcement learning with reinforce

    Junzi Zhang, Jongho Kim, Brendan O’Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce. InAAAI conference on artificial intelligence, 2021

    2021

  49. [60]

    Reasoning with reinforced functional token tuning.arXiv preprint arXiv:2502.13389, 2025

    Kongcheng Zhang, Qi Yao, Baisheng Lai, Jiaxing Huang, Wenkai Fang, Dacheng Tao, Mingli Song, and Shunyu Liu. Reasoning with reinforced functional token tuning.arXiv preprint arXiv:2502.13389, 2025

    2025 arXiv

  50. [61]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812, 2025

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812, 2025

    2025 arXiv

  51. [62]

    Process-based self-rewarding language models.arXiv preprint arXiv:2503.03746, 2025

    Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. Process-based self-rewarding language models.arXiv preprint arXiv:2503.03746, 2025

    2025 arXiv

  52. [63]

    Self-consistency of the internal reward models improves self-rewarding language models.arXiv preprint arXiv:2502.08922, 2025

    Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. Self-consistency of the internal reward models improves self-rewarding language models.arXiv preprint arXiv:2502.08922, 2025. 14

    2025 arXiv

  53. [64]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomcza...

    2024

  54. [65]

    Landscape of thoughts: Visualizing the reasoning process of large language models.arXiv preprint arXiv:2503.22165, 2025

    Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. Landscape of thoughts: Visualizing the reasoning process of large language models.arXiv preprint arXiv:2503.22165, 2025

    2025

  55. [66]

    Texygen: A benchmarking platform for text generation models

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. InInternational ACM SIGIR Conference on Research and Development in Information Retrieval, 2018

    2018

  56. [67]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877, 2024

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  57. [68]

    ground-truth

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084, 2025. 15 Appendix Table of Contents A More Experimental Results and Discussion 17 A.1 ...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.