Pith. sign in

REVIEW 4 major objections 7 minor 8 cited by

Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Using a language model's own confidence as the reward, RLSC lifts math accuracy by 9–22 points with no labels and only 10–20 training steps.

desk verdict The headline gains are not supported by the paper's own math: the loss is self-distillation, not self-confidence RL, and the AIME evaluation likely leaks the training set. read the letter →

arxiv 2506.06395 v3 pith:ZTCDUZCN submitted 2025-06-05 cs.CL cs.LG

Pengyi Li , Matvey Skripkin , Alexander Zubrey , Andrey Kuznetsov , Ivan Oseledets This is my paper
classification cs.CLcs.LG
keywords reinforcementlearningself-confidencezero-labelmodesharpeninglanguagemodelpost-trainingmathematicalreasoningfew-shotRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RLSC is the claim that post-training for reasoning can be driven entirely by the model's own self-confidence: the reward is the probability the model assigns to its own sampled answers, so no labels, preference models, or handcrafted reward functions are needed. The paper derives this objective from Test-Time RL's majority voting, showing that voting is equivalent to sharpening the output distribution toward its mode, and turns that into a differentiable loss. With just 16 generations per question and 10–20 optimizer steps on Qwen2.5-Math-7B, the method reports accuracy gains of roughly 10–22 points across five competition-math benchmarks. If the claims hold, post-training becomes dramatically cheaper and simpler, and a pretrained model's internal signal may be enough to improve itself.

What carries the argument

The central object is the self-confidence functional $F(p_\theta) = \sum_y p_\theta(y|x)^2$, the probability that two i.i.d. samples from the model's output distribution coincide. The paper calls the operation that maximizes it 'mode sharpening': it concentrates probability mass on the most likely response, which is the same effect majority voting achieves, but expressed as a differentiable objective. The load-bearing identity is the log-trick gradient $\nabla_\theta F = \mathbb{E}_{y \sim p_{\mathrm{old}}}[p_{\mathrm{old}}(y|x)\nabla_\theta \log p_\theta(y|x)]$, which converts the functional into a weighted maximum-likelihood loss $L_1 = -\sum_y p_{\mathrm{old}}(y|x)\log p_\theta(y|x)$; the smoothed variant $L_2 = -\sum_y(p_{\mathrm{old}}(y|x)+\alpha)\log p_\theta(y|x)$ adds a constant to stabilize optimization. Because the weighting probabilities come from a frozen copy of the model, the reward is entirely self-generated, and gradients flow only into the updated model's log-likelihoods.

What would settle it

Train RLSC on the AIME2024 set and evaluate only on held-out AIME problems that are provably absent from the training questions; if the +13.4% gain evaporates, the result is memorization rather than mode sharpening. A second decisive test is to apply RLSC to a model whose confidence is adversarially miscalibrated — for example, one that is overconfident on plausible-looking distractors — and check whether accuracy rises along with confidence.

Watch

Extended reading notes

Core claim

The paper's central claim is that maximizing the self-confidence functional $F(p_\theta) = \mathbb{E}_{y \sim p_\theta(y|x)}[p_\theta(y|x)] = \sum_y p_\theta(y|x)^2$ is a sufficient training objective for improving reasoning accuracy in a strong pretrained model. This functional is the collision probability of two independent samples from the model's output distribution, and the paper argues that maximizing it is exactly what majority-voting pseudo-labeling does implicitly: it sharpens the distribution toward its mode. The gradient identity $\nabla_\theta F = \mathbb{E}_{y \sim p_{\mathrm{old}}}[p_{\mathrm{old}}(y|x)\nabla_\theta \log p_\theta(y|x)]$, obtained with a frozen copy $p_{\mathrm{old}}$, yields the reward-weighted loss $-\sum_y p_{\mathrm{old}}(y|x)\log p_\theta(y|x)$, optionally smoothed by a constant $\alpha$. Trained for 10–20 steps on the AIME2024 set with 16 samples per question, the Qwen2.5-Math-7B model gains +13.4 points on AIME24, +21.2 on MATH500, +21.7 on Minerva Math, +20.8 on Olympiadbench, and +9.7 on AMC23; the paper reports the same pattern, more weakly, at the 1.5B scale and attributes the effect to mode sharpening producing more concise and direct reasoning.

Load-bearing premise

The entire method rests on the assumption that a pretrained model's self-assigned probability on a sampled answer is a trustworthy reward signal — that sharpening the model's confidence moves it toward correct answers rather than merely toward overconfident ones.

Editorial extensions

If this is right

  • Post-training for reasoning no longer needs human labels, preference models, verifiers, or reward engineering; the frozen model's own output probabilities are the reward signal.
  • Very small compute budgets suffice: 16 samples per question and 10–20 optimizer steps on a single training set transfer to gains on AIME24, MATH500, Minerva Math, Olympiadbench, and AMC23, with smaller gains on GSM8K and MMLU-Stem.
  • RLSC removes the preprocessing burden of TTRL, which needed 64 samples per question and a clean separation of answer from reasoning trace for majority-vote pseudo-labels; RLSC needs only the model's token log-probabilities.
  • Fine-tuned models become more concise: the paper observes the RLSC model answering correctly with shorter, more direct reasoning, without any 'think step by step' prompting, though it does not formally quantify the length reduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: if RLSC transfers beyond mathematics, it suggests a general post-training recipe — maximize the collision probability of the model's own output distribution — which would turn self-consistency from a decoding-time heuristic into a training signal for code, factoid QA, and other tasks where the mode is often correct.
  • Editorial: the method's validity depends on the pretrained model's confidence being correlated with correctness. A direct way to test this is to apply RLSC to a deliberately overconfident or adversarially miscalibrated model; if accuracy fails to rise along with confidence, the reported gains may be a calibration artifact rather than genuine reasoning improvement.
  • Editorial: because the paper trains on the AIME2024 dataset and evaluates on AIME24 without stating a train/test split, the headline +13.4% number should be read cautiously until the overlap is checked; a cleaner design trains on one competition set and evaluates on another, disjoint one.
  • Editorial: the self-confidence functional is estimated from only 16 Monte-Carlo samples per question, so the gradient is noisy; the smoothing constant $\alpha$ may be stabilizing the optimization more than the paper's preliminary ablation suggests, and a full sweep over sample counts and $\alpha$ would clarify how much of the robustness is built-in versus tuned.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Reinforcement Learning via Self-Confidence (RLSC), a fine-tuning method that uses the model's own output probability distribution as a reward signal, with the stated goal of 'sharpening' the distribution toward its mode and thereby eliminating the need for labels, preference models, or reward engineering. The authors derive a self-confidence objective F(p_theta) = sum_y p_theta(y|x)^2, propose losses L1 and L2 based on a frozen copy of the model, and apply the method to Qwen2.5-Math-7B using only 16 samples per question and 10 or 20 training steps on an AIME2024 dataset. They report large accuracy gains on AIME24, MATH500, Minerva Math, Olympiadbench, and AMC23, and claim that the model learns concise reasoning without prompting. The paper also includes qualitative examples and a short ablation deferral.

Significance. If the method worked as described, it would be a noteworthy contribution: zero-label, low-compute post-training for reasoning models, with a simple differentiable objective replacing handcrafted rewards or majority-vote pseudo-labeling. The paper is transparent about its loss and training setup, and re-evaluating baselines with a common public script is good practice. However, the central gradient derivation is mathematically incorrect, so the implemented loss does not maximize the stated self-confidence objective; the reported gains therefore cannot be attributed to the proposed mechanism. Additionally, the AIME24 training/evaluation overlap and an unreliable answer label in a qualitative example further undermine the empirical claims. The idea of deriving self-training from ensemble agreement is appealing, but as presented the central claim is not supported and would require a corrected derivation and substantially clarified evaluation before a fresh assessment.

major comments (4)
  1. [§2.2, Eqs. (4)–(7)] Equation (4) is not the gradient of F(p_theta) = sum_y p_theta(y)^2. The correct gradient is nabla F = 2 sum_y p_theta(y) nabla p_theta(y) = 2 E_{y~p_theta}[p_theta(y) nabla log p_theta(y)], whereas Eq. (4) uses E_{y~p_old}[p_old(y) nabla log p_theta(y)] with p_old frozen. Consequently, L1 in Eq. (5) is the cross-entropy from p_old to p_theta; its minimizer is p_theta = p_old, and at initialization its gradient is exactly zero. Adding the alpha term in L2 introduces a maximum-entropy regularizer -alpha sum_y log p_theta, which pushes the distribution toward uniform rather than toward the mode. None of these losses implements the 'mode sharpening' objective that the paper's central claim rests on, so the reported gains cannot be attributed to RLSC as derived.
  2. [§2.3 and §3.1] The training set is described in §2.3 as 'the AIME2024 dataset' and the evaluation benchmark in §3.1 is 'AIME24 [14]', with no train/test split stated; reference [14] is the same NuminaMath corpus in both places. If the 16 samples per question are generated from the same problems that appear in the AIME24 evaluation, then the reported +13.4% improvement on AIME24 reflects memorization or overlap rather than generalization. The paper must state explicitly whether the AIME24 evaluation problems are disjoint from the training questions and, if not, report results on a held-out split.
  3. [§3.3 / Appendix Case 1] The qualitative example in Appendix Case 1 labels the RLSC output 'Final Answer (Correct): 2', but for a = 2 the piecewise function does not intersect y = 2017 twice (the second-branch intersection x = 1006.5 lies outside x < a), so this answer appears incorrect. This indicates that the evaluation pipeline does not reliably verify answers, which calls into question the accuracy numbers in Table 2 and the 'emergent behavior' claims in §3.2.
  4. [§3.1, Table 2] The text says the original Qwen model 'often fails to function properly' under direct evaluation. If the baseline's outputs are not in the expected format and are counted as wrong by the shared evaluation script, the large deltas may be largely due to format adaptation rather than improved reasoning. The paper should provide a breakdown of parse failures versus mathematical errors for the baseline and the tuned model to support the attribution of the improvements to RLSC.
minor comments (7)
  1. [Title page] The author affiliation 'Skotech' should be 'Skoltech'.
  2. [§1, Contribution 3] 'massage templates' should be 'message templates'.
  3. [§2.3] The text contains a typo, 'absense' should be 'absence'.
  4. [Table 2] The delta entries for GSM8K and MMLU are written as '2.0' and '5.3' without the '+' sign used elsewhere in the row; please make the formatting consistent.
  5. [Table 1] Table 1 is difficult to read: the 'Loss function' and 'Functional' columns appear garbled (e.g., 'RLHF lossp old logpE pθ [pθ]'); please reformat the table so each row clearly associates a loss with its optimized functional.
  6. [Algorithm 1] The pseudocode contains formatting issues, including broken line breaks inside variable names and missing indentation around 'loss.backward()' and 'optimizer.step()', which make the algorithm harder to follow.
  7. [References] References [14] and [15] are identical; the paper uses [14] for both the training corpus and the AIME24 evaluation set, which should be clarified in the text.

Circularity Check

1 steps flagged · score 6.0 of 10

The central loss is self-distillation into the frozen model, not maximization of the stated self-confidence objective; the claimed mode sharpening reduces to the input distribution by construction.

  1. other [Section 2.2, Eqs. (3)-(5)]
    "∇θF(pθ) = X y ∇θpθ(y|x)·p θ(y|x) = E y∼pθ [∇θpθ(y|x)] = E y∼pold [pold(y|x)· ∇θ logp θ(y|x)] (4) ... This leads to the following training loss: L1 =− X y pold(y|x)·logp θ(y|x) (5)"

    The gradient of L1 is −Σ_y p_old(y)∇log p_θ(y). At initialization p_θ=p_old, Σ_y p_old ∇log p_old = Σ_y ∇p_old = 0, so L1 yields no parameter update and its minimizer is p_θ=p_old. Thus Eq. (5) is KL(p_old||p_θ), i.e., self-distillation into the frozen starting distribution, not a descent step for F=Σ p_θ^2. The only nonzero term when initialized, −αΣ(1/p_old)∇p_θ, is the gradient of an entropy-increasing regularizer, not of mode sharpening. The predicted sharpening therefore reduces to the paper's own input p_old by construction.

full rationale

The paper's derivation chain from majority voting to the self-confidence objective F(p_θ)=Σ_y p_θ(y)^2 is internally broken at Eq. (4): the expectation over p_θ is replaced by a frozen p_old weighting, which turns the loss into self-distillation. Minimizing L1 keeps p_θ equal to p_old, so the central mechanism cannot produce the reported sharpening; any benchmark gains must come from other components (format adaptation, evaluation conditions, or possible AIME24 train/eval overlap), which are correctness concerns rather than circularity. No load-bearing self-citations appear, and the α smoothing is a hand-set hyperparameter, not a fitted constant. Because the central claimed derivation reduces to its own input distribution, the circularity score is 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on the assumption that maximizing self-confidence improves correctness, that the AIME training and evaluation sets are disjoint, and that the frozen model distribution is a valid target for the gradient. The only hand-tuned constants are standard hyperparameters plus the smoothing alpha.

free parameters (5)
  • alpha (smoothing constant) = 0.1
    Added to the self-confidence weight in L2 to stabilize optimization; the paper says even small values like 0.1 improve convergence and generalization, so it is a hand-tuned constant.
  • number of samples per question = 16
    Chosen for the few-shot setup; no ablation shown.
  • temperature = 0.5
    Used for generating completions; a standard but hand-chosen hyperparameter.
  • learning rate = 1e-5
    AdamW learning rate for fine-tuning, chosen by standard practice.
  • training steps = 10 or 20
    The paper reports results with 10 or 20 steps without specifying which number corresponds to each benchmark.
assumptions (4)
  • domain assumption Maximizing the model's self-confidence (mode sharpening) improves answer accuracy on downstream benchmarks.
    The entire method rests on this; the paper provides no proof and its own GPQA result for the 1.5B model shows a drop.
  • domain assumption The AIME2024 training set is disjoint from the AIME24 evaluation set.
    Necessary for the claimed AIME improvement to represent generalization; never stated.
  • domain assumption p_old, the frozen model, is a valid weighting for the policy gradient of the self-confidence objective.
    The gradient estimator in Eq. (4) is unbiased only when p_old approximates the current policy; used as a single-step approximation without importance weighting.
  • domain assumption Qwen2.5-Math's output distribution is sufficiently calibrated that its mode is usually correct.
    If the model's most likely answer is often wrong, sharpening would entrench errors; the paper does not measure calibration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models." pith.science (2026). https://pith.science/paper/ZTCDUZCN

@misc{pith2026250606395,
  author       = {Pith},
  title        = {Pith review of: Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZTCDUZCN}},
  note         = {Machine review of arXiv:2506.06395}
}
read the original abstract

Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision.

Figures

Figures reproduced from arXiv: 2506.06395 by the authors.

Figure 1
Figure 1. Combined visualization: (a) RL via Self Confidence workflow schema; (b) Probability distribution before and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Relationship-Centered Care: Relatedness and Responsible Design for Human Connections in Mental-Health Care

    cs.HC 2026-03 accept novelty 7.0 of 10

    Length-aware GFlowNet matching of α-power base distributions elicits either stronger LLM reasoning (α>1) or restored creativity (α<1) without external supervision, matching or beating RLIF and GRPO baselines.

  2. On-Policy Self-Distillation without Any Supervision

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A language model improves its own math reasoning by distilling its majority-vote consensus into the prefixes of its own disagreeing answers, with no external labels.

  3. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.

  4. Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Self-rewarding RL can be stabilized by ensembling multiple policy models' majority-vote rewards, reaching within 3.6% of verifiable-reward RL on math benchmarks.

  5. Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Ranking

    cs.CV 2025-09 conditional novelty 6.0 of 10

    EvoQuality lets a VLM self-train for image quality assessment using majority-voted pairwise preferences as pseudo-labels, improving PLCC by about 32% and rivaling supervised models.

  6. Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Selecting the most confident 32-token prefix and completing only it gives better accuracy per compute than majority voting on five math reasoning datasets, using only the model's own confidence as a selector.

  7. No Free Lunch: Rethinking Internal Feedback for LLM Reasoning

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Internal feedback rewards (entropy and self-certainty) improve base LLM math reasoning only in early training and degrade later, with little benefit for instruct models.

  8. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

23 extracted references · 2 canonical work pages · cited by 8 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    Spc: Evolving self-play critic via adversarial games for llm reasoning.arXiv preprint arXiv:2504.19162, 2025

    Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yee K Wong. Spc: Evolving self-play critic via adversarial games for llm reasoning.arXiv preprint arXiv:2504.19162, 2025

    arXiv 2025

  4. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 8 arXivTemplateA PREPRINT

    arXiv

  5. [5]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    work page 2021

  6. [6]

    Evalchemy: Automatic evals for llms, 2024

    Etash Guha, Negin Raoof, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad-Falcon, et al. Evalchemy: Automatic evals for llms, 2024

    work page 2024

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  8. [8]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

Show all 23 references
  1. [9]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.1...

    2024 arXiv

  2. [10]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  3. [11]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  4. [12]

    Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    2021

  5. [13]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  6. [15]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repositor...

    2024

  7. [16]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  8. [17]

    Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    2024 arXiv

  9. [18]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  10. [19]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  11. [20]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  12. [21]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  13. [22]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  14. [23]

    Absolute zero: Reinforced self-play reasoning with zero data.arXiv preprint arXiv:2505.03335, 2025

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zi- long Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data.arXiv preprint arXiv:2505.03335, 2025

    2025 arXiv

  15. [24]

    Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084, 2025

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084, 2025. 9 arXivTemplateA PREPRINT A Case Study We present the performance of our model o...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.