Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that rule-based verifiers in LLM math reasoning produce many false negatives that silently harm reinforcement learning, and shows that a small LLM verifier that corrects those false negatives improves pass rates by up to…

desk verdict Solid measurement of verifier false negatives, a plausible cheap fix, and a theory section that should be ignored as written. read the letter →

arxiv 2505.14625 v2 pith:IHVWNFHO submitted 2025-05-20 cs.LG cs.AIcs.CL

Zhangchen Xu , Yuetai Li , Fengqing Jiang , Bhaskar Ramasubramanian , Luyao Niu , Bill Yuchen Lin , Radha Poovendran This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords falsenegativesanswerverificationreinforcementlearningGRPOLLMreasoningmathematicalrewardsignalsTinyV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that rule-based answer verifiers—programs that decide whether a model's answer matches the ground truth—frequently reject mathematically correct answers, and that these false negatives quietly degrade reinforcement learning for LLM reasoning. In Big-Math-RL-Verified, it finds that 38.5% of the 226K prompt–response pairs that the Prime Verifier marks as incorrect are actually correct, and it shows both empirically and theoretically that this corrupts the reward signal and slows convergence. The remedy is TinyV, a 1.5B parameter LLM verifier fine-tuned on 638K labeled examples, deployed as an add-on that re-checks only the responses the rule-based verifier rejects. Using TinyV during GRPO training raises pass rates on the authors' hard-to-verify benchmark by up to 10 points over the rule-based baseline while adding only about 6% compute. The paper's broader message is that the verifier's false-negative rate, not just the policy, is a first-order bottleneck in RL for mathematical reasoning.

What carries the argument

The load-bearing object is the false-negative (FN) detector TinyV: a lightweight 1.5B LLM binary classifier that takes a question, ground-truth answer, and model answer and returns 'True' or 'False', trained by supervised fine-tuning on 638K instances built from LLM annotations plus synthetically generated equivalence-preserving variations. It is deployed as an add-on to a rule-based verifier: the rule-based verifier runs first, and TinyV is queried only when the response is marked incorrect, flipping rewards for rescued correct answers from 0 to 1 at about 6% extra compute. The theoretical side is a step-wise learnability measure, the reverse KL divergence $D_k = D_{KL}(\pi_{k-1}\|\pi_k)$ between consecutive GRPO policies; Theorem 1 proves $\delta_k = D_{k,GT} - D_{k,FN} > 0$ whenever the ground-truth success probability is higher and grows at a non-exponential rate. This mechanism explains why correcting FNs improves training: it restores informative gradient signals on hard prompts that otherwise would have none.

What would settle it

Run the same TinyV-trained RL runs but score HardVerify-Math with a fully symbolic equivalence checker (or a second, independently trained LLM judge) and check whether the ~10-point gap over the rule-based baseline shrinks to the ~1-point level seen on MATH; if it does, the claim that TinyV improves true mathematical correctness is falsified.

Watch

Extended reading notes

Core claim

The discovery on the paper's own terms: false negatives in verification are prevalent, harmful, and fixable. Among responses that Prime Verifier labels incorrect but that contain a boxed answer, 87K of 226K (38.5%) are actually correct, and 42.1% of the affected unique prompts contain at least one correct answer that was rejected. During RL training, the fraction of prompts experiencing at least one false-negative rollout climbs to 46.7% (Qwen2.5-7B) and 50.5% (Qwen2.5-Math-7B), and LLM-based relabeling shows a 15 percentage-point gap in the all-wrong ratio early in training, meaning Prime Verifier deprives the policy of instructive gradients on exactly the hard prompts. The theoretical result, Theorem 1, states that policies trained with ground-truth rewards have strictly greater step-wise learnability, measured as reverse KL divergence between consecutive GRPO policies, than policies trained with false-negative-corrupted rewards. TinyV corrects this by supervised fine-tuning on 638K real and synthetic false-negative/true-negative examples, and training with it lifts HardVerify-Math scores from 58.64% to 68.68% (Qwen2.5-7B) and from 62.65% to 69.08% (Qwen2.5-Math-7B) while accelerating convergence.

Load-bearing premise

The argument assumes the LLM judge that scores the authors' hard-to-verify benchmark agrees with the LLM labels used to train TinyV—if they share a bias, part of the reported gain measures self-consistency rather than correctness—and the convergence theorem likewise assumes ground-truth success probabilities rise steadily without blowing up.

Editorial extensions

If this is right

  • Training with TinyV rather than the rule-based verifier alone raises final pass rates on HardVerify-Math by about 10 points for Qwen2.5-7B and by about 6.4 points for Qwen2.5-Math-7B.
  • TinyV-trained models converge to the baseline's final performance earlier in training, so the same quality can be reached with fewer RL steps.
  • Models trained on easily verifiable questions (DeepScaleR-style) perform markedly worse on hard-to-verify questions than models trained on hard-to-verify data with TinyV, suggesting that the verifier's coverage determines what the policy learns.
  • Because most false negatives come from natural-language elements, formatting, and notation differences, a taxonomy-based understanding of equivalence is needed for scalable verification beyond string matching.
  • On standard benchmarks (MATH, AMC, Olympiad), the gains are roughly one point and AMC is mixed, indicating that the benefit concentrates on hard-to-verify answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If false negatives corrupt rewards in math verification, the same failure mode is likely to appear in other verifiable domains the paper lists—theorem proving, code, medicine—where rule-based checks are common; the paper only demonstrates the math case.
  • An external evaluation should test whether the 10-point gap persists when HardVerify-Math is scored by an independent judge (e.g., a symbolic equivalence checker or a different LLM) rather than the LLM judge used in the paper; if the judge shares TinyV's training bias, part of the gain could be self-consistency with the annotator rather than true correctness.
  • TinyV inherits the rule-based verifier's answer-extraction (boxed-only) mechanism, so extending it to full-solution verification could recover additional false negatives but would likely cost more than the reported 6% overhead.
  • The theoretical result depends on the GRPO policy dynamics lemma; applying the same FN-correction idea to PPO or offline methods may show a different magnitude of benefit, though the qualitative direction should hold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that rule-based verifiers used in RL for LLM math reasoning produce many false negatives (FNs)—correct responses marked incorrect—and that these FNs degrade policy learning. The authors quantify FNs in Big-Math-RL-Verified (38.5% of seemingly incorrect responses with a boxed answer are actually correct), analyze FN causes, present an empirical study of FN effects during GRPO training, propose a lightweight verifier called TinyV that augments Prime Verifier in an add-on mode, and introduce a 250-question HardVerify-Math benchmark. They report that TinyV improves final pass rates by up to 10 points on HardVerify-Math and yields faster convergence, with only ~6% training overhead. A theoretical claim (Theorem 1) is offered to show that false negatives reduce step-wise learnability measured by reverse KL divergence.

Significance. If the empirical and theoretical claims hold, the paper addresses a practically important and under-studied failure mode: rule-based verifiers can reject mathematically correct responses, and correcting those false negatives can improve RL training for reasoning. The empirical FN measurement is a genuine strength: two independent LLM annotators were used, a manual review of 200 FN cases reported 99.5% agreement, and the RL experiments use standard GRPO with consistent trends. The proposed TinyV is lightweight (1.5B parameters, 6% overhead), and the code is released. The main significance is therefore conditional on resolving two load-bearing issues: the theoretical proof contains algebraic and conceptual errors, and the headline benchmark gain may be inflated by evaluator-annotator agreement rather than reflecting true mathematical correctness on an independent signal.

major comments (4)
  1. [Section 4.2, Lemma 1] Lemma 1 states P_GT^k > P_FN^k for all k, and Theorem 1 and Lemma 4 both rely on this strict inequality (delta_k > 0 is used as the integration interval length). However, no proof is given, and the statement is not obvious: the FN-reward policy is trained with different rewards and could in principle have a higher true success probability at some step. The lemma needs a proof or a counterexample that clarifies when it holds.
  2. [Appendix D.2, Lemma 4] The integral representation of delta_k assumes, without stating it, that the FN-caused success-probability gap is equal at consecutive steps: f(delta_k) equals D(P_FN^k, P_FN^{k-1}) only if P_GT^k - P_FN^k = P_GT^{k-1} - P_FN^{k-1}. This is an unstated assumption and is not implied by Lemma 1. As written, Equation (18) is not justified, so the integral form and the resulting proof of Theorem 1 do not follow.
  3. [Appendix D.2, Lemma 5 and Theorem 1] The simplification for T(b) in Lemma 5 is algebraically incorrect. For example, with b=0.5, epsilon=0.01, and a=0.6, the claimed expression T(b)=(d(b)-d'(b)(a-b))/d(b)^2 gives about 1.96, while direct computation from W+(b)+W-(b)+aW+'(b)-(1-a)W-'(b) gives about 0.54. The subsequent positivity argument for T(b) therefore does not establish the lemma. Additionally, the statement of Theorem 1 cites only Lemma 1 and Assumption 1, but the proof of Lemma 5 and the use of the condition b<a<2b rely on Assumption 2; the theorem statement and proof are inconsistent on this point.
  4. [Section 5.3 and Table 1] The headline claim of up to 10% improvement rests on HardVerify-Math, a 250-question benchmark curated by the authors and scored with an unspecified LLM-based judge. TinyV is trained on labels produced by LLMs (QWEN2.5-72B-INSTRUCT and GROK-3-MINI-HIGH), so if the judge shares the annotators' equivalence standards, the reported gain may partly reflect agreement with those standards rather than true mathematical correctness. The external rule-based benchmarks in Table 1 show much smaller gains (roughly 1 point on MATH and Olympiad for Qwen2.5-7B), and AMC is worse with TinyV (43.37 vs 44.58). To make the claim load-bearing, the authors should specify the judge, release the benchmark, and report results on a human-verified subset or on an independent rule-based signal.
minor comments (5)
  1. [Table 1] The row label 'DeepScalaR' is inconsistent with 'DeepScaleR' used elsewhere in the paper.
  2. [Figure 2] The y-axis label 'Precentage(%)' is a typo for 'Percentage(%).'
  3. [Section 5.4, Takeaway 5] The claim that TinyV 'consistently outperforms' Prime Verifier is contradicted by the AMC result in Table 1 for Qwen2.5-7B (43.37 vs 44.58); this regression should be acknowledged and discussed.
  4. [Section 5.3] The paper says 'we instead employ LLM-based evaluations' for HardVerify-Math but does not specify which LLM is used, the prompt, or the temperature; this information is essential for reproducibility.
  5. [Section 5.1 and Appendix F] The HardVerify-Math benchmark is described and illustrated but not released with the code; since it is the basis of the headline result, the benchmark should be made available.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline HardVerify-Math gain is partly circular: TinyV is trained on LLM-annotated labels, and the benchmark where the largest gain appears is scored by an unspecified LLM judge, so part of the 10-point gap may reflect judge–annotator agreement rather than independent mathematical correctness.

  1. fitted input called prediction [Section 5.1 (Curation of TINYV), Section 5.3 (Benchmarks and Evaluation Setups), Table 1]
    "The real false negative and true negative data are sourced from Section 3, where the correctness of the responses were annotated by LLMs. For the more challenging HardVerify-Math, we instead employ LLM-based evaluations to assess performance."

    TINYV is SFT-trained to reproduce LLM-annotated correctness labels (Qwen2.5-72B-Instruct and Grok-3-Mini-High). Its headline 10-point gain on HardVerify-Math (68.68% vs 58.64% for Qwen2.5-7B) is scored by an unidentified LLM judge. Thus both the reward signal used for training and the evaluation signal are LLM equivalence judgments, so the benchmark is not an external ground truth. The gain may largely reflect agreement between policies trained on TINYV's learned LLM labels and the judge's criteria. The rule-based MATH/AMC/Olympiad results show only ~1-point gains, so the large advertised effect lacks independent corroboration.

full rationale

The paper's core derivation chain is not circular in the sense of a fitted parameter being renamed a prediction, and the theoretical claim is a conditional formal argument from stated assumptions. However, the most prominent empirical result is partially circular: TinyV is trained to predict LLM annotations, and HardVerify-Math, the benchmark showing the largest improvement, is evaluated with an LLM-based judge of unspecified identity. Without a rule-based or human ground truth for that benchmark, the 10.04-point gap could be driven by agreement between the LLM standards used in training and those used in evaluation. This does not invalidate the whole paper: MATH, AMC, and Olympiad are scored with the external Prime Verifier and show smaller but real gains, providing some independent support for the claim that reducing false negatives helps RL. The theoretical Theorem 1 is also not circular; it is a deduction from Lemma 1 and Assumptions 1-2, though Lemma 1 is asserted without proof and carries much of the substance. Overall, the central idea has independent content, but the headline magnitude rests on an evaluation loop that is not fully external, warranting a score of 4.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central applied claim depends on the accuracy of LLM labels for FN detection and on the (unreleased) HardVerify-Math evaluation. The theoretical claim depends on the GRPO closed-form dynamics and two smoothness assumptions on the success probability, plus an unstated constant-gap assumption. Hand-chosen data thresholds and sample sizes affect the empirical results but are not part of the theory.

free parameters (3)
  • Training question pass-rate thresholds (data selection) = 0.05 < p(x) <= 0.2 (Llama-3.1-8B), p(x) <= 0.25 (DeepSeek-distilled)
    Hand-chosen criteria to select 5,000 challenging questions from Big-Math; affects the training distribution and results.
  • Number of TinyV training instances = 159,000 balanced from 638,000 total
    Sampled to balance True/False labels; affects the verifier's accuracy.
  • HardVerify-Math selection thresholds = 125 Big-Math questions with pass rate < 0.05, plus 125 manually selected from Olympiad/MATH
    Hand-picked benchmark construction; directly defines the headline evaluation.
assumptions (5)
  • domain assumption GRPO policy update has the closed form given in Lemma 2 (cited to [28])
    Used as the starting point for the theoretical analysis; if the closed-form dynamics are wrong, Theorem 1 does not follow.
  • domain assumption Assumption 1: ground-truth success probability P_GT_k increases with k
    Stated in Section 4.2; required for the integrand positivity in Lemma 5.
  • domain assumption Assumption 2: P_GT_k < 2 P_GT_{k-1}
    Stated in Section 4.2; requires success probabilities not to grow exponentially.
  • ad hoc to paper The FN-caused success gap P_GT_k - P_FN_k is equal at consecutive steps k-1 and k (unstated)
    Lemma 4's integral path connects D(P_GT_k, P_GT_{k-1}) to D(P_FN_k, P_FN_{k-1}) only if the gap is constant; no justification is given.
  • domain assumption LLM annotations define the ground truth for mathematical equivalence
    The 38.5% FN rate and TinyV training labels depend on Qwen2.5-72B-Instruct and Grok-3-Mini-High judgments; only 200 samples were manually checked.
invented entities (2)
  • TinyV (fine-tuned Qwen2.5-1.5B verifier) independent evidence
    purpose: Re-evaluate rule-based verifier rejections to detect false negatives and correct rewards in RL
    A concrete model artifact; its behavior is testable on any benchmark, and the authors evaluate it on standard and custom benchmarks.
  • HardVerify-Math (250-question benchmark)
    purpose: Evaluate verifiers on hard-to-verify answers prone to false negatives
    The benchmark is curated by the authors and not released in the paper; no independent external validation or public access is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning." pith.science (2026). https://pith.science/paper/IHVWNFHO

@misc{pith2026250514625,
  author       = {Pith},
  title        = {Pith review of: TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHVWNFHO}},
  note         = {Machine review of arXiv:2505.14625}
}
read the original abstract

Reinforcement Learning (RL) has become a powerful tool for enhancing the reasoning abilities of large language models (LLMs) by optimizing their policies with reward signals. Yet, RL's success relies on the reliability of rewards, which are provided by verifiers. In this paper, we expose and analyze a widespread problem--false negatives--where verifiers wrongly reject correct model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals that over 38% of model-generated responses suffer from false negatives, where the verifier fails to recognize correct answers. We show, both empirically and theoretically, that these false negatives severely impair RL training by depriving the model of informative gradient signals and slowing convergence. To mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments existing rule-based methods, which dynamically identifies potential false negatives and recovers valid responses to produce more accurate reward estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts pass rates by up to 10% and accelerates convergence relative to the baseline. Our findings highlight the critical importance of addressing verifier false negatives and offer a practical approach to improve RL-based fine-tuning of LLMs. Our code is available at https://github.com/uw-nsl/TinyV.

Figures

Figures reproduced from arXiv: 2505.14625 by the authors.

Figure 1
Figure 1. This figure illustrates a false negative case [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. This figure demonstrates false negatives in Big-Math-RL-Verified by source (upper) and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. This figure demonstrates the impact of FNs on training efficiency by comparing [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (15 more)
Figure 3
Figure 3. Figure 3: The fraction of unique prompts in the training dataset that encounter at least one false-negative rollout across steps. The x-axis rep￾resents the training step, and the y￾axis shows the cumulative fraction of prompts affected by false negatives. Intuitively, prompts f…
Figure 5
Figure 5. Figure 5: This figure demonstrates the curation and deployment of TINYV. In this subsection, we outline the process for creating TINYV , focusing on dataset curation, model training, and deployment setup. Dataset Curation. To develop a reliable veri￾fier capable of handling dive…
Figure 6
Figure 6. Figure 6: Performance trends of Qwen2.5-7B on the AMC, MATH and Olympiad benchmark, com￾paring TINYV with Prime Verifier. The darker lines are smoothed using a sliding window whose size is 5% of the total training steps. We observe that model trained with TINYV converges faster …
Figure 7
Figure 7. Figure 7: This figure compares perfor￾mance of HardVerify-Math Bench between Big-Math (hard to verify) and DeepScaleR (easy to verify) datasets. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Prompt Template for TINYV Training and Inference. Example 1 (Olympiad Benchmark) Question: Determine all real numbers $x>0$ for which\n\n$$\n\\log _{4} x-\\ log _{x} 16=\\frac{7}{6}-\\log _{x} 8\n$$ Ground Truth: $2^{-2 / 3}$, $8$ Model Output: 8, \\frac{1}{\\sqrt[3]{4…
Figure 9
Figure 9. Figure 9: HardVerify-Math Bench Examples. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: This figure shows the source distribution of [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: This figure compares the model performance of T [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: This figures compares the average time cost of T [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Prompt Template for Labeling FN Responses (Part 1) [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Prompt Template for Labeling FN Responses (Part 2) [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Prompt Template for Labeling FN Categories (Part 1) [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Prompt Template for Labeling FN Categories (Part 2) [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: Prompt Template for Labeling FN Categories (Part 3) [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]
Figure 18
Figure 18. Figure 18: Prompt Template for Generating Synthetic FN Examples [PITH_FULL_IMAGE:figures/full_fig_p033_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

    cs.AI 2026-08 conditional novelty 6.0 of 10

    LLMs under one-shot greedy decoding enumerate or materialize acceptable sets much worse than they judge membership, a gap that persists across scale, family, and generation and is dominated by omissions.

  2. Trading Human Curation for Synthetic Augmentation in RLVR

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Gated synthetic augmentations of a 10-task human base substitute for ~87 extra human RLVR tasks on aggregate held-out pass@1, with cost-adjusted trade rate ρ_cost in [1.4×, 11.6×].

Reference graph

Works this paper leans on

48 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    2025

  2. [2]

    Online difficulty filtering for reasoning oriented reinforcement learning.arXiv preprint arXiv:2504.03380, 2025

    Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning.arXiv preprint arXiv:2504.03380, 2025

    arXiv 2025

  3. [3]

    Robotxr1: Enabling embodied robotic intelligence on large language models through closed-loop reinforcement learning, 2025

    Liam Boyle, Nicolas Baumann, Paviththiren Sivasothilingam, Michele Magno, and Luca Benini. Robotxr1: Enabling embodied robotic intelligence on large language models through closed-loop reinforcement learning, 2025

    work page 2025

  4. [4]

    xverify: Efficient answer verifier for reasoning model evaluations, 2025

    Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evaluations, 2025

    work page 2025

  5. [5]

    Towards reasoning era: A survey of long chain- of-thought for reasoning large language models.arXiv preprint arXiv:2503.09567, 2025

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain- of-thought for reasoning large language models.arXiv preprint arXiv:2503.09567, 2025

    arXiv 2025

  6. [6]

    Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    2023

  7. [7]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025

    arXiv 2025

  8. [8]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    work page 2025

Show all 48 references
  1. [9]

    Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023

    2023

  2. [10]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  3. [11]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  4. [12]

    A survey on llm-as-a-judge, 2025

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025

    2025

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.1...

    2024 arXiv

  7. [15]

    Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms, 2024

    Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms, 2024

    2024

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  9. [17]

    Putting rl back in rlhf.https://huggingface

    Shengyi Costa Huang and Arash Ahmadian. Putting rl back in rlhf.https://huggingface. co/blog/putting_rl_back_in_rlhf_with_rloo, June 12 2024. Hugging Face Blog

    2024

  10. [18]

    Math-Verify: A robust mathematical expression evaluation system.https: //github.com/huggingface/Math-Verify, 2025

    Hugging Face. Math-Verify: A robust mathematical expression evaluation system.https: //github.com/huggingface/Math-Verify, 2025. Accessed: 2025-05-15

    2025

  11. [19]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  12. [20]

    Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025. 11

    2025

  13. [21]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  14. [22]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2025

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2025

    2025

  15. [23]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  16. [24]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction- following models.https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  17. [25]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024

    2024

  18. [26]

    Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl.Notion Blog, 2025

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl.Notion Blog, 2025

    2025

  19. [27]

    General- reasoner: Advancing llm reasoning across all domains, 2025

    Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General- reasoner: Advancing llm reasoning across all domains, 2025

    2025

  20. [28]

    Reinforcement learning with verifiable rewards: Grpo’s effective loss, dy- namics, and success amplification, 2025

    Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo’s effective loss, dy- namics, and success amplification, 2025

    2025

  21. [29]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    2025

  22. [30]

    OpenAI Evals: A framework for evaluating llms.https://github.com/openai/ evals, 2025

    OpenAI. OpenAI Evals: A framework for evaluating llms.https://github.com/openai/ evals, 2025. Accessed: 2025-05-15

    2025

  23. [31]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024

    2024

  24. [32]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  25. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM, March 2025

    2025

  27. [35]

    Swe-rl: Advancing llm reason- ing via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reason- ing via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    2025 arXiv

  28. [36]

    Deepseek-prover: Advancing theorem proving in llms through large- scale synthetic data.arXiv preprint arXiv:2405.14333, 2024

    Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large- scale synthetic data.arXiv preprint arXiv:2405.14333, 2024. 12

    2024 arXiv

  29. [37]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to llm reasoning: from rejection sampling to reinforce, 2025

    2025

  30. [38]

    Not all rollouts are useful: Down- sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818, 2025

    Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in llm reinforcement learning.arXiv preprint arXiv:2504.13818, 2025

    2025 arXiv

  31. [39]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024

  32. [40]

    Limo: Less is more for reasoning, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025

    2025

  33. [41]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yux- uan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi ...

    2025

  34. [42]

    Vapo: Efficient and reliable rein- forcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable rein- forcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    2025 arXiv

  35. [43]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    2025

  36. [44]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  37. [45]

    LLM as a judge

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  38. [46]

    Is the model’s answer mathematically equivalent to the ground truth?

  39. [47]

    Are there minor formatting differences that don’t affect the answer’s correctness?

  40. [48]

    reasoning

    Is the model’s answer more precise or in a different but valid format? ## Examples Here are some examples of questions, ground truth answers, and model answers. All of them are correct. **Example 1 (Order-Insensitive):** <question>Determine all real values of $x$ for which $(x...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.