{"id":"11466756-fe69-4493-9a97-1dbd9e401006","arxiv_id":"2505.14625","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Rule-based verifiers wrongly reject over 38% of correct math answers, and the lightweight LLM verifier TinyV recovers enough of these to speed up RL training and lift pass rates by up to 10% on hard-to-verify problems.","lead":"This paper measures how often rule-based math verifiers reject correct answers (over 38% of rejected responses in a large math dataset) and shows these false negatives slow reinforcement learning for LLM reasoning. It then builds TinyV, a small language model verifier that rechecks rejected answers, and reports faster convergence and improved pass rates on math benchmarks, most notably on hard-to-verify questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline HardVerify-Math gain may be inflated by evaluator–annotator agreement: TinyV is trained on LLM labels and the benchmark is scored by an unspecified LLM judge, so the 10-point gap may reflect bias agreement rather than true mathematical correctness.","rationale":"The reader's weakest assumption identifies exactly the load-bearing risk: the largest reported gain is on a self-curated, LLM-judged benchmark with no released data, and TinyV is trained on LLM labels. This is not a matter of outside-consensus disagreement but of internal circularity: the training signal and the evaluation metric may share a systematic bias. The independent benchmarks (MATH, AMC, Olympiad), which use the external Prime Verifier, show only modest and sometimes negative changes, so they do not support a 10-point real-world improvement. The theoretical proof in Appendix D is also not valid as written—Lemma 1 is asserted without proof and effectively assumes the conclusion—but it is secondary to the empirical claim. I agree with the reader's CONDITIONAL verdict: the concern is addressable by releasing the benchmark and judge prompts and by adding an independent rule-based or human evaluation of HardVerify-Math. My stress-test does not move the verdict, so it is UNCHANGED. I credit the paper for the 99.5% manual review accuracy on the FN annotation sample, the sizable 638K-instance training set, and the public code, which make the central mechanism plausible even if the headline magnitude is not yet established.","tokens_in":26408,"tokens_out":3701,"duration_ms":34473,"concrete_test":"Re-annotate all 250 HardVerify-Math responses from the TINYV and Prime Verifier trained models using a rule-based verifier with symbolic equivalence (e.g., Math-Verify) or human experts blind to the training condition, then recompute Table 1's HardVerify-Math column. If the TINYV–Prime Verifier difference drops from ~10 points to ~1–2 points, the headline improvement is an artifact of LLM-judge–annotator agreement; if the gap persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that correcting false negatives improves RL for LLM reasoning—rests heavily on Table 1's HardVerify-Math column (68.68% vs 58.64% for Qwen2.5-7B). This benchmark is curated by the authors (Section 5.2, only 250 questions) and scored with an unspecified 'LLM-based evaluation' (Section 5.3). TinyV is trained via SFT on labels produced by QWEN2.5-72B-INSTRUCT and GROK-3-MINI-HIGH (Sections 3 and 5.1). If the evaluation judge is an LLM with the same biases as the annotators, TinyV is effectively rewarded for matching those biases rather than for mathematical equivalence. The independent, rule-based benchmarks (MATH, AMC, Olympiad) show only ~1-point gains and AMC is actually worse for Qwen2.5-7B (43.37 vs 44.58), so the real-world effect size is uncertain. A 10-point gap on 250 items is ~25 questions, with no error bars and no released benchmark to check. The theoretical argument (Theorem 1) does not rescue this: Lemma 1 (P_GT^k > P_FN^k for all k) is asserted without proof and essentially assumes the conclusion, while the proof relies on restrictive Assumptions 1–2. The evaluation circularity is therefore the load-bearing weakness: without an independent correctness signal, the headline improvement cannot be distinguished from annotator–judge agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that rule-based verifiers used in RL for LLM math reasoning produce many false negatives (FNs)—correct responses marked incorrect—and that these FNs degrade policy learning. The authors quantify FNs in Big-Math-RL-Verified (38.5% of seemingly incorrect responses with a boxed answer are actually correct), analyze FN causes, present an empirical study of FN effects during GRPO training, propose a lightweight verifier called TinyV that augments Prime Verifier in an add-on mode, and introduce a 250-question HardVerify-Math benchmark. They report that TinyV improves final pass rates by up to 10 points on HardVerify-Math and yields faster convergence, with only ~6% training overhead. A theoretical claim (Theorem 1) is offered to show that false negatives reduce step-wise learnability measured by reverse KL divergence.","tokens_in":26818,"tokens_out":4109,"duration_ms":37798,"significance":"If the empirical and theoretical claims hold, the paper addresses a practically important and under-studied failure mode: rule-based verifiers can reject mathematically correct responses, and correcting those false negatives can improve RL training for reasoning. The empirical FN measurement is a genuine strength: two independent LLM annotators were used, a manual review of 200 FN cases reported 99.5% agreement, and the RL experiments use standard GRPO with consistent trends. The proposed TinyV is lightweight (1.5B parameters, 6% overhead), and the code is released. The main significance is therefore conditional on resolving two load-bearing issues: the theoretical proof contains algebraic and conceptual errors, and the headline benchmark gain may be inflated by evaluator-annotator agreement rather than reflecting true mathematical correctness on an independent signal.","major_comments":[{"comment":"Lemma 1 states P_GT^k > P_FN^k for all k, and Theorem 1 and Lemma 4 both rely on this strict inequality (delta_k > 0 is used as the integration interval length). However, no proof is given, and the statement is not obvious: the FN-reward policy is trained with different rewards and could in principle have a higher true success probability at some step. The lemma needs a proof or a counterexample that clarifies when it holds.","section":"Section 4.2, Lemma 1"},{"comment":"The integral representation of delta_k assumes, without stating it, that the FN-caused success-probability gap is equal at consecutive steps: f(delta_k) equals D(P_FN^k, P_FN^{k-1}) only if P_GT^k - P_FN^k = P_GT^{k-1} - P_FN^{k-1}. This is an unstated assumption and is not implied by Lemma 1. As written, Equation (18) is not justified, so the integral form and the resulting proof of Theorem 1 do not follow.","section":"Appendix D.2, Lemma 4"},{"comment":"The simplification for T(b) in Lemma 5 is algebraically incorrect. For example, with b=0.5, epsilon=0.01, and a=0.6, the claimed expression T(b)=(d(b)-d'(b)(a-b))/d(b)^2 gives about 1.96, while direct computation from W+(b)+W-(b)+aW+'(b)-(1-a)W-'(b) gives about 0.54. The subsequent positivity argument for T(b) therefore does not establish the lemma. Additionally, the statement of Theorem 1 cites only Lemma 1 and Assumption 1, but the proof of Lemma 5 and the use of the condition b<a<2b rely on Assumption 2; the theorem statement and proof are inconsistent on this point.","section":"Appendix D.2, Lemma 5 and Theorem 1"},{"comment":"The headline claim of up to 10% improvement rests on HardVerify-Math, a 250-question benchmark curated by the authors and scored with an unspecified LLM-based judge. TinyV is trained on labels produced by LLMs (QWEN2.5-72B-INSTRUCT and GROK-3-MINI-HIGH), so if the judge shares the annotators' equivalence standards, the reported gain may partly reflect agreement with those standards rather than true mathematical correctness. The external rule-based benchmarks in Table 1 show much smaller gains (roughly 1 point on MATH and Olympiad for Qwen2.5-7B), and AMC is worse with TinyV (43.37 vs 44.58). To make the claim load-bearing, the authors should specify the judge, release the benchmark, and report results on a human-verified subset or on an independent rule-based signal.","section":"Section 5.3 and Table 1"}],"minor_comments":[{"comment":"The row label 'DeepScalaR' is inconsistent with 'DeepScaleR' used elsewhere in the paper.","section":"Table 1"},{"comment":"The y-axis label 'Precentage(%)' is a typo for 'Percentage(%).'","section":"Figure 2"},{"comment":"The claim that TinyV 'consistently outperforms' Prime Verifier is contradicted by the AMC result in Table 1 for Qwen2.5-7B (43.37 vs 44.58); this regression should be acknowledged and discussed.","section":"Section 5.4, Takeaway 5"},{"comment":"The paper says 'we instead employ LLM-based evaluations' for HardVerify-Math but does not specify which LLM is used, the prompt, or the temperature; this information is essential for reproducibility.","section":"Section 5.3"},{"comment":"The HardVerify-Math benchmark is described and illustrated but not released with the code; since it is the basis of the headline result, the benchmark should be made available.","section":"Section 5.1 and Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper's core empirical finding—that rule-based verifiers produce many false negatives—appears solid and is carefully measured. The main risk is that the central theoretical argument (Theorem 1) has identifiable algebraic and conceptual gaps, and the strongest empirical result depends on an in-house benchmark evaluated by an unspecified LLM judge. These issues are fixable in a revision, but they are load-bearing and should not be glossed over."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core measurement is trustworthy: on Big-Math, Prime Verifier marks 38.5% of seemingly incorrect responses as wrong when two independent LLM annotators agree they are correct, with 99.5% agreement on manual review of 200 cases. The taxonomy of FN types is detailed and genuinely useful, and the finding that FNs accumulate during RL and depress prompt efficiency early in training is credible. TinyV, a 1.5B verifier that re-checks only negatives, is a sensible add-on with about 6% overhead. That is a real contribution to RL for math reasoning.\n\nThe soft spots are in the headline claims. The 10-point gain on HardVerify-Math is the paper's main result, but that benchmark is self-curated (250 questions) and scored with an unspecified LLM judge. TinyV is trained on LLM labels, so if the judge shares the annotators' standards, part of the gain is agreement with bias, not mathematical correctness. The independent rule-based benchmarks (MATH, AMC, Olympiad) show only about 1 point changes, with AMC slightly negative for Qwen2.5-7B. No error bars, and the benchmark and judge are not released.\n\nThe theory section is not valid as written. Lemma 1, that ground-truth-trained policies have higher success probability than FN-trained ones, is asserted without proof and essentially assumes the conclusion. Lemma 5's simplification of T(b) is algebraically wrong, so Theorem 1 does not follow. A reviewer can ignore the theory, but the authors should fix or remove it.\n\nThe citation pattern is fine, and the code is promised. The FN measurement alone is worth citing. This paper deserves a serious referee; my verdict would be conditional, with the main demands being: release HardVerify-Math and specify the judge, add error bars, and repair or cut the theory.","headline":"Solid measurement of verifier false negatives, a plausible cheap fix, and a theory section that should be ignored as written.","tokens_in":27348,"tokens_out":3214,"would_cite":true,"duration_ms":27301,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that rule-based verifiers in LLM math reasoning produce many false negatives that silently harm reinforcement learning, and shows that a small LLM verifier that corrects those false negatives improves pass rates by up to…","keywords":["false negatives","answer verification","reinforcement learning","GRPO","LLM reasoning","mathematical reasoning","reward signals","TinyV"],"falsifier":"Run the same TinyV-trained RL runs but score HardVerify-Math with a fully symbolic equivalence checker (or a second, independently trained LLM judge) and check whether the ~10-point gap over the rule-based baseline shrinks to the ~1-point level seen on MATH; if it does, the claim that TinyV improves true mathematical correctness is falsified.","tokens_in":26236,"feed_emoji":"✅","tokens_out":9667,"duration_ms":78607,"temperature":0.7,"pith_summary":"The paper's central claim is that rule-based answer verifiers—programs that decide whether a model's answer matches the ground truth—frequently reject mathematically correct answers, and that these false negatives quietly degrade reinforcement learning for LLM reasoning. In Big-Math-RL-Verified, it finds that 38.5% of the 226K prompt–response pairs that the Prime Verifier marks as incorrect are actually correct, and it shows both empirically and theoretically that this corrupts the reward signal and slows convergence. The remedy is TinyV, a 1.5B parameter LLM verifier fine-tuned on 638K labeled examples, deployed as an add-on that re-checks only the responses the rule-based verifier rejects. Using TinyV during GRPO training raises pass rates on the authors' hard-to-verify benchmark by up to 10 points over the rule-based baseline while adding only about 6% compute. The paper's broader message is that the verifier's false-negative rate, not just the policy, is a first-order bottleneck in RL for mathematical reasoning.","feed_headline":"Fixing verifier false negatives boosts math-reasoning RL 10%","feed_subtitle":"A 1.5B re-checker rescues correct answers that rule-based verifiers reject, accelerating RL training.","key_machinery":"The load-bearing object is the false-negative (FN) detector TinyV: a lightweight 1.5B LLM binary classifier that takes a question, ground-truth answer, and model answer and returns 'True' or 'False', trained by supervised fine-tuning on 638K instances built from LLM annotations plus synthetically generated equivalence-preserving variations. It is deployed as an add-on to a rule-based verifier: the rule-based verifier runs first, and TinyV is queried only when the response is marked incorrect, flipping rewards for rescued correct answers from 0 to 1 at about 6% extra compute. The theoretical side is a step-wise learnability measure, the reverse KL divergence $D_k = D_{KL}(\\pi_{k-1}\\|\\pi_k)$ between consecutive GRPO policies; Theorem 1 proves $\\delta_k = D_{k,GT} - D_{k,FN} > 0$ whenever the ground-truth success probability is higher and grows at a non-exponential rate. This mechanism explains why correcting FNs improves training: it restores informative gradient signals on hard prompts that otherwise would have none.","core_discovery":"The discovery on the paper's own terms: false negatives in verification are prevalent, harmful, and fixable. Among responses that Prime Verifier labels incorrect but that contain a boxed answer, 87K of 226K (38.5%) are actually correct, and 42.1% of the affected unique prompts contain at least one correct answer that was rejected. During RL training, the fraction of prompts experiencing at least one false-negative rollout climbs to 46.7% (Qwen2.5-7B) and 50.5% (Qwen2.5-Math-7B), and LLM-based relabeling shows a 15 percentage-point gap in the all-wrong ratio early in training, meaning Prime Verifier deprives the policy of instructive gradients on exactly the hard prompts. The theoretical result, Theorem 1, states that policies trained with ground-truth rewards have strictly greater step-wise learnability, measured as reverse KL divergence between consecutive GRPO policies, than policies trained with false-negative-corrupted rewards. TinyV corrects this by supervised fine-tuning on 638K real and synthetic false-negative/true-negative examples, and training with it lifts HardVerify-Math scores from 58.64% to 68.68% (Qwen2.5-7B) and from 62.65% to 69.08% (Qwen2.5-Math-7B) while accelerating convergence.","pith_inferences":["If false negatives corrupt rewards in math verification, the same failure mode is likely to appear in other verifiable domains the paper lists—theorem proving, code, medicine—where rule-based checks are common; the paper only demonstrates the math case.","An external evaluation should test whether the 10-point gap persists when HardVerify-Math is scored by an independent judge (e.g., a symbolic equivalence checker or a different LLM) rather than the LLM judge used in the paper; if the judge shares TinyV's training bias, part of the gain could be self-consistency with the annotator rather than true correctness.","TinyV inherits the rule-based verifier's answer-extraction (boxed-only) mechanism, so extending it to full-solution verification could recover additional false negatives but would likely cost more than the reported 6% overhead.","The theoretical result depends on the GRPO policy dynamics lemma; applying the same FN-correction idea to PPO or offline methods may show a different magnitude of benefit, though the qualitative direction should hold."],"forward_implications":["Training with TinyV rather than the rule-based verifier alone raises final pass rates on HardVerify-Math by about 10 points for Qwen2.5-7B and by about 6.4 points for Qwen2.5-Math-7B.","TinyV-trained models converge to the baseline's final performance earlier in training, so the same quality can be reached with fewer RL steps.","Models trained on easily verifiable questions (DeepScaleR-style) perform markedly worse on hard-to-verify questions than models trained on hard-to-verify data with TinyV, suggesting that the verifier's coverage determines what the policy learns.","Because most false negatives come from natural-language elements, formatting, and notation differences, a taxonomy-based understanding of equivalence is needed for scalable verification beyond string matching.","On standard benchmarks (MATH, AMC, Olympiad), the gains are roughly one point and AMC is mixed, indicating that the benefit concentrates on hard-to-verify answers."],"supporting_citations":[{"why":"Supplies the Big-Math-RL-Verified dataset whose 226K seemingly incorrect responses are analyzed for false negatives and whose 5K hard prompts are used for RL training.","marker":"[1]"},{"why":"Defines Prime Verifier, the rule-based baseline whose false negatives are the paper's object of study and the primary comparator for TinyV.","marker":"[7]"},{"why":"Provides GRPO, the RL algorithm used in training and in the theoretical step-wise learnability analysis.","marker":"[33]"},{"why":"Supplies the closed-form GRPO policy update dynamics (Lemma 2) that the proof of Theorem 1 relies on.","marker":"[28]"},{"why":"Gives the prompt-efficiency metric η_k and the DAPO-style dynamic sampling logic used to measure how false negatives cut informative gradients.","marker":"[41]"},{"why":"Provides the DeepScaleR easily-verifiable training data and zero-RL setup used as the comparison baseline.","marker":"[26]"},{"why":"Math-Verify is the second rule-based verifier shown to produce the same false-negative behavior and used as another baseline.","marker":"[18]"},{"why":"The DeepSeek-R1-Distill-Qwen-7B model generates the rollouts whose verifier false negatives are analyzed and annotated.","marker":"[13]"},{"why":"MATH500 is one of the standard benchmarks used to show that TinyV's improvements are not confined to the authors' curated hard-to-verify set.","marker":"[16]"},{"why":"Olympiad Bench provides another external evaluation set where TinyV training shows gains over the rule-based baseline.","marker":"[14]"}],"fun_headline_variants":["Verifier false negatives undercut RL; TinyV fixes them","TinyV rescues correct answers rejected by verifiers, boosting RL","38% of rejected answers are actually correct; TinyV recovers them","Fix verifier mistakes to unlock up to 10% more RL math wins","TinyV turns verifier false negatives into faster RL convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the LLM judge that scores the authors' hard-to-verify benchmark agrees with the LLM labels used to train TinyV—if they share a bias, part of the reported gain measures self-consistency rather than correctness—and the convergence theorem likewise assumes ground-truth success probabilities rise steadily without blowing up.","fun_headline_variants_meta":{"raw":{"variants":["Verifier false negatives undercut RL; TinyV fixes them","TinyV rescues correct answers rejected by verifiers, boosting RL","38% of rejected answers are actually correct; TinyV recovers them","Fix verifier mistakes to unlock up to 10% more RL math wins","TinyV turns verifier false negatives into faster RL convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2173,"prompt_tokens":1073,"completion_tokens":1100,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":1007}},"tokens_in":689,"tokens_out":1100,"duration_ms":10915,"temperature":1.0,"reasoning_tokens":1007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:32:30.855128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same TinyV-trained RL runs but score HardVerify-Math with a fully symbolic equivalence checker (or a second, independently trained LLM judge) and check whether the ~10-point gap over the rule-based baseline shrinks to the ~1-point level seen on MATH; if it does, the claim that TinyV improves true mathematical correctness is falsified.","supporting_citations":[{"cited_title":"Reinforcement learning with verifiable rewards: Grpo’s effective loss, dy- namics, and success amplification, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the closed-form GRPO policy update dynamics (Lemma 2) that the proof of Theorem 1 relies on."},{"cited_title":"Math-Verify: A robust mathematical expression evaluation system.https: //github.com/huggingface/Math-Verify, 2025","cited_arxiv_id":null,"evidence_quote":"Math-Verify is the second rule-based verifier shown to produce the same false-negative behavior and used as another baseline."}],"review_version":1}