Pith. sign in

REVIEW 3 major objections 6 minor 12 cited by

General-Reasoner: Advancing LLM Reasoning Across All Domains

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Direct reinforcement learning on base language models can generalize beyond math and code when the reward comes from a learned answer-verifier rather than exact-match rules.

desk verdict Plausible recipe for general-domain zero-RL with a compact verifier, but the verifier's reward signal is validated only against its own teacher and one key number is inconsistent. read the letter →

arxiv 2505.14652 v5 pith:JNGMXH25 submitted 2025-05-20 cs.CL

Xueguang Ma , Qian Liu , Dongfu Jiang , Ge Zhang , Zejun Ma , Wenhu Chen This is my paper · ORCID
classification cs.CL
keywords reinforcementlearningzero-RLGRPOgenerativeverifieranswerverificationgeneralreasoningverifiablequestionsLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In the zero-RL setting, reinforcement learning is applied directly to a base model without supervised fine-tuning; this had mostly worked for math and code because those domains have abundant verifiable data and easy rule-based checks. General-Reasoner tries to show the same recipe can be lifted to all domains by mining about 230,000 web questions with verifiable short answers across physics, chemistry, finance, and other fields, and by replacing rule-based checks with a compact generative verifier that judges answer equivalence. The authors report that models trained this way gain roughly ten points on general benchmarks like MMLU-Pro and SuperGPQA over their base models, keep or improve math performance, and let their best 14-billion-parameter model match or beat a frontier commercial model on GPQA and TheoremQA. If the claim holds, a single RL pass over diverse verifiable questions, with no supervised fine-tuning and no math-specific reward engineering, is enough to produce broad reasoning ability.

What carries the argument

The load-bearing component is General-Verifier, a 1.5-billion-parameter generative answer verifier: given a question, a ground-truth answer, and a student answer, it produces a short chain of thought and then a final yes-or-no decision on equivalence, replacing exact-match or rule-based reward computation. It is trained on the labels and candidate solutions produced during dataset construction and initialized from a 1.5B math-oriented base model. Around it sit the WebInstruct-verified dataset of roughly 230,000 diverse verifiable questions and the GRPO objective, which uses the verifier's binary reward plus a length penalty to update the policy.

What would settle it

Take a random sample of one thousand student answers from non-math domains in WebInstruct-verified, have domain experts label each as correct or not, and compare General-Verifier's decisions against those labels. If the verifier's agreement with experts is close to chance on unstructured answer types, the reward signal is unreliable and the reported benchmark gains cannot be attributed to improved reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that zero-RL reasoning is not confined to mathematics and coding once two bottlenecks are removed: the shortage of verifiable question-answer data outside math, and the rigidity of rule-based answer checking. The authors remove the first by curating approximately 230,000 web-derived questions whose short answers can be verified, and the second with General-Verifier, a compact generative model that decides answer equivalence through chain-of-thought. Trained with GRPO on this combination, models initialized from base checkpoints of 4B, 7B, and 14B parameters show consistent gains across twelve benchmarks; the best 14B model reaches 56.1 percent on GPQA and 54.4 percent on TheoremQA, compared with 50.0 and 43.6 percent for a frontier commercial model, while staying competitive on math.

Load-bearing premise

The whole method depends on the small verifier being right about whether a student answer is genuinely equivalent to the ground truth; if it is wrong on out-of-distribution answers, the reinforcement learning optimizes the wrong reward and the reported gains do not represent better reasoning.

Editorial extensions

If this is right

  • If correct, zero-RL can serve as a general post-training stage for base models, not just a math-and-code trick.
  • A compact generative verifier can substitute for hand-written rule-based checkers, making RL feasible in any domain with short, verifiable answers.
  • Training on diverse domains does not sacrifice math: full-data models match or exceed math-only RL baselines while gaining on general benchmarks.
  • The best 14B model matches or beats a frontier commercial instruction-tuned model on GPQA, TheoremQA, and MMLU-Pro with no instruction tuning and no distillation.
  • The short response lengths, around one thousand tokens, indicate that broad reasoning gains do not require extremely long chain-of-thought rollouts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its verifier is only validated against the same model family that produced its labels and dataset filtering; a human-annotated out-of-distribution verifier test is the natural next check.
  • The dataset filtering step, which keeps only questions where eight candidate solutions neither all fail nor all succeed, likely concentrates training on questions a frontier model can partially solve, possibly narrowing the distribution the final model sees.
  • If the verifier is the true bottleneck, iteratively retraining it on human-confirmed judgments is a plausible scaling path, though it risks self-confirmation.
  • Because the reward includes a length penalty and final models stay near one thousand tokens, the result hints that overthinking is not a necessary byproduct of verifier-based RL; an explicit ablation on the penalty would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces General-Reasoner, a zero-RL training recipe that applies GRPO directly to base Qwen models on a 230k web-crawled, verifiable QA dataset spanning domains beyond math and code. Rewards are supplied by a compact 1.5B generative verifier fine-tuned from Qwen2.5-Math, trained to judge answer equivalence with chain-of-thought, replacing rule-based checkers. The authors report consistent gains across four backbones on 12 benchmarks, with the best model, General-Reasoner-Qw3-14B, roughly matching or exceeding GPT-4o on GPQA and TheoremQA, and they present ablations for domain diversity and the verifier.

Significance. If the empirical claims hold, this is an important demonstration that verifiable-reward RL can generalize beyond math and code without supervised fine-tuning, and the released dataset and verifier would be useful community assets. The paper's strengths include consistent gains across four different base backbones, a direct ablation of full-data versus math-only training in Table 4, a direct verifier comparison in Table 5, and fairly detailed hyperparameters in Table 9. The main caveat is that the verifier, which is the central reward mechanism, is only validated against the same Gemini family that generated its training labels and the dataset filtering solutions, so the reward accuracy question is not yet settled.

major comments (3)
  1. [§5.3] The verifier validation in §5.3 is conditioned on Gemini's affirmative judgments: the 50k pairs are 'answer-verification pairs that Gemini deemed correct,' and the reported 78.7% agreement therefore measures how often the verifier accepts answers that Gemini accepts, which is effectively recall on Gemini's positive set. It does not measure precision, i.e., how often the verifier rejects genuinely wrong answers, and it does not resolve the circularity that the verifier was fine-tuned on Gemini-2.0-Flash labels while the training data were filtered by Gemini-generated candidate solutions. Because the GRPO reward in Eqs. (1)-(3) is driven by this verifier, an over-accepting verifier would reinforce incorrect outputs and the external-benchmark gains could reflect format or surface adaptation rather than reasoning improvement. Please add a false-positive analysis on a human-annotated or independently labeled sample (including Gemini-negative pairs), and report agreement in both directions.
  2. [Tables 2 and 4] The same model, General-Reasoner-7B, is reported with 38.8 GPQA-D in Table 2 and 34.3 GPQA in Table 4, while the MMLU-Pro (58.9) and SuperGPQA (34.2) values match exactly; the Math-Related aggregate also differs (45.3 in Table 2 versus 48.5 in Table 4). Since Table 4 is the direct evidence for the data-diversity claim, this inconsistency is load-bearing. Please explain the evaluation protocol used for each table and report the numbers under a single consistent protocol.
  3. [§5.2 / Table 5] The verifier-versus-rule-based ablation in Table 5 is performed for only 120 steps on a single 4B model with no repeated runs or variance estimates; the reported differences, such as 60.1 versus 58.1 on MMLU-Pro and 39.4 versus 37.9 on GPQA, may be within run-to-run noise for this setting. Given that the verifier is the central methodological contribution, please report multiple seeds or confidence intervals, or explicitly frame these differences as preliminary rather than as conclusive evidence.
minor comments (6)
  1. [§2.2] The text cites 'Deepseek-R1-Zero [4]', but the DeepSeek-R1 reference is [1] in the bibliography; please fix the citation.
  2. [§4.3] There is a typo: 'Nemotraon-CrossThink' should be 'Nemotron-CrossThink', and 'the closet to our general-purpose reasoning model' should be 'the closest'.
  3. [§4.1] The phrase 'model initialzied with Qwen3-Base' contains a typo; it should read 'model initialized'.
  4. [§5.3] The sentence 'we study their agreement with Gemini-2.0-Flash' and the later 'the our model-based verifier achieved 78.7%' contain wording issues; 'the our' should be 'our'.
  5. [Table 2] In the 14B models block, 'Qwen3-14B-Instruct (non-think)Qwen3-14B-Base' is missing a space between the model name and the backbone; please fix the formatting.
  6. [Appendix A.3] The first sentence, 'We provide detailed evaluation results on MMLU-pro, SuperGPQA and BBEH are listed in below', is grammatically incomplete; please rephrase.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial circularity: the General-Verifier's reward accuracy is validated only by agreement with Gemini-2.0-Flash, the same system that generated the verifier's training labels and filtered the 230k dataset; the final benchmark results are external, so the central claim is not fully circular.

  1. other [Section 3.1, Section 3.2, Section 5.3 (Figure 5)]
    "we leverage our dataset creation pipeline, specifically Gemini-2.0-generated candidate answers and verification annotations, to train a compact 1.5B-parameter generative verifier model. ... In a sample of 50k answer-verification pairs that Gemini deemed correct, the rule-based verifier achieved only 22.2% agreement on average, whereas our model-based verifier achieved 78.7%."

    The verifier's accuracy is the load-bearing premise for using its output as the GRPO reward in Eq. (1)-(3). The only validation offered for that accuracy is agreement with Gemini-2.0-Flash, which is the same system that produced the verifier's training labels and the eight candidate solutions used to filter the 230k dataset. Agreement with the label source measures the verifier's fit to that source, not correctness against independent ground truth. Moreover, the 50k pairs are conditioned on 'Gemini deemed correct', so the comparison excludes false positives, which are exactly the errors that would put wrong answers into the RL reward. Thus the claimed 'robust and reliable reward signals' reduce to self-consistency with the model that defined the labels, not independent verification.

full rationale

The headline empirical claim—that GRPO-trained General-Reasoner models outperform baselines on external benchmarks such as MMLU-Pro, GPQA, and TheoremQA—is not circular: those evaluations are independent of the training data and verifier, and the comparisons are against public baselines. However, the mechanism that is claimed to make the method work is the General-Verifier reward signal in Eq. (1)-(3), and the paper's only direct validation of that reward signal is Section 5.3: agreement with Gemini-2.0-Flash on answer-verification pairs that Gemini itself had marked correct. Since the verifier was fine-tuned on Gemini-2.0-generated labels and the 230k dataset was filtered using Gemini-2.0-Flash's eight candidate solutions, the validation oracle and the supervision source are the same system. This is a closed validation loop: it measures self-consistency with the label generator, not correctness, and it does not measure false positives, which are precisely the errors that would corrupt GRPO rewards. The final downstream benchmark gains provide indirect evidence, but they do not isolate the verifier's reward accuracy from other factors such as data diversity or the length penalty. I therefore assign a score of 5: partial circularity in the load-bearing verifier-validation chain, while the central benchmark comparisons remain externally grounded.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of the verifier, which is trained on Gemini-generated annotations and validated only against Gemini agreement. The dataset curation also assumes Gemini-2.0-Flash solutions are informative for filtering, and assumes WebInstruct human answers are correct. No independent human verification of the verifier or the dataset is reported, and no deduplication against evaluation benchmarks is described.

free parameters (2)
  • Reward shaping constants = -0.5 for extraction failure; -0.05 * min(10, |len_ground_truth - len_answer|) length penalty
    Section 4.1 defines the GRPO reward with these hand-picked constants; no sensitivity analysis is provided, yet the training results depend on them.
  • Dataset filtering thresholds = exclude questions where all 8 Gemini solutions fail or all 8 succeed
    Section 3.1 uses these hand-chosen criteria to keep approximately 230k of the original WebInstruct entries; changing these thresholds changes the training distribution.
assumptions (4)
  • domain assumption Human-written answers on original web pages are correct and verifiable.
    Section 3.1 relies on this when retaining entries with human answers.
  • domain assumption Gemini-2.0-Flash candidate solutions and verification labels are accurate enough to filter data and train the verifier.
    Sections 3.1 and 3.2 use Gemini outputs as the ground truth for dataset filtering and verifier training.
  • domain assumption Evaluation benchmarks are not contaminated by WebInstruct-verified training data.
    Never stated; no deduplication is described in Sections 3.1 or 4.2.
  • domain assumption A 1.5B verifier initialized from Qwen2.5-Math-1.5B can generalize to student answers outside its training distribution.
    Section 3.2 assumes this; the only reported evidence is agreement with Gemini-2.0-Flash.

how reviews work

0 comments
Cite this review

Pith. "Pith review of General-Reasoner: Advancing LLM Reasoning Across All Domains." pith.science (2026). https://pith.science/paper/JNGMXH25

@misc{pith2026250514652,
  author       = {Pith},
  title        = {Pith review of: General-Reasoner: Advancing LLM Reasoning Across All Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNGMXH25}},
  note         = {Machine review of arXiv:2505.14652}
}
read the original abstract

Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs). Particularly, the "Zero" reinforcement learning introduced by Deepseek-R1-Zero, enables direct RL training of base LLMs without relying on an intermediate supervised fine-tuning stage. Despite these advancements, current works for LLM reasoning mainly focus on mathematical and coding domains, largely due to data abundance and the ease of answer verification. This limits the applicability and generalization of such models to broader domains, where questions often have diverse answer representations, and data is more scarce. In this paper, we propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains. Our key contributions include: (1) constructing a large-scale, high-quality dataset of questions with verifiable answers curated by web crawling, covering a wide range of disciplines; and (2) developing a generative model-based answer verifier, which replaces traditional rule-based verification with the capability of chain-of-thought and context-awareness. We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc. Our comprehensive evaluation across these 12 benchmarks (e.g. MMLU-Pro, GPQA, SuperGPQA, TheoremQA, BBEH and MATH AMC) demonstrates that General-Reasoner outperforms existing baseline methods, achieving robust and generalizable reasoning performance while maintaining superior effectiveness in mathematical reasoning tasks.

Figures

Figures reproduced from arXiv: 2505.14652 by the authors.

Figure 1
Figure 1. Effectiveness of our General-Reasoner trained with diverse verifiable reasoning questions using model-based verifier compared to baseline methods on various reasoning tasks. ∗ The first two authors have equal contribution. Technical Report. Work in progress. arXiv:2505.14652v5 [cs.CL] 9 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Data creation pipeline: It consists of QA mining, Extraction and Verification. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. MMLU-Pro evaluation score at differ￾ent training step using model-based verifier and rule-based verifier [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Agreement comparison between rule-based and model-based verifiers against Gemini-2.0- [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ArchEval: Measuring AI Agents as Computer Architects

    cs.AR 2026-07 conditional novelty 7.0 of 10

    LLM agents beat architecture baselines with full simulator harnesses, but only one configuration stays above baseline without feedback, and performance modeling remains weak.

  2. Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Vocabulary dropout prevents diversity collapse in LLM co-evolution by masking proposer logits, yielding average +4.4 point solver gains on mathematical reasoning benchmarks at 8B scale.

  3. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning

    cs.LG 2026-02 reject novelty 6.0 of 10

    GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.

  4. Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancement

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Amplifying neurons that encode bias awareness, found with 45 yes/no questions, reduces gender/race/religion bias in three LLMs while roughly preserving general reasoning.

  5. Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Reshaping GRPO advantages by per-token influence on the model's final answer improves math reasoning by about 2 average points over a strong GRPO baseline.

  6. VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains

    cs.AI 2025-07 conditional novelty 6.0 of 10

    VerifyBench systematically compares specialized and general LLM verifiers across 3,989 math, physics, chemistry, and biology questions, finding an accuracy-recall tradeoff and strong sensitivity to input structure.

  7. One Token to Fool LLM-as-a-Judge

    cs.LG 2025-07 conditional novelty 6.0 of 10

    LLM reward models falsely accept empty 'master key' responses such as ':' or 'Thought process:' across many models, and a fine-tuning defense reduces these false positives to near zero.

  8. Reinforcing General Reasoning without Verifiers

    cs.LG 2025-05 conditional novelty 6.0 of 10

    VeriFree trains LLMs with RL by maximizing the likelihood of the reference answer after generated reasoning, matching verifier-based RL without any verifier.

  9. Coupled Variational Reinforcement Learning for Language Model General Reasoning

    cs.CL 2025-12 conditional novelty 5.0 of 10

    CoVRL trains an LLM on a mixture of question-only and answer-guided reasoning traces, using the model's own answer probability as reward, and reports consistent gains on math and general-reasoning benchmarks.

  10. Hermes 4 Technical Report

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Hermes 4 releases three open-weight reasoning models (14B, 70B, 405B) trained with synthetic data and a length-control SFT stage, evaluated on mathematics, code, knowledge, and alignment benchmarks.

  11. Reinforcement Pre-Training

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Reinforcement pre-training (RPT) trains an LLM with a verifiable reward for correct next-token prediction after generating a chain of thought, improving token accuracy and downstream reasoning over its base model.

  12. Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem

    cs.CL 2025-06 conditional novelty 5.0 of 10

    One-shot critique fine-tuning, training on critiques of candidate solutions to a single problem, yields large reasoning gains on math and logic benchmarks at far lower compute than one-shot RL.

Reference graph

Works this paper leans on

34 extracted references · 5 canonical work pages · cited by 12 Pith papers

  1. [1]

    DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI Team. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2501.12948

    arXiv 2025

  2. [2]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URLhttps://arxiv.org/abs/2503.18892

    arXiv 2025

  3. [3]

    DAPO: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    arXiv 2025

  4. [4]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

    2025

  5. [5]

    Deepcoder: A fully open-source 14b coder at o3-mini level, 2025

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025. Notion Blog

    2025

  6. [6]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    arXiv 2024

  7. [7]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URLhttps://arxiv.org/abs/2501.19393

    arXiv 2025

  8. [8]

    MMLU-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Processin...

    work page 2024

Show all 34 references
  1. [9]

    MAmmoTH2: Scaling instructions from the web

    Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. MAmmoTH2: Scaling instructions from the web. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=yVu5dnPlqA

    2024

  2. [10]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark, 2023. URLhttps://arxiv.org/abs/2311.12022

    2023 arXiv

  3. [11]

    SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines, 2025

    M-A-P. SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines, 2025. URL https://arxiv.org/abs/2502.14739. 10

    2025 arXiv

  4. [12]

    TheoremQA: A theorem-driven question answering dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023 doi

  5. [13]

    Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q

    Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Q...

    2025 arXiv

  6. [14]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (...

    2021

  7. [15]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/ abs/2110.14168

    2021 arXiv

  8. [16]

    OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scienti...

    2024 arXiv

  9. [17]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URLhttps://arxiv.org/abs/2201.11903

    2023 arXiv

  10. [18]

    A survey on test-time scaling in large language models: What, how, where, and how well?, 2025

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025. URL https://arxiv.org/...

    2025 arXiv

  11. [19]

    Efficient test-time scaling via self-calibration, 2025

    Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration, 2025. URLhttps://arxiv.org/abs/2503.00031

    2025 arXiv

  12. [20]

    Openai o1 system card, 2024

    OpenAI Team. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    2024 arXiv

  13. [21]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  14. [22]

    QwQ-32B: Embracing the power of reinforcement learning, March 2025

    Qwen Team. QwQ-32B: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  15. [23]

    Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining.arXiv:2505.07608, 2025

    Xiaomi LLM-Core Team. Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining.arXiv:2505.07608, 2025. URL https://arxiv.org/abs/2505. 07608

    2025 arXiv

  16. [24]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  17. [25]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  18. [26]

    Nemotron-CrossThink: Scaling self-learning beyond math reasoning, 2025

    Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-CrossThink: Scaling self-learning beyond math reasoning, 2025. URL https://arxiv.org/abs...

    2025

  19. [27]

    Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains, 2025

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains, 2025. URLhttps://arxiv.org/abs/2503.23829

    2025 arXiv

  20. [28]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv. org/abs/2312.11805

    2024 arXiv

  21. [29]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/abs/2210.09261

    2022 arXiv

  22. [30]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with l...

    2022

  23. [31]

    GPT-4o system card, 2024

    OpenAI. GPT-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

    2024 arXiv

  24. [32]

    Final Decision: Yes

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URLhttps://arxiv.org/abs/2503.24290. 12 A Appendix A.1 Limitation This work focuses o...

    2025 arXiv

  25. [33]

    Ground Truth Answer: 1 y =−x 2 +Cx

  26. [34]

    Final Decision: Yes A.5 Detailed Hyper-Parameters We provide the detailed hyperparameters for training our General-Reasoner variants in Table 9

    Student Answer: y= 1 Cx−x 2 We can rewrite the ground truth answer as: 1 y =Cx−x 2 y= 1 Cx−x 2 Comparing this with the student’s answer, we see that they are the same. Final Decision: Yes A.5 Detailed Hyper-Parameters We provide the detailed hyperparameters for training our Ge...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.