Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that supervised fine-tuning on mathematical problem-solving with long, self-reflective reasoning chains, and rule-based reinforcement learning on math queries, generalize to broader reasoning tasks, whereas conventional…

desk verdict A useful, well-run empirical comparison of five MPS training strategies, with a real but addressable confound in the long-vs-short CoT ablation. read the letter →

arxiv 2507.04391 v1 pith:BAL3LK6O submitted 2025-07-06 cs.CL

Ruochen Zhou , Minrui Xu , Shiqi Chen , Junteng Liu , Yunqi Li , Xinxin Lin , Zhengyu Chen , Junxian He This is my paper
classification cs.CL
keywords mathematicalproblem-solvingreasoninggeneralizationlongchain-of-thoughtrule-basedreinforcementlearninginstructiontuningcontinualpretrainingself-reflectionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether teaching a large language model to solve math problems builds general reasoning skills or only raises math scores. It compares five training strategies—continual pretraining on math text, instruction tuning on short and on long chain-of-thought solutions, fine-tuning on STEM question-answer data, and rule-based reinforcement learning on math queries—across five math and eight general reasoning benchmarks. The central finding is that long, self-reflective reasoning chains and rule-based reinforcement learning on math problems transfer to other reasoning domains, while ordinary short chain-of-thought math instruction tuning usually does not and often degrades non-math performance. This matters because math is one of the few domains where large volumes of verifiable, long reasoning traces can be generated, so it could be a practical route to stronger general reasoning in language models.

What carries the argument

The carrying mechanism is the long, self-reflective chain-of-thought response—a solution trace that includes intermediate verification, self-checks, and alternative paths—used as supervised fine-tuning data, plus rule-based reinforcement learning with format and correctness rewards on math queries. The paper's behavioral evidence for the mechanism is response length: after long-CoT math training, the model produces substantially longer outputs on non-math benchmarks, and the short-answer ablation (rewriting s1.1's responses to about 419 tokens) erases the generalization gains. The rule-based RL component works by using a $\boxed{}$ answer-format constraint and ground-truth verification as rewards, which lets the model discover extended reasoning without demonstration data.

What would settle it

Train two models from the same base on the same long-CoT math solutions: one with answers truncated to short length without rewriting content, and one with the original long answers. If the truncated-length model keeps the cross-domain gains, then length is not the active ingredient; if it loses them while the original retains them, the conclusion is supported. Alternatively, a length-capped decoding evaluation of LIMO or s1.1 (max tokens set to base-model levels) that still showed the gains would contradict the 'long reasoning mode' mechanism.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is an empirical asymmetry: the way a model learns math determines whether that learning travels. Fine-tuning on a few hundred math problems solved with long, self-correcting chains—the LIMO recipe, for instance—improved Qwen2.5-32B-Instruct on every general reasoning task tested, with relative gains of 10.2% on GPQA and 11.8% on LogiQA, while also lengthening the model's responses in non-math domains. Fine-tuning on the same kind of problems with short step-by-step answers, by contrast, often lowered general reasoning scores; in the paper's control experiment, rewriting s1.1's long answers into short ones removed the cross-domain benefit. Rule-based reinforcement learning on math queries produced similar transfer, beating the base model and a math-data SFT model on most benchmarks. The paper concludes that short reasoning chains acquire math-specific behavior, whereas long, self-reflective chains activate a general 'long reasoning mode' that extends into other domains.

Load-bearing premise

The load-bearing premise is that the s1.1-short ablation changed only the length of the reasoning chain; if rewriting the long answers with GPT-4o also changed answer quality, self-correction behavior, or format, the causal link between chain length and cross-domain generalization is not established.

Editorial extensions

If this is right

  • Conventional short-CoT math instruction tuning should not be assumed to strengthen general reasoning; math-benchmark gains can coexist with losses elsewhere.
  • Long-CoT math fine-tuning can transfer across domains with very few examples, suggesting that data quality and format may matter more than scale.
  • Rule-based RL on math queries is a viable alternative path to general reasoning improvement, and an SFT warmup before RL helps.
  • Continual pretraining on math text gives modest broad improvements, but they are less consistent than those from long-CoT SFT.
  • Because diverse non-math SFT datasets failed to generalize broadly, improving general reasoning may require new training objectives rather than more instruction-tuning data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If long-chain length is the real driver, then methods that extend inference-time budgets or induce self-correction in any domain could reproduce the transfer without math-specific data; a length-capped evaluation would test this.
  • The paper's mechanism suggests the math domain's value is its verifiable long CoT traces, not math content per se; training on long CoT in other verifiable domains, such as code execution, should transfer similarly if the hypothesis is right.
  • The GPT-4o rewriting ablation leaves open that content quality, not length, caused the drop; a cleaner test would compress long answers by removing only self-correction and verification phrases while preserving content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports an empirical study of whether training LLMs on mathematical problem-solving (MPS) transfers to general reasoning tasks. The authors compare five strategies: continual pretraining on math text (DeepSeekMath, RhO-Math), SFT on STEM QA (MAmmoTH2), SFT on short-CoT math data (Math-CoT, Math-PoT), SFT on long self-reflective CoT (LIMO, s1.1), and rule-based RL on math queries (SimpleRL family). They evaluate on five MPS and eight general-reasoning benchmarks. They report that continual pretraining generalizes modestly, short-CoT SFT generalizes poorly and often hurts, while long-CoT SFT and rule-based RL on math queries improve general reasoning, with gains of 10–12% relative on GPQA and LogiQA for LIMO. A preliminary study of non-MPS SFT datasets (Magicoder, Magpie, OpenOrca) finds only localized gains.

Significance. The question addressed here is timely and practically important given the current emphasis on long-reasoning models and the widespread use of math SFT in post-training. The paper's strengths include the breadth of the evaluation (13 benchmarks), the use of multiple independent checkpoints per strategy, and the inclusion of a same-base ablation (s1.1 vs s1.1-short). If the central pattern is confirmed, the results would provide useful guidance for data selection in post-training. However, the causal interpretation is not yet fully supported: the same-base ablation changes content as well as length, the cross-strategy comparisons mix base-model families, and the RL results lack an SFT-only control. The claims are directionally plausible and worth publishing after these controls are added or the causal language is softened.

major comments (3)
  1. [3.3] Section 3.3, Table 2: The sentence "the performance of s1.1-short is consistently lower than that of the base model" is not supported by the reported numbers: on GPQA-diamond, s1.1-short achieves 53.5 versus 50.0 for Qwen2.5-32B-Instruct. Moreover, the proposed ablation does not isolate response length as the causal variable; GPT-4o rewriting to produce 418.9-token answers also removes self-reflection and verification steps and changes content and formatting. The drop relative to s1.1 (or to the base model on most tasks) could therefore be due to these co-varying factors rather than to length itself. A control that compresses the same reasoning steps while preserving self-correction behavior, or a training study that varies only length, is needed before the conclusion that longer reasoning paths are beneficial can be drawn.
  2. [3.2 and 3.3] The headline comparison in Tables 1 and 2 between short-CoT SFT models (Math-CoT, Math-PoT, based on Mistral-7B) and long-CoT models (LIMO, s1.1, based on Qwen2.5-32B-Instruct) confounds the training strategy with base-model capacity, size, and alignment state. Statements such as "conventional SFT on short reasoning chains generalizes poorly" are supported by only one same-base control, s1.1-short, which has the content-rewriting confound noted above. To make the central claim robust, the paper needs same-base short-CoT SFT models on Qwen2.5-32B-Instruct (or an equivalent controlled comparison), or the conclusions must be explicitly limited to the specific models and configurations studied.
  3. [3.4] Table 3 does not include the SFT-only warmup checkpoint as a control for the SimpleRL models. The paper attributes the improvements of Qwen2.5-Math-SimpleRL and Qwen2.5-32B-SimpleRL to rule-based RL, but these models begin with an SFT stage, and the reported baseline (e.g., Qwen2.5-Math-7B) is not that SFT-stage checkpoint. Without reporting the SFT-only model's generalization performance, it is not possible to determine how much of the gain comes from RL versus the SFT warmup. I request that the SFT-only control be added for each SimpleRL variant, or that the attribution to RL be substantially softened.
minor comments (5)
  1. [Abstract] There is a typo in the abstract: "In constrast" should be "In contrast".
  2. [3.2] The claim that Math-PoT SFT "declines in eight tasks" is not supported by Table 1: relative to Mistral-7B, Math-PoT improves on ZebraLogic, ProofWriter, and DocMath, and declines on seven of the listed non-MPS benchmarks. Please correct the count or clarify which task subset is being referenced.
  3. [3.3] The benchmark name is rendered inconsistently: the text and Figure 1 use "GPQA" while Table 2 uses "GPQAdiamond". Please specify the exact evaluation set (e.g., GPQA diamond subset) and the prompting/evaluation protocol in the main text or appendix.
  4. [3.3] There is a typo in the ablation description: "using the same question from s1 1 but with a shorter CoT answer" should refer to "s1.1" consistently.
  5. [3.2] The data-coverage analysis in Figure 3 is qualitative and based on PCA projections. Reporting a quantitative overlap metric (e.g., nearest-neighbor distribution distance or retrieval-based overlap) would strengthen the claim that WebInstruct and OpenWebMath cover general reasoning topics better than MetaMath.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all central claims are empirical benchmark comparisons, not derivations from fitted inputs.

full rationale

This paper is an empirical study; it contains no fitted parameters, no closed-form derivation, and no prediction that is constructed from its target values. The central claims (long-CoT SFT and rule-based RL on MPS generalize to non-MPS reasoning, while short-CoT SFT does not) are supported by direct evaluation of checkpoints on standard external benchmarks (GPQA, LogiQA, ARC-c, etc.), with improvements or degradations measured rather than derived. The s1.1-short ablation in §3.3 is a possible methodological confound because GPT-4o rewriting changes content, self-reflection, and format along with length, so it may not isolate response length as the causal variable; however, a confounded ablation is not circularity under the stated criteria, since the comparison is not equivalent to its own inputs by construction. Self-citations to SimpleRL (Zeng et al., 2025a,b), DART-Math, and CodeI/O (with overlapping author Junxian He) are present, but they are not load-bearing in a derivational sense: the SimpleRL checkpoints are used as evaluated artifacts on independent benchmarks, and the CodeI/O mention is a passing pointer to concurrent work. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is invoked. The paper's conclusion could be threatened by confounds or base-model mismatches (e.g., short-CoT SFT models are Mistral-7B while long-CoT models are Qwen2.5-32B), but those are correctness/validity risks, not circular reasoning. Under the hard rule that circularity must be exhibited as equation-level reduction or fitted-parameter-renamed-as-prediction, no step qualifies; the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study is empirical; it depends on assumptions about benchmark validity, model representativeness, and the isolation of response length in the ablation, rather than mathematical axioms or fitted parameters.

assumptions (3)
  • domain assumption Benchmark scores on GPQA, LogiQA, ProofWriter, etc. are valid and generalizable measures of broad reasoning ability.
    The paper uses these benchmarks to conclude about general reasoning; if these tests are narrow or contaminated, the conclusion weakens. See Section 3.1.
  • domain assumption The open-source checkpoints used (DeepSeekMath, Rho-Math, MAmmoTH2, LIMO, s1.1, SimpleRL) are representative implementations of the respective training strategies.
    The paper assumes results from these specific models generalize to the strategy class they represent. Section 2.
  • domain assumption The s1.1-short ablation, generated by GPT-4o rewriting, preserves all properties of the training data except response length.
    This is load-bearing for the claim that response length drives the generalization gain. Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?." pith.science (2026). https://pith.science/paper/BAL3LK6O

@misc{pith2026250704391,
  author       = {Pith},
  title        = {Pith review of: Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BAL3LK6O}},
  note         = {Machine review of arXiv:2507.04391}
}
read the original abstract

There has been a growing interest in enhancing the mathematical problem-solving (MPS) capabilities of large language models. While the majority of research efforts concentrate on creating specialized models to solve mathematical problems, it remains unknown how learning mathematical problem-solving generalizes to help develop other reasoning abilities. In this paper, we present an empirical investigation into the generalization potential of various MPS training approaches, such as continual pretraining, instruction tuning, and rule-based reinforcement learning across various data sources, including both short and long chain-of-thought (CoT) samples. Evaluation on 5 mathematical and 8 general reasoning benchmarks show that continual pretraining on math text is able to generalize to general reasoning tasks to some extent. In constrast, instruction tuning on conventional, short MPS samples provides limited benefits and, in many cases, even impairs generalization performance. Notably, training with long CoT responses for MPS samples and incorporating rule-based reinforcement learning on MPS queries exhibit distinct behavior, significantly enhancing generalization by extending the model's reasoning processes into other domains. These results suggest that traditional approaches to learning MPS with short reasoning chains largely fail to achieve robust generalization. However, the emerging paradigm of longer reasoning chains, coupled with self-reflection, offers a promising direction for improving generalized reasoning abilities through learning from specialized domains.

Figures

Figures reproduced from arXiv: 2507.04391 by the authors.

Figure 1
Figure 1. Relative gain (%) on GPQA and LogiQA of five different MPS training ap￾proaches. SFT-short: Supervised fine-tuning on short CoT data. SFT-STEM: Supervised fine￾tuning on STEM-related data. CPT: Continual pretraining. SFT-long: Supervised fine-tuning on long CoT data. RL: Rule-based reinforcement learning on MPS queries. Cognitive neuroscience research has demonstrated that learning to solve mathematical problems en￾… view at source ↗
Figure 2
Figure 2. An overview of incorporating mathematical reasoning ability into general instruction [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the query embedding distributions: WebInstruct, MetaMath, and Open [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Outputs of Qwen2.5-32B-Instruct and LIMO to a question from LogiQA [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Relative gain (%) of models trained with instruction tuning on three types of non-MPS [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A consistency reward parsed by a 7B LLM, plus a two-stage refine-then-monitor pipeline and reformulated easy questions, improves MCQ-RL accuracy-with-consistency in law and medicine (58.9 vs 51.4 average Acc+).

Reference graph

Works this paper leans on

57 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zebralogic: Benchmarking the logical reasoning ability of language models

    Yejin Choi Bill Yuchen Lin, Ronan Le Bras. Zebralogic: Benchmarking the logical reasoning ability of language models. https://huggingface.co/spaces/allenai/ZebraLogic, 2024

    work page 2024

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  3. [3]

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022

    arXiv 2022

  4. [4]

    Instruction pre-training: Language models are supervised multitask learners

    Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruction pre-training: Language models are supervised multitask learners. arXiv preprint arXiv:2406.14491, 2024

    arXiv 2024

  5. [5]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

    work page 2025

  6. [6]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    arXiv 2018

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 57 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. 2025

    2025

  2. [10]

    Arithmetic and the brain

    Stanislas Dehaene, Nicolas Molko, Laurent Cohen, and Anna J Wilson. Arithmetic and the brain. Current opinion in neurobiology, 14 0 (2): 0 218--224, 2004

    2004

  3. [11]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023

    2023 arXiv

  4. [12]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024

    2024

  5. [13]

    Deepseek-coder: When the large language model meets programming--the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  6. [14]

    What explains the relationship between spatial and mathematical skills? a review of evidence from brain and behavior

    Zachary Hawes and Daniel Ansari. What explains the relationship between spatial and mathematical skills? a review of evidence from brain and behavior. Psychonomic bulletin & review, 27: 0 465--482, 2020

    2020

  7. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 a

    2021

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 b

    2021

  9. [17]

    O1 replication journey -- part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?, 2024

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey -- part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?, 2024

    2024

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  11. [19]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with l...

    2022

  12. [20]

    Numinamath, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024

    2024

  13. [21]

    Codei/o: Condensing reasoning patterns via code input-output prediction, 2025

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction, 2025

    2025

  14. [22]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023

    2023

  15. [23]

    Rho-1: Not all tokens are what you need, 2024

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need, 2024

    2024

  16. [24]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  17. [25]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp.\ 22631--22648. PMLR, 2023

    2023

  18. [26]

    Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms

    Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352, 2024

    2024 arXiv

  19. [27]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023

    2023 arXiv

  20. [28]

    The llama 3 herd of models

    Meta. The llama 3 herd of models. https://arxiv.org/abs/2407.21783, 2024

    2024 arXiv

  21. [29]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    2025

  22. [30]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    2023

  23. [31]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, 2024

    2024

  24. [32]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023

    2023 arXiv

  25. [33]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023

    2023

  26. [34]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  27. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  28. [36]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  29. [37]

    Proofwriter: Generating implications, proofs, and abductive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020

    2012 arXiv

  30. [38]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884, 2024

    2024 arXiv

  31. [39]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms. 2025

    2025

  32. [40]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  33. [41]

    Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving, 2024

    Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving, 2024. URL https://arxiv.org/abs/2407.13690

    2024 arXiv

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  35. [43]

    Magicoder: Source code is all you need

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023

    2023 arXiv

  36. [44]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. ArXiv, abs/2406.08464, 2024

    2024 arXiv

  37. [45]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  38. [46]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  39. [47]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  40. [48]

    Mammoth2: Scaling instructions from the web, 2024

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web, 2024

    2024

  41. [49]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025 a

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025 a . URL https://arxiv.org/abs/2503.18892

    2025 arXiv

  42. [50]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason, 2025 b . Notion Blog

    2025

  43. [51]

    Mr-ben: A comprehensive meta-reasoning benchmark for large language models

    Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. Mr-ben: A comprehensive meta-reaso...

    2024 arXiv

  44. [52]

    Docmath-eval: Evaluating math reasoning capabilities of llms in understanding long and specialized documents, 2024

    Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. Docmath-eval: Evaluating math reasoning capabilities of llms in understanding long and specialized documents, 2024

    2024

  45. [53]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  47. [55]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  48. [56]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  49. [57]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.