Pith. sign in

REVIEW 3 major objections 6 minor 11 cited by

OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mid-training with high-quality math data and a two-stage learning-rate schedule turns Llama base models into bases that scale under reinforcement learning, matching Qwen2.5 at the same size.

desk verdict Useful controlled study of mid-training for RL, but the Qwen-parity claim is confounded by benchmark overlap in the decay-stage training data. read the letter →

arxiv 2506.20512 v1 pith:IYTJU3EK submitted 2025-06-25 cs.CL cs.AIcs.LG

Zengzhi Wang , Fan Zhou , Xuefeng Li , Pengfei Liu This is my paper
classification cs.CLcs.AIcs.LG
keywords mid-trainingreinforcementlearningscalingchain-of-thoughtmathematicalreasoningbasemodelscalabilitydataqualityrateschedule
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Different base model families respond very differently to reinforcement learning: Qwen models improve steadily, while Llama models collapse into repetitive outputs and premature final answers. This paper claims the cause is the pretraining data distribution, not something fixed in the architecture, and that a deliberate mid-training stage can fix it. The authors show that continued pretraining on high-quality math web corpora, QA-style chain-of-thought data, and a small amount of instruction-following data reshapes Llama so that RL training works. The result is OctoThinker, whose RL-trained 3B model scores 65.2 on MATH500 versus Qwen2.5-3B-Zero's 66.4. If the claim holds, mid-training becomes a general lever for making any base model RL-scalable, separating data quality from architecture choices.

What carries the argument

The load-bearing object is the Stable-then-Decay mid-training recipe: a first stage of 200B tokens on a high-quality math-Web mix at constant learning rate, followed by a 20B-token decay stage at cosine-annealed learning rate over three branches (long-CoT, short-CoT, and hybrid). This works because the second stage's purpose is distribution shaping, injecting QA data whose format matches downstream RL tasks, with instruction-following data as a small unlocker. The supporting machinery is the RL-side stabilization kit: a complex prompt template and a progressive maximum response length scheduler (2K to 4K to 8K), which prevent long-CoT mid-training from cascading into verbosity collapse.

What would settle it

A direct experiment would settle it: mid-train Llama-3.2-3B on a corpus intentionally sampled to match Qwen2.5's data distribution, run the paper's GRPO setup, and watch response length and format. If the boxed-empty repetition and length explosion to 4,096 tokens still appear, then the divergence is not caused by pretraining data distribution, and the paper's causal claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that continued pretraining on the right data mixture, not architectural modification, is what determines whether a base model can benefit from R1-Zero-style reinforcement learning. The authors observe that Qwen2.5-3B produces steadily longer, coherent reasoning traces under GRPO training, while Llama-3.2-3B collapses into boxed-empty answers and repetitive text. They then show that mid-training Llama on MegaMath-Web-Pro, adding long- and short-CoT QA data, and including a small slice of instruction-following data changes that dynamic. The final recipe, Stable-then-Decay, trains 200B tokens at constant learning rate and then 20B tokens per branch with cosine decay across three CoT-focused mixtures. After RL, OctoThinker-Long-3B reaches 65.2 on MATH500 against Qwen2.5-3B-Zero's 66.4, which the paper reads as closing the RL-scalability gap between model families.

Load-bearing premise

The load-bearing premise is that Llama's poor RL behavior is caused by its pretraining data distribution, which mid-training can repair, rather than by architecture, tokenizer, or other fixed properties of the base model.

Editorial extensions

If this is right

  • Corpus quality is the first-order lever: mid-training on MegaMath-Web-Pro or MegaMath-Web-Pro-Max improves RL outcomes, while FineMath-4plus at the same budget does not.
  • QA-style data, especially long chain-of-thought, is what pushes RL performance up, but it destabilizes training unless instruction-following data, a structured prompt template, and a progressive length cap are added.
  • Scaling the mid-training budget from 20B to 100B tokens keeps improving downstream RL performance even when few-shot base-model scores have plateaued.
  • After RL, the OctoThinker family matches Qwen2.5 at 1B and 3B scale on math benchmarks, so mid-training can substitute for whatever made Qwen a friendlier RL base.
  • Distributions matter more than volume: QA data drawn from structured downstream sets such as OpenMathInstruct-2, OpenR1, and NuminaMath outperforms web-derived QA, with gains saturating past a 30% QA mix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the distributional account is right, the same recipe should rescue other base models that collapse under RL, and standard few-shot base-model evals should be supplemented by RL-specific signals like response-length growth and format-collapse rate.
  • The pattern of base-eval plateaus alongside continued RL gains suggests that perplexity or few-shot accuracy are the wrong screening signals for choosing pretraining data; downstream RL performance is the more direct criterion.
  • A sharper test of the data-quality story would ablate format from content, comparing the same questions with and without long CoT, to see whether the benefit comes from the reasoning trace or from exposure to harder problems.
  • The choice to branch the decay stage implies future base models may be designed as multi-branch families sharing a stable trunk rather than as a single monolithic checkpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates why Llama and Qwen base models exhibit divergent reinforcement learning (RL) dynamics on mathematical reasoning, and proposes mid-training as an intervention that can make Llama more RL-scalable. Through controlled 20B-token mid-training experiments on Llama-3.2-3B, the authors identify math web corpus quality, QA-format data (especially long CoT), instruction-following data, and token budget as factors that affect downstream RL performance. They then scale up to a two-stage 'Stable-then-Decay' recipe: 200B tokens of stable-stage training on a curated math-heavy mixture followed by 20B tokens of decay-stage training with three data branches (Long, Short, Hybrid). The resulting OctoThinker models are trained with R1-Zero-style RL (GRPO on MATH8K) and reported to reach performance on par with Qwen2.5-3B-Zero on four indicator benchmarks (MATH500, GSM8K, OlympiadBench, AMC23), closing the gap between Llama and Qwen in RL settings.

Significance. If the central claim holds, the paper would provide a practical recipe for converting general-purpose base models into RL-scalable reasoning models, at a mid-training cost that is substantial but far smaller than full pretraining. The open release of the MegaMath-Web-Pro-Max corpus (70B+ tokens), the three model branches, and training scripts would be valuable community assets. The controlled ablations in Section 3 (data quality, QA nature, instruction data, budget) are well-designed and the observed RL dynamics differences are clearly shown. However, the load-bearing claim of parity with Qwen is currently threatened by two confounds: the decay-stage training corpora overlap the evaluation benchmarks, and the final recipe was selected by monitoring the same benchmarks used for the headline comparison. The scientific contribution therefore depends on whether these confounds can be removed or quantified.

major comments (3)
  1. [§4.2.1, Table 5, Fig. 14] The parity claim with Qwen2.5-3B-Zero is confounded by overlap between the decay-stage training data and the evaluation benchmarks. Section 4.2.1 states that OpenR1-Math-220K and OpenMathInstruct-2 are 'derived from structured downstream datasets (e.g., GSM8K, MATH)', and Table 5 includes 15% OpenR1, 17.5% OpenMathInstruct-2, 10% NuminaMath-1.5, and 15% AM-DeepSeek-Distilled-40M in the branches whose RL curves are shown in Figure 14. The four benchmarks used for the parity comparison are MATH500, GSM8K, OlympiadBench, and AMC23, with MATH500 being a subset of MATH. The Qwen2.5-3B-Zero baseline did not see these benchmark-derived corpora during mid-training. Consequently, the observed 'parity' may reflect partial memorization of evaluation problem families rather than mid-training-induced RL scalability. Please quantify n-gram overlap between decay-stage corpora and the evaluation suites, repeat the decay-stage recipe on overlap-filtered data, and/or demonstrate the parity on held-out competition benchmarks (e.g., AIME 2024/2025) that were never present in any training corpus. Without such evidence, the central claim of the paper is not established.
  2. [§4.2.1, Fig. 17, Fig. 14] The final recipe was selected using the same benchmarks on which the gains are reported. The 30% QA ratio, the choice of OpenR1/OpenMathInstruct-2/NuminaMath over MegaMath-QA, and the branch selection were all determined by inspecting RL accuracy curves on MATH500, GSM8K, OlympiadBench, and AMC23 (Figure 17), and these exact four benchmarks are then used to report the Qwen-parity result (Figure 14). This is selection on the evaluation set, so the reported improvement is not an unbiased estimate of the recipe's effect. To support the central claim, the paper should report results on a held-out suite that played no role in the pilots (e.g., AIME24/AIME25, HARP, or a disjoint random split of MATH not used in any training stage), or show that the qualitative ranking of the pilot configurations is invariant across a broader set of tasks. Absent this, the headline magnitude of the parity is uncertain.
  3. [§2.2, §3.2–3.5, §7] The paper's broad scientific framing—that pretraining data distribution is what makes a base model suitable for RL—is stronger than the evidence supports. The experiments show that changing the data via mid-training changes RL behavior for Llama, but they do not control for architecture, tokenizer, initialization, or optimization details baked into Llama versus Qwen. Section 2.2 acknowledges this only as a hypothesis ('we attribute the reason to their potential divergence of pre-training despite their opaque details'), and Section 7 repeats the causal claim without new evidence. A concrete test would be to apply the same mid-training recipe to another base model family, or to compare Llama mid-trained on a Qwen-like mixture against Qwen mid-trained on a Llama-like mixture. As written, the title and abstract generalize beyond what the controlled manipulations can establish.
minor comments (6)
  1. [Figure 14] The axis labels for GSM8K, OlympiadBench, and AMC23 contain stray 'xx' and should be corrected.
  2. [§3.5, Figure 11] The 70B and 100B checkpoints come from a single cosine-LR run, so the claim that scaling the mid-training budget 'consistently' improves RL performance would be strengthened by stating that no repeated runs were used or by adding a second run for at least one token budget.
  3. [§3.1, §4.2.2] The controlled experiments in Section 3 use a 20B-token cosine-LR schedule on the original Llama-3.2-3B base, while the final recipe first trains 200B tokens with a constant LR (Table 3); the paper does not verify that the qualitative findings (e.g., long-CoT over short-CoT, the unlocking effect of instruction data) would be identical after a 200B stable stage.
  4. [§4.3, Tables 6–8] The text states that each decay branch shows 'consistent gains' over the stable-stage model, but Tables 6–8 contain counterexamples (e.g., OctoThinker-1B-Long on AMC23 at 7.50 vs. Stable at 10.00, and OctoThinker-3B-Short on AMC23 at 2.50 vs. Stable at 5.00); the overstatement should be softened or the exceptions discussed.
  5. [§3.4, §4.2] The decay-stage QA corpora (OpenR1, AM-DeepSeek-Distilled) are distilled from proprietary long-CoT models, but this dependence is only implicit in the 'Future Work' section; it should be stated prominently in the main text because it affects reproducibility and the interpretation of the recipe's transferability.
  6. [§2.1, Figures 5–14] The RL dynamics plots appear to come from single runs without reported variance or smoothing; please clarify in the figure captions or text whether these are single trajectories, as the 'stability' and 'unstability' claims are otherwise anecdotal.

Circularity Check

2 steps flagged · score 6.0 of 10

Decay-stage QA mixtures are built from GSM8K/MATH-derived corpora and competition sets, and the QA ratio is tuned on the same four RL benchmarks used to claim Qwen parity, so the headline RL-scaling result is partly a training/eval overlap artifact.

  1. fitted input called prediction [Section 4.2.1, Table 5, Section 5/Figure 14]
    "OpenR1-Math-220K and OMI2 are derived from structured downstream datasets (e.g., GSM8K, MATH), while MegaMath-QA is sourced from less curated web documents. These differences in data source and distribution substantially impact downstream RL performance, highlighting the importance of distributional alignment between mid-training data and downstream tasks."

    Decay-stage Table 5 places 10-17.5% OMI2, 10-15% OpenR1, 10% NuminaMath-1.5, and 10-15% AM-DeepSeek-Distilled in the mixtures; these are, per the quote, derived from GSM8K/MATH and competition-style benchmarks. The evaluation for the headline parity (Figure 14) uses MATH500 (a MATH subset), GSM8K, OlympiadBench, and AMC23, and the QA ratio/branch choices were selected from pilot RL runs on exactly these four tasks (Figure 17). Thus the post-RL score is partly a measurement of benchmark-derived training exposure, while the Qwen2.5 baseline did not receive this benchmark-derived mid-training. The claim that mid-training itself, rather than overlap, closes the Qwen gap is therefore not established.

  2. other [Section 2.1 (RL Setup and Evaluation)]
    "For RL training prompts, we adopt the MATH8K dataset due to its moderate difficulty and concise composition. ... we employ GSM8K, MATH500, OlympiadBench, and AMC23 as indicator tasks to analyze RL dynamics."

    The RL stage trains on MATH8K, drawn from MATH levels 3-5, while MATH500 is a subset of MATH and GSM8K is one of the source families for OMI2 used in the decay stage. Evaluation on these tasks therefore further aligns the RL training signal with the reported metrics; the improvement is not a clean out-of-distribution measure of RL scalability, though it is a lesser confound than the decay-stage overlap.

full rationale

Most of the paper is a legitimate empirical study: the stable-stage experiments use web corpora (MegaMath-Web-Pro-Max, DCLM) that are not built from the evaluation benchmarks, and the base-model gains on 13 benchmarks are not circular. The citation to MegaMath-Web-Pro is prior work with independent corpus-quality validation and does not, by itself, make the argument circular. The circularity concern is concentrated in the decay stage and in the headline comparison. Section 4.2.1 states that the chosen QA corpora are derived from the same downstream datasets (GSM8K, MATH) as the evaluation, and the paper deliberately selects the 'distribution-aligned' mixture using RL runs measured on MATH500, GSM8K, OlympiadBench and AMC23. Since the final claim (Figure 14) is precisely that OctoThinker reaches 'performance on par with Qwen2.5' on those benchmarks, the result is partly forced by the training/evaluation overlap: the model has seen benchmark-derived question families, while the Qwen baseline has not received this treatment. This is a fitted-input-called-prediction pattern, not a formal self-definition. The paper would need contamination-controlled held-out evaluation (e.g., fresh benchmarks, or explicit decontamination of OMI2/OpenR1/NuminaMath against MATH500/GSM8K/AMC23/OlympiadBench) to establish that mid-training itself, rather than benchmark-derived training data, drives the parity. Score 6 reflects partial circularity: the stable-stage scaling results retain independent content, but the central Qwen-parity claim as evaluated is compromised.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are postulated; the contributions are datasets, models, and a training recipe. MegaMath-Web-Pro-Max is a corpus, not an invented entity. The main conceptual cost is that the recipe is tuned on the evaluation benchmarks, and the QA corpora overlap with benchmark families, so part of the measured effect may be distribution matching rather than general capability.

free parameters (6)
  • fasttext recall threshold = 0.4
    Chosen from the corpus quality vs quantity tradeoff in Figure 4; controls which documents enter MegaMath-Web-Pro-Max.
  • decay-stage QA mixing ratio = 30%
    Selected after ablations at 10/20/30/40% in Section 4.2.1 and Figure 17; higher ratios plateaued.
  • web:QA:instruction pilot ratio = 89:10:1
    Hand-set in Section 3.4 to test instruction-following effects; not derived from any theory.
  • stable-stage data mixture weights = 0.725 MegaMath-Web-Pro-Max, 0.10 DCLM, 0.10 code blocks, 0.05 MegaMath-QA, 0.0125 MegaMath-Code, 0.0125 MegaMath Trans…
    Assigned in Table 2 without ablation; based on MegaMath-Llama settings with reduced synthetic data.
  • RL sampling-to-gradient-update ratio = 2
    Chosen empirically in Section 2.1 to stabilize RL; affects all RL curves in the paper.
  • progressive maximum length schedule = 2K to 4K to 8K at steps 0, 200, 320
    Introduced in Section 3.4 to stabilize long-CoT RL; part of the final RL recipe.
assumptions (4)
  • domain assumption GRPO on MATH8K with KL coefficient 0 and 4096/8192 max lengths is a valid proxy for RL scalability
    All conclusions about RL-friendliness are drawn from this single RL setup (Section 2.1); no evidence it transfers to other RL algorithms, reward designs, or prompt sets.
  • domain assumption Llama's RL failure is caused by pretraining data distribution, not architecture or optimization
    Section 2.2 states 'we attribute the reason to their potential divergence of pre-training despite their opaque details'; the interventions never vary architecture, so this premise is untested.
  • domain assumption Few-shot base-model evaluation and zero-shot RL-model evaluation are comparable improvement measures
    Base scores in Tables 6-8 use few-shot prompts while RL results use zero-shot (Section 2.1); differences in protocol can distort gain sizes.
  • domain assumption MegaMath-Web-Pro-Max is comparable in quality to MegaMath-Web-Pro
    The claim rests on a 5B-token pre-training check (Figure 4) and classifier scores, not on an independent benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling." pith.science (2026). https://pith.science/paper/IYTJU3EK

@misc{pith2026250620512,
  author       = {Pith},
  title        = {Pith review of: OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IYTJU3EK}},
  note         = {Machine review of arXiv:2506.20512}
}
read the original abstract

Different base language model families, such as Llama and Qwen, exhibit divergent behaviors during post-training with reinforcement learning (RL), especially on reasoning-intensive tasks. What makes a base language model suitable for reinforcement learning? Gaining deeper insight into this question is essential for developing RL-scalable foundation models of the next generation. In this work, we investigate how mid-training strategies shape RL dynamics, focusing on two representative model families: Qwen and Llama. Our study reveals that (1) high-quality mathematical corpora, such as MegaMath-Web-Pro, significantly improve both base model and RL performance, while existing alternatives (e.g., FineMath-4plus) fail to do so; (2) further adding QA-style data, particularly long chain-of-thought (CoT) reasoning examples, enhances RL outcomes, and instruction data further unlocks this effect; (3) while long-CoT improves reasoning depth, it can also induce verbosity of model responses and unstability of RL training, underscoring the importance of data formatting; (4) scaling mid-training consistently leads to stronger downstream RL performance. Building on these insights, we introduce a two-stage mid-training strategy, Stable-then-Decay, in which base models are first trained on 200B tokens with a constant learning rate, followed by 20B tokens across three CoT-focused branches with learning rate decay. This yields OctoThinker, a family of models demonstrating strong RL compatibility and closing the performance gap with more RL-friendly model families, i.e., Qwen. We hope our work will help shape pre-training strategies for foundation models in the RL era. To support further research, we release our open-source models along with a curated math reasoning-intensive corpus of over 70 billion tokens (i.e., MegaMath-Web-Pro-Max).

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.

  2. CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    CPMobius uses iterative coach-player reinforcement learning to improve mathematical reasoning in LLMs without external training data, yielding +4.9 average accuracy gains on Qwen2.5-Math-7B-Instruct.

  3. MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A teacher-guided dense reward for GRPO distillation improves small-model tool-use accuracy and out-of-domain generalization on executable-tool benchmarks.

  4. SSRL: Self-Search Reinforcement Learning

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.

  5. URPO: A Unified Reward & Policy Optimization Framework for Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A seven-billion-parameter model trained with one unified GRPO loop improves instruction following, reasoning, and reward modeling at the same time.

  6. Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

    cs.CL 2025-11 reject novelty 5.0 of 10

    Reasoning LLMs seem better at retrieving hierarchical facts not because they know more but because they navigate better; the key supporting RL experiment is missing from the paper.

  7. RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction

    cs.RO 2025-09 conditional novelty 5.0 of 10

    Robot policies trained on human interventions that rewind to a familiar state and then correct the mistake achieve higher long-horizon success and better data efficiency than imitation on full demonstrations alone.

  8. Large-Scale Diverse Synthesis for Mid-Training

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    BoostQA is a 100B-token synthesized QA corpus whose mid-training on a 40B-token subset improves Llama-3 8B by 12.74% on average across MMLU and CMMLU and reaches top average performance on 12 benchmarks.

  9. LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A knowledge point graph walk synthesizes a 50B token QA dataset that reportedly lifts Llama-3 8B average MMLU and CMMLU scores by 11.51%.

  10. Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards

    stat.ML 2025-11 reject novelty 4.0 of 10

    OBLR-PO combines an SNR-based learning rate and a gradient-weighted baseline for RLVR, but the main theory is undermined by a flawed smoothness proof.

  11. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

53 extracted references · 10 canonical work pages · cited by 11 Pith papers

  1. [1]

    Behl, S \' e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J

    Marah I Abdin, Jyoti Aneja, Harkirat S. Behl, S \' e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, ...

    arXiv doi:10.48550/arxiv.2412.08905 2024

  2. [2]

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S \' e bastien Bubeck, Martin Cai, Caio C \' e sar Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dix...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  3. [3]

    Smollm2: When smol goes big - data-centric training of a small language model

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \' a zquez, Guilherme Penedo, Lewis Tunstall, Andr \' e s Marafioti, Hynek Kydl \' cek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan - Son Nguyen, Cl \' e mentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, C...

    arXiv doi:10.48550/arxiv.2502.02737 2025

  4. [4]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and...

    work page 2019

  5. [5]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4WnqRR915j

    2024

  6. [6]

    Puzzle: Distillation-based NAS for inference-optimized llms

    Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein, and Ran E...

    arXiv doi:10.48550/arxiv.2411.19146 2024

  7. [7]

    Llama-nemotron: Efficient reasoning models

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949, 2025

    arXiv 2025

  8. [8]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  2. [10]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on ...

    2023 doi

  3. [11]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023 b

    2023 arXiv

  4. [12]

    Tran, Mike Zhang, Shiqi Chen, Tianyu Pang, Chao Du, Xinyi Wan, Wei Lu, and Min Lin

    Longxu Dou, Qian Liu, Fan Zhou, Changyu Chen, Zili Wang, Ziqi Jin, Zi-Yan Liu, Tongyao Zhu, Cunxiao Du, Penghui Yang, Haonan Wang, Jiaheng Liu, Yongchi Zhao, Xiachong Feng, Xin Mao, Man Tsung Yeung, Kunat Pipatanakul, Fajri Koto, Min Si Thu, Hynek Kydl'ivcek, Ze-Xuan Liu, Quns...

    2025 arXiv

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  6. [14]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  8. [16]

    Infi MM -webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning

    Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. Infi MM -webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning. In The 4th Workshop on Mathematical Reasoning...

    2024

  9. [17]

    Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scienti...

    2024

  10. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  11. [19]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  12. [20]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Z...

    2024 doi arXiv

  13. [21]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024 b

    2024 arXiv

  14. [22]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    HuggingFace. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  15. [23]

    Mawps: A math word problem repository

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages...

    2016

  16. [24]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 doi arXiv

  17. [25]

    T " ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024 b

    2024 arXiv

  18. [26]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman - Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur - Ari, and Vedant Misra

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman - Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur - Ari, and Vedant Misra. Solving quantitative reasoning problems with language m...

    2022

  19. [27]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  20. [28]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. CoRR, abs/2305.20050, 2023. doi:10.48550/ARXIV.2305.20050. URL https://doi.org/10.48550/arXiv.2305.20050

    2023 doi arXiv

  21. [29]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  22. [30]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  23. [31]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023

    2023

  24. [32]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog https://pretty-radio-b75.notion.site/DeepSca...

    2025

  25. [33]

    The llama 3 herd of models

    Meta, Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  26. [34]

    A diverse corpus for evaluating and developing english math word problem solvers

    Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975--984, 2020

    2020

  27. [35]

    Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Chris Wilhelm, Michael Wilson, Luke S

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2025 arXiv

  28. [36]

    Introducing openai o3 and o4-mini | openai, April 2025

    OpenAI. Introducing openai o3 and o4-mini | openai, April 2025. URL https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  29. [37]

    OpenAI, Ahmed El-Kishky, Daniel Selsam, Francis Song, Giambattista Parascandolo, Hongyu Ren, Hunter Lightman, Hyung Won, Ilge Akkaya, Ilya Sutskever, Jason Wei, Jonathan Gordon, Karl Cobbe, Kevin Yu, Lukasz Kondraciuk, Max Schwarzer, Mostafa Rohaninejad, Noam Brown, Shengjia Z...

    2024 arXiv

  30. [38]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jKHmjlpViu

    2024

  31. [39]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080--2094, 2021

    2021

  32. [40]

    Zettlemoyer

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hanna Hajishirzi, Pang Wei Koh, and Luke S. Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr. 2025. URL ht...

    2025 arXiv

  33. [41]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  34. [42]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  35. [43]

    Alan Wake, Bei Chen, C. X. Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Fan Zhou, Feng Hu, Guoyin Wang, Heng Ji, Howard Qiu, Jiangcheng Zhu, Jun Tian, Katherine Su, Lihuan Zhang, Liying Li, Ming Song, Mou Li, Peng Liu, Qicheng Hu, Shawn Wang, Shijun Z...

    2024 doi arXiv

  36. [44]

    Reinforcement learning for reasoning in large language models with one training example

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example. CoR...

    2025 doi arXiv

  37. [45]

    Mathpile: A billion-token-scale pretraining corpus for math

    Zengzhi Wang, Xuefeng Li, Rui Xia, and Pengfei Liu. Mathpile: A billion-token-scale pretraining corpus for math. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=RSvhU69sbG

    2024

  38. [46]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, ...

    2022

  39. [47]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  40. [48]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  41. [49]

    Qwen2.5 technical report

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  42. [50]

    DAPO: an open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

    2025 doi arXiv

  43. [51]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  44. [52]

    Wildchat: 1m chat GPT interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chat GPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM

    2024

  45. [53]

    Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. Megamath: Pushing the limits of open math corpora. ArXiv, abs/2504.02807, 2025. URL https://arxiv.org/abs/2504.02807

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.