Pith. sign in

REVIEW 6 cited by

MAmmoTH2: Scaling Instructions from the Web

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.03548 v4 pith:BGXKFVRS submitted 2024-05-06 cs.CL

Xiang Yue , Tuney Zheng , Ge Zhang , Wenhu Chen This is my paper
classification cs.CL
keywords datainstructionreasoningtuningllmsmammoth2performancebenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Turn On-Policy Distillation with Prefix Replay

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.

  2. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  3. Coupled Variational Reinforcement Learning for Language Model General Reasoning

    cs.CL 2025-12 conditional novelty 5.0 of 10

    CoVRL trains an LLM on a mixture of question-only and answer-guided reasoning traces, using the model's own answer probability as reward, and reports consistent gains on math and general-reasoning benchmarks.

  4. Scaling Reasoning without Attention

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A purely attention-free Mamba-2 model, fine-tuned with PromptCoT-style synthetic data, reaches state-of-the-art scores on AIME and LiveCodeBench among open models of its size and surpasses Gemma3-27B on three benchmarks.

  5. Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DART filters and rewrites expert reasoning demonstrations with the student model's own rollouts, then fine-tunes on those outcome-consistent adapted trajectories.

  6. A Survey on Large Language Models for Mathematical Reasoning

    cs.AI 2025-06 conditional novelty 1.0 of 10

    Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.

Pith tools