Pith. sign in

REVIEW 10 cited by

AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.16400 v3 pith:T2OGXCBQ submitted 2025-05-22 cs.LG cs.AIcs.CL

Yang Chen , Zhuolin Yang , Zihan Liu , Chankyu Lee , Peng Xu , Mohammad Shoeybi , Bryan Catanzaro , Wei Ping This is my paper
classification cs.LGcs.AIcs.CL
keywords modelsreasoningtrainingcodelearningmathpromptsbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite recent progress in large-scale reinforcement learning (RL) for reasoning, the training recipe for building high-performing reasoning models remains elusive. Key implementation details of frontier models, such as DeepSeek-R1, including data curation strategies and RL training recipe, are often omitted. Moreover, recent research indicates distillation remains more effective than RL for smaller models. In this work, we demonstrate that large-scale RL can significantly enhance the reasoning capabilities of strong, small- and mid-sized models, achieving results that surpass those of state-of-the-art distillation-based models. We systematically study the RL training process through extensive ablations and propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts. Notably, we find that math-only RL not only significantly enhances the performance of strong distilled models on math benchmarks (e.g., +14.6% / +17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks (e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition, extended code-only RL iterations further improve performance on code benchmarks with minimal or no degradation in math results. We develop a robust data curation pipeline to collect challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across both domains. Finally, we identify key experimental insights, including curriculum learning with progressively increasing response lengths and the stabilizing effect of on-policy parameter updates. We find that RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (e.g., distillation), but also pushes the limits of the model's reasoning ability, enabling it to solve problems that were previously unsolvable.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks

    cs.CL 2025-07 conditional novelty 7.0 of 10

    A 5,352-pair benchmark shows thinking LLMs outperform non-thinking ones as code judges, but position bias and formatting sensitivity remain strong.

  2. Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

    cs.CL 2026-07 unverdicted novelty 6.0 of 10

    Scaling zero-RL training to a 1T-parameter model yields competitive math performance and spontaneous cognitive behaviors such as self-verification and structured formatting.

  3. MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A teacher-guided dense reward for GRPO distillation improves small-model tool-use accuracy and out-of-domain generalization on executable-tool benchmarks.

  4. Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ReMix cuts reinforcement finetuning rollout volume by 30x to 450x on math reasoning by mixing historical and on-policy data with a convex KL constraint and a mid-training switch to on-policy updates.

  5. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  6. OckBench: Measuring the Efficiency of LLM Reasoning

    cs.CL 2025-11 conditional novelty 5.0 of 10

    OckBench measures LLM reasoning efficiency by pairing accuracy with decoding token count, showing large token-usage gaps among models with comparable accuracy.

  7. PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning

    cs.AI 2025-09 reject novelty 5.0 of 10

    PaVeRL-SQL reports SOTA execution accuracy on Spider2.0-SQLite using partial-match rewards and verbal RL, but overclaims SOTA on Spider and BIRD.

  8. Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A 2.75B-active-parameter MoE model, trained with a token-budget-stabilized RL method (C3PO), matches or exceeds several 7-8B dense reasoning models on AIME, LiveCodeBench, and GPQA benchmarks.

  9. AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A 7B reasoning model trained with carefully balanced SFT and RL beats prior small models on math and code benchmarks, with the paper documenting scaling and temperature heuristics.

  10. JT-Math: A Multi-Stage Framework for Advanced Mathematical Reasoning in Large Language Models

    cs.CL 2025-07 conditional novelty 4.0 of 10

    JT-Math-8B, an open 8B model family trained with a multi-stage math-focused pipeline, reports math benchmark averages above o1-mini and several 7B open models.

Pith tools