Pith. sign in

REVIEW 4 major objections 5 minor 19 cited by

Magistral

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pure reinforcement learning, with no cold-start reasoning traces, lifts a capable instruct model to strong math and coding performance, while preserving or improving other capabilities.

desk verdict Solid and transparent RLVR report, but the headline AIME gain lacks a documented decontamination step; referee should require it. read the letter →

arxiv 2506.10910 v1 pith:YRPWNEAX submitted 2025-06-12 cs.CL

Mistral-AI: Abhinav Rastogi , Albert Q. Jiang , Andy Lo , Gabrielle Berrada , Guillaume Lample , Jason Rute , Joep Barmentlo , Karmesh Yadav
show 91 more authors
Kartik Khandelwal Khyathi Raghavi Chandu Léonard Blier Lucile Saulnier Matthieu Dinot Maxime Darrin Neha Gupta Roman Soletskyi Sagar Vaze Teven Le Scao Yihan Wang Adam Yang Alexander H. Liu Alexandre Sablayrolles Amélie Héliou Amélie Martin Andy Ehrenberg Anmol Agarwal Antoine Roux Arthur Darcet Arthur Mensch Baptiste Bout Baptiste Rozière Baudouin De Monicault Chris Bamford Christian Wallenwein Christophe Renaudin Clémence Lanfranchi Darius Dabert Devon Mizelle Diego de las Casas Elliot Chane-Sane Emilien Fugier Emma Bou Hanna Gauthier Delerce Gauthier Guinet Georgii Novikov Guillaume Martin Himanshu Jaju Jan Ludziejewski Jean-Hadrien Chabran Jean-Malo Delignon Joachim Studnia Jonas Amar Josselin Somerville Roberts Julien Denize Karan Saxena Kush Jain Lingxiao Zhao Louis Martin Luyu Gao Lélio Renard Lavaud Marie Pellat Mathilde Guillaumin Mathis Felardos Maximilian Augustin Mickaël Seznec Nikhil Raghuraman Olivier Duchenne Patricia Wang Patrick von Platen Patryk Saffer Paul Jacob Paul Wambergue Paula Kurylowicz Pavankumar Reddy Muddireddy Philomène Chagniot Pierre Stock Pravesh Agrawal Romain Sauvestre Rémi Delacourt Sanchit Gandhi Sandeep Subramanian Shashwat Dalal Siddharth Gandhi Soham Ghosh Srijan Mishra Sumukh Aithal Szymon Antoniak Thibault Schueller Thibaut Lavril Thomas Robert Thomas Wang Timothée Lacroix Valeriia Nemychnikova Victor Paltz Virgile Richard Wen-Ding Li William Marshall Xuanyu Zhang Yunhao Tang
This is my paper
classification cs.CL
keywords reinforcementlearningverifiablerewardsreasoningmodelslargelanguageGRPOmultimodaldatafilteringconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a reasoning model can be built with pure reinforcement learning from verifiable rewards, with no cold-start reasoning traces distilled from an existing reasoning model. The authors report that their RL pipeline alone lifts pass@1 accuracy on the AIME'24 math benchmark from 26.8% to 73.6%, and on LiveCodeBench v5 from 29.1% to 59.4%, starting from a capable instruct model. They also claim that training on text-only problems preserves or improves multimodal understanding, instruction following, and function calling, and that pure RL works for a 24-billion-parameter model, contradicting earlier reports that small models need distillation to compete. If true, the result matters because it makes reasoning capability a product of scalable RL and verifiable rewards rather than of access to proprietary reasoning traces, and it suggests RL on text can strengthen capabilities across modalities.

What carries the argument

The central object is the verifiable-reward training loop: an online, asynchronous GRPO system in which generators continuously produce completions, a rule-based math verifier (parsers plus SymPy) or a sampled 20-test C++/Python code runner assigns correctness rewards, and a soft length penalty plus a fastText language-consistency reward shape behavior. The algorithm removes the KL divergence penalty, uses a relaxed upper clip threshold ($\epsilon_{\text{high}}$) to preserve exploration, normalizes the loss by total generation length, normalizes advantages per minibatch, and drops groups with zero advantage. The argument runs on this loop: staged difficulty filtering (a weaker model first, then an RL-trained grader) and consensus-based ground-truth cleaning supply problems that are hard but solvable, while the RL loop converts increased completion length and reward into benchmark gains.

What would settle it

Run the same RL pipeline under two controls: (1) replace the correctness reward with a random binary reward while keeping format, length, and language rewards; and (2) hold out the problems that the consensus filter removed (where the model's majority answer disagreed with the reference) and test the final model on them. If the random-reward run shows similar AIME gains, the reported improvements are not driven by verifiable correctness; if the held-out problems are solved poorly, the filter deleted valid hard problems and data curation, not RL, is carrying the result.

Watch

Extended reading notes

Core claim

The central claim is that an end-to-end online RLVR pipeline, built from scratch with the authors' own models and infrastructure, is sufficient to turn an instruct model into a strong reasoner. On AIME'24, pass@1 rises from 26.8 to 73.6 and majority voting reaches 90.0; on AIME'25 pass@1 rises from 21.2 to 64.9; LiveCodeBench v5 rises from 29.1 to 59.4. The paper attributes these gains to a specific recipe: GRPO with the KL penalty removed, Clip-Higher exploration, length- and advantage-normalized losses, zero-advantage group filtering, staged data difficulty with an RL-trained grader, and a language-consistency reward that keeps thoughts and answers in the user's language. The paper further claims that RL on text preserves or improves multimodal, instruction-following, and function-calling abilities, and that pure RL on a 24B model reaches performance comparable to SFT on distilled traces, with RL on top of SFT giving the best result.

Load-bearing premise

The whole training signal rests on the automated verifiers and the filtered dataset: if the rule-based math parser and SymPy comparison, the 20 sampled code tests, and the consensus-based removal of supposedly wrong ground truths do not track true solution quality, the model optimizes against a distorted target and the benchmark gains will not transfer.

Editorial extensions

If this is right

  • Reasoning capability can be trained with verifiable rewards alone, removing the need to license or reproduce proprietary reasoning traces for cold-start.
  • Text-only RL acts as a general capability booster: multimodal benchmarks (MMMU, MMMU-Pro) improve even though training is text-only.
  • Pure RL is viable for smaller models, so the recipe transfers beyond frontier-scale checkpoints, and combining distillation SFT with RL yields the strongest small model.
  • The staged difficulty and consensus-based data cleaning suggest that dataset curation and verifier quality are as important as the RL algorithm itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If verifiable-reward quality is the true driver, the same recipe should transfer to other domains with automated checkers (e.g., formal verification, unit-test-heavy software engineering, structured data extraction), and improving verifier coverage should compound the gains.
  • The consensus-based ground-truth cleaning implies a non-trivial fraction of benchmark labels are wrong; a testable consequence is that evaluating on a cleaned label set would raise scores for many RL-trained models, not just this one.
  • The reported cross-modal transfer (text-only RL improving MMMU/MMMU-Pro) suggests reasoning skill is partly modality-agnostic; injecting a small amount of vision-language verifiable data could test whether the effect saturates or compounds.
  • The paper's recipe differs from earlier pure-RL reports in several ways (no KL, Clip-Higher, two-stage data filtering, group filtering); ablating these one at a time on a fixed base model would reveal which ingredient is responsible for the small-model gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Magistral (arXiv:2506.10910) introduces Mistral's first reasoning models, Magistral Medium and Magistral Small, trained with a scalable reinforcement learning (RL) pipeline built in-house. Medium is trained with pure RLVR on top of Mistral Medium 3, without cold-start reasoning traces, and is reported to raise AIME'24 pass@1 from 26.8 to 73.6 and LiveCodeBench v5 from 29.1 to 59.4. Small is a 24B model first SFT'd on traces from Medium and then further trained with RL; its weights are released under Apache 2.0. The paper details GRPO modifications (removal of KL penalty, loss normalization, relaxed upper clip, filtering of zero-advantage groups), an asynchronous distributed training infrastructure, two-stage difficulty filtering for math and code data, multilingual AIME evaluations, ablations on batch size and advantage normalization, analyses of weight-space dynamics, and evidence that RL on text preserves or improves multimodal and instruction-following capabilities.

Significance. If the reported gains are genuine, the paper provides strong evidence that pure RLVR at scale, without distillation or cold-start traces, can produce large improvements on math and coding benchmarks for a capable instruct model. It also contributes useful negative results (proportional code rewards, entropy bonus terms) and an unusually candid set of ablations, including batch-size and advantage-normalization studies. The open-source release of Magistral Small under Apache 2.0 and the detailed infrastructure description are valuable for reproducibility and follow-up work. The central quantitative claims, however, hinge on the assumption that the evaluation benchmarks were excluded from the training data, and this assumption is not documented in the manuscript. Because the data curation sections describe training pools drawn from exactly the same genres as AIME and LiveCodeBench, the absence of a decontamination analysis is a load-bearing gap that prevents full confidence in the headline numbers.

major comments (4)
  1. [Sections 4.1, 4.2, and 5.2] The manuscript reports that pure RL on Mistral Medium 3 yields a ~50% AIME'24 pass@1 gain (26.8 to 73.6) and a ~30-point LiveCodeBench v5 gain (29.1 to 59.4), but it never states that AIME-24/25, LiveCodeBench, MATH-500, GPQA, or HLE were excluded from the ~700k math and 35k code training pools. The curation in Sections 4.1 and 4.2 filters for format, difficulty, and answer consistency, but no decontamination step (exact-match, near-duplicate, or n-gram overlap analysis) is described. Since AIME problems are exactly numerical-answer math problems, and the code pool is drawn from contest data of the same genre as LiveCodeBench, the reported gains are currently consistent with training-set overlap rather than reasoning generalization. The authors should add a decontamination analysis, reporting the number of training examples matching each benchmark and re-evaluating the models on the deduplicated or held-out subsets.
  2. [Section 4.1, Stage-2 difficulty filter] The second-stage filter removes problems where a majority of the RL-graded model's samples agree on a final answer that disagrees with the reference, on the rationale that such problems likely have incorrect ground truths. This criterion also removes hard problems that the model consistently fails, thereby biasing the training distribution toward easier or model-aligned items. If benchmark problems are present in the training pool, this filter would preferentially delete hard benchmark items, inflating measured performance on those exact problems. The paper should quantify how many benchmark problems are affected by this filter and provide evidence (e.g., manual inspection or a validated subset) that the removed problems are indeed mislabeled rather than merely difficult for the current model.
  3. [Section 5.3] For Magistral Small, the SFT checkpoint used as the initial policy for the subsequent RL stage was selected as 'the best checkpoint on AIME'24' among four training epochs. This selection on the target evaluation benchmark, combined with the reported AIME'24 results for the final SFT+RL model, means the Small AIME'24 numbers are optimistically biased and do not cleanly measure the pipeline's generalization. The authors should select the checkpoint using a separate validation set (e.g., MATH or a held-out subset of the SFT data) and report the AIME'24 results for that checkpoint, or at least disclose the selection procedure explicitly and discuss its impact on the headline AIME'24 comparison.
  4. [Tables 2 and 3] The paper reports averages over 64 runs for AIME and 16 runs for LiveCodeBench but provides no standard errors or significance tests. Several comparative claims are within a plausible noise range: for example, in Table 3, SFT versus SFT+RL on AIME'24 maj@64 goes from 90.0 to 83.3 (a decrease), and GPQA moves from 68.8 to 68.2; in Table 2, Magistral Medium's AIME'24 pass@1 of 73.6 is compared against DeepSeek-R1's 79.8 with no variance information. Adding confidence intervals or at least reporting the per-run dispersion would materially strengthen the validity of the stated comparisons.
minor comments (5)
  1. [Section 2.1, final GRPO equation] The statement that the loss is shown 'with all modifications highlighted in red' is not visible in the printed version; please use a different convention, such as equation annotations or a textual list of the modifications.
  2. [Section 5.1] The benchmark is referred to as 'the MATH dataset' in the text and 'MATH-500' in the tables; please use consistent terminology throughout.
  3. [Section 5.4] The multilingual AIME 2024 versions are described only as 'created by translating the questions'; the translation process, the translator model or human efforts, and any quality-control steps should be described, and ideally the translated benchmarks should be released for reproducibility.
  4. [Section 2.2.2, code correctness] The code verifier samples 20 tests per problem within each response group, but the paper does not discuss how sensitive the reward signal is to this random test selection; a brief analysis or a fixed-seed policy would be helpful.
  5. [Section 3] The claim that recomputing the key-value cache is unnecessary 'potentially due to off-policy corrections inherent to the loss function' is an empirical assertion without direct evidence; please either cite a supporting ablation or soften the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported benchmark gains are measured against external benchmarks, and the RL reward and data filters do not define the evaluation metrics.

full rationale

The paper's load-bearing empirical claims are benchmark improvements on AIME-24, LiveCodeBench, MATH-500, GPQA, and related external evaluations. The training reward in Section 2.2.2, which uses a rule-based math parser with SymPy and 20 sampled code tests, is not the same object as the reported evaluation metrics: AIME and LiveCodeBench scores are computed from held-out problem sets with their own answer keys, not from the training verifier. The two-stage difficulty filter in Section 4.1 does use a checkpoint produced by the authors' own RL pipeline to grade and filter the math pool, which is self-referential in a broad sense, but it is not a definitional identity: the filter is a data-curation heuristic, and the final evaluation is external to it. The absence of a documented decontamination step for the 699k math and 35k code pools is a real correctness risk, because training/eval overlap could inflate the reported gains, but contamination is an empirical validity threat rather than a circularity pattern under the stated definitions: the paper does not define the evaluation benchmarks in terms of the training reward or filter. No fitted parameter is renamed as a prediction, and no load-bearing claim reduces by construction to its own inputs.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central claim rests on the reliability of automated verifiers, the validity of self-filtered training data, and several training-stability assumptions that are only empirically asserted. The listed free parameters are the main hand-tuned hyperparameters of the RL recipe.

free parameters (5)
  • epsilon_high = 0.26 to 0.28, manually adjusted during training
    Section 2.1: upper clipping threshold was tuned to keep group entropy stable; this is a hand-set hyperparameter that the authors say is crucial.
  • unpenalized length margin (lmax - lcache) = increased 16k -> 24k -> 32k during stages
    Section 5.2: the length penalty threshold was raised to prevent generation length stagnation; stage schedule chosen by hand.
  • batch size (nbatch) = 8k -> 4k -> 2k for Medium; 2048 for Small
    Section 5.2 and Section 6.3: batch sizes were decreased to manage KV-cache memory and stability; ablations show performance depends weakly on batch size.
  • reward weights = 0.1 format, 0.9 correctness, 0.1 language
    Section 2.2: these weights are chosen so a fully correct well-formatted response gets total reward 1.0; no sensitivity analysis.
  • difficulty filter thresholds = 16 samples per problem; remove never-solved or high-success problems
    Section 4.1: the two-stage filter uses 16 generations per problem and removes extremes; exact pass-rate thresholds are not specified, and the filter relies on a model trained by the same pipeline.
assumptions (7)
  • standard math GRPO with group-relative advantage is a valid policy-gradient estimator for this setting.
    Section 2.1 builds on Shao et al. 2024; the paper does not re-derive GRPO, it treats it as a sound baseline.
  • domain assumption The fastText classifier reliably identifies the language of each response component after stripping LaTeX and code blocks.
    Section 2.2.4 uses fastText to enforce language consistency; no accuracy evaluation of the classifier on CoT text is provided.
  • domain assumption The rule-based math verifier (parsers plus SymPy) correctly decides answer equivalence for all retained problems.
    Section 2.2.2 states the verifier normalizes and compares answers; failures would reward or punish the model incorrectly.
  • domain assumption Sampling 20 code tests per group gives a reliable correctness signal.
    Section 2.2.2 randomly selects 20 tests; if these tests are not representative, the reward is noisy.
  • ad hoc to paper Mid-generation weight updates in the asynchronous pipeline keep training effectively on-policy.
    Section 3 argues recomputing stale KV caches is unnecessary 'potentially due to off-policy corrections'; this is an unproven assumption.
  • ad hoc to paper Removing the KL penalty does not cause policy collapse when epsilon_high is tuned.
    Section 2.1 removes KL divergence entirely, contrary to standard GRPO; stability is asserted from runs, not from a formal argument.
  • ad hoc to paper Self-filtered training data (graded by a model trained with the same pipeline) is a valid distribution for learning.
    Section 4.1 uses the RL-trained model's consensus to remove supposedly incorrect ground truths; this could bias the training set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Magistral." pith.science (2026). https://pith.science/paper/YRPWNEAX

@misc{pith2026250610910,
  author       = {Pith},
  title        = {Pith review of: Magistral},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRPWNEAX}},
  note         = {Machine review of arXiv:2506.10910}
}
read the original abstract

We introduce Magistral, Mistral's first reasoning model and our own scalable reinforcement learning (RL) pipeline. Instead of relying on existing implementations and RL traces distilled from prior models, we follow a ground up approach, relying solely on our own models and infrastructure. Notably, we demonstrate a stack that enabled us to explore the limits of pure RL training of LLMs, present a simple method to force the reasoning language of the model, and show that RL on text data alone maintains most of the initial checkpoint's capabilities. We find that RL on text maintains or improves multimodal understanding, instruction following and function calling. We present Magistral Medium, trained for reasoning on top of Mistral Medium 3 with RL alone, and we open-source Magistral Small (Apache 2.0) which further includes cold-start data from Magistral Medium.

Figures

Figures reproduced from arXiv: 2506.10910 by the authors.

Figure 1
Figure 1. Performance of Magistral Medium on common reasoning benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Magistral’s system prompt. The system prompt spells out the format and language guidelines for the model. The same system prompt is utilized for both mathematical and coding problems. 3 Infrastructure In this section, we present our infrastructure for online training. We adopt a distributed RL training system similar to those proposed in several prior works [Espeholt et al., 2018, Hu et al., 2024, Noukhovitch et al.… view at source ↗
Figure 3
Figure 3. Online training pipeline. 1) Generators continuously output completions to prompts from input data sources. 2) Whenever a completion is finished, it is sent to the appropriate verifier. 3) Each sequence is sent to a different data parallel group using a pre-set permutation until every data parallel group has enough sequences to form a batch. 4) A single gradient step is performed and the trainer and generators are u… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Performance of Magistral Small compared with different training setups on various benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Impact of batch and minibatch sizes on RL training rewards. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Results for training with different advantage normalizations in GRPO. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Reward and length evolution in w(α1, α2) hyperplane. Black arrow trajectory is a projection of intermediate checkpoints of Magistral Small RL-only run on the hyperplane. Black points are perturbed checkpoints computed using Equation 2. Intermediate values are computed …
Figure 9
Figure 9. Figure 9: Reward scaling with output length. Each point corresponds to a perturbed checkpoint computed with Equation 2. We generate 8192 completions with the checkpoint and evaluate mean output length and raw reward (reward without length penalty). We perform linear regression o…
Figure 10
Figure 10. Figure 10: Performance on multimodal benchmarks. trained on text-only data, one might expect the multimodal performance to degrade. However, on the contrary, we discover that the models not only retain their multimodal capabilities, but unexpectedly develop enhanced multimodal r…
Figure 11
Figure 11. Figure 11: Binary vs proportional reward for code problems. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Impact of εhigh on the entropy distribution throughout training. (a) Entropy evolution throughout training of a 3B model on a math only dataset. Entropy drops with entropy bonus, while higher εhigh maintains entropy, allowing for better exploration. (b) Entropy evolut…
Figure 13
Figure 13. Figure 13: Benchmark performance of Magistral Medium fine-tuned on open-source traces. [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: A physics multimodal problem and its solution generated by Magistral Medium. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: A chemistry multimodal problem and its solution generated by Magistral Medium. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: A biology multimodal problem and its solution generated by Magistral Medium. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    LLM first-answer accuracy on procedural arithmetic drops from 61% on 5-step tasks to 20% on 95-step tasks, with frequent failures including skipped steps, premature answers, and hallucinated operations.

  2. Simultaneous Speech-to-Speech Translation Without Aligned Data

    cs.CL 2026-02 conditional novelty 7.0 of 10

    Hibiki-Zero performs simultaneous speech-to-speech translation without word-level aligned data, using sentence-level supervision plus GRPO reinforcement learning with BLEU-based process rewards, and reports state-of-t...

  3. rStar2-Agent: Agentic Reasoning Technical Report

    cs.CL 2025-08 conditional novelty 7.0 of 10

    A 14B model trained with agentic RL and a resample-on-correct rollout strategy scores 80.6% on AIME24 and 69.8% on AIME25, nearly matching DeepSeek-R1 (671B) in one week on 64 GPUs.

  4. Flow Matching Policy Gradients

    cs.LG 2025-07 conditional novelty 7.0 of 10

    FPO trains flow-based policies with PPO by replacing the likelihood ratio with an exponentiated flow matching loss difference.

  5. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

  6. TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    TSHA is a new 80,000-pair benchmark for indoor safety hazard assessment; current vision-language models score roughly 45-85, and fine-tuning on TSHA raised Qwen2.5-VL-3B by 18.3 points on TSHA's test set.

  7. A Novel Hierarchical Multi-Agent System for Payments Using LLMs

    cs.MA 2026-02 conditional novelty 6.0 of 10

    HMASP, a four-level hierarchy of LLM agents, handles end-to-end payment workflows in simulation, reaching 95-99% task success with the strongest open-weight model.

  8. Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Reasoning in LLMs helps only for complex 27-class emotion recognition and hurts simple binary sentiment, across seven model families and 504 configurations.

  9. Are Large Reasoning Models Interruptible?

    cs.CL 2025-10 conditional novelty 6.0 of 10

    Interrupting large reasoning models or changing the problem mid-thought sharply degrades accuracy, with up to 60% drops and distinct failure modes: reasoning leakage, panic, and self-doubt.

  10. Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Training a Qwen2.5-7B model with GRPO on math and puzzle data improves both domains, code transfer depends on the starting model, and template or reward mismatches sharply hurt performance.

  11. Contrastive ESA: Human Evaluation of Multiple Translations at Once

    cs.CL 2026-07 reject novelty 5.0 of 10

    cESA presents multiple translations side-by-side for error-span scoring; k=3 is reported as optimal, but the baseline comparison is misattributed and the stability evidence is inconsistent.

  12. Cost of Reasoning in non-English Languages: A Case Study on Japanese

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Japanese reasoning-language control is feasible with CPT plus GRPO, but incurs a capability cost and does not free-improve cultural Japanese performance.

  13. LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics

    cs.AI 2026-02 conditional novelty 5.0 of 10

    A live benchmark auto-extracts self-contained lemmas from recent arXiv papers and finds top LLMs solve only 10–15% at pass@1.

  14. The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling

    stat.ML 2026-02 conditional novelty 5.0 of 10

    Temperature scaling is the unique accuracy-preserving linear recalibrator, and per-step tempering of a toy LLM can make sequence entropy non-monotonic.

  15. Hermes 4 Technical Report

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Hermes 4 releases three open-weight reasoning models (14B, 70B, 405B) trained with synthetic data and a length-control SFT stage, evaluated on mathematics, code, knowledge, and alignment benchmarks.

  16. Confidence-Weighted Token Set Cover for Early Hypothesis Pruning in Self-Consistency

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A weighted set cover algorithm prunes intermediate hypotheses during self-consistency, cutting token use by 10 to 35 percent across five LLMs on three math benchmarks while keeping accuracy.

  17. MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization

    cs.CL 2025-07 conditional novelty 4.0 of 10

    MiroMind-M1 open-sources a two-stage SFT plus RLVR recipe with a new context-aware multi-stage policy optimization (CAMPO) that claims competitive AIME24, AIME25, and MATH500 scores among Qwen-2.5-based models.

  18. Trustworthy AI in Digital Health: A Comprehensive Review of Robustness and Explainability

    cs.AI 2026-08 conditional novelty 3.0 of 10

    A survey of robustness and explainability methods for digital health AI, proposing a taxonomy and illustrating known XAI tools, without new empirical or theoretical results.

  19. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

31 extracted references · 7 canonical work pages · cited by 19 Pith papers

  1. [1]

    What matters in on-policy reinforcement learning? a large-scale empirical study, 2020

    Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? a large-scale empirical study, 2020. URL https://arxiv.org/abs/2006.05990

    arXiv 2020

  2. [2]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  3. [3]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407--1416. PMLR, 2018

    work page 2018

  4. [4]

    Polyglot Benchmark

    Paul Gauthier. Polyglot Benchmark . https://github.com/Aider-AI/polyglot-benchmark, 2024. URL https://github.com/Aider-AI/polyglot-benchmark. GitHub repository. Coding problems sourced from Exercism language tracks

    work page 2024

  5. [5]

    Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, ...

    arXiv 2025

  6. [6]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    arXiv 2021

  7. [7]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024

    arXiv 2024

  8. [8]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. URL https://arxiv.org/abs/2503.24290

    arXiv 2025

Show all 31 references
  1. [9]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face . Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  2. [10]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  3. [11]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  4. [12]

    Fasttext.zip: Compressing text classification models

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H \'e rve J \'e gou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016

    2016 arXiv

  5. [13]

    Visualizing the loss landscape of neural nets, 2018

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets, 2018. URL https://arxiv.org/abs/1712.09913

    2018 arXiv

  6. [14]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  7. [15]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  8. [16]

    Mistral large 2

    MistralAI. Mistral large 2. https://mistral.ai/news/mistral-large-2407, 2024

    2024

  9. [17]

    Mistral medium 3

    MistralAI. Mistral medium 3. https://mistral.ai/fr/news/mistral-medium-3, 2025

    2025

  10. [18]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint arXiv:2410.18252, 2024

    2024 arXiv

  11. [19]

    Codeforces cots

    Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. https://huggingface.co/datasets/open-r1/codeforces-cots, 2025

    2025

  12. [20]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  13. [21]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  14. [22]

    Iterative methods for sparse linear systems

    Youcef Saad. Iterative methods for sparse linear systems. SIAM, 2003

    2003

  15. [23]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  16. [24]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  17. [25]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  18. [26]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025

    2025 arXiv

  19. [27]

    Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale llm trainin

    Bo Wu, Sid Wang, Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, et al. Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale llm trainin. arXiv preprint arXiv:2505.24034, 2025

    2025 arXiv

  20. [28]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  21. [29]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  22. [30]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of ACL, 2025

    2025

  23. [31]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.