{"id":"171b39ac-7e9b-44c2-bac5-c0da0b4b23c6","arxiv_id":"2506.10910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Pure RL alone, without distilled reasoning traces, turned Mistral's base models into strong reasoning models on math and coding benchmarks.","lead":"Mistral trained its first reasoning models with pure reinforcement learning instead of copying reasoning traces from other AI models. The smaller model's weights are open source, and the paper reports gains on math and coding benchmarks plus preserved image understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination step is documented for the 700k math / 35k code training pools; if AIME-24 or LiveCodeBench items appear in training, the headline '~50% AIME-24 gain from pure RL' would reflect training-set overlap rather than reasoning generalization.","rationale":"The reader's weakest-assumption centered on verifier/reward fidelity and consensus filtering. That is a real risk, but the AIME-24 result itself is a strong sanity check on verifier quality: a badly wrong verifier would likely not produce 73.6 pass@1 on a clean external exam. The more acute, load-bearing risk is that the benchmark is not external to training. The paper describes very large, loosely sourced pools (700k math samples, code contest data) and never states that evaluation benchmarks were removed. Public AIME items are a natural component of such pools, and LiveCodeBench problems are by construction contest problems. This concern does not require assuming bad faith; it is an unstated necessary condition for the headline result. It is also independently suggested by the consensus filter in Section 4.1, which can remove hard problems and inflate apparent learnability. A contamination audit is cheap and decisive. The paper's other strengths—disclosed failed approaches, released Small weights, and the PCA/length analysis—are real but do not bear on this specific condition.","tokens_in":17238,"tokens_out":6538,"duration_ms":74575,"concrete_test":"Request or independently compute exact and near-duplicate overlap between the training pools in Sections 4.1 and 4.2 and the evaluation sets in Section 5.1 (AIME-24, AIME-25, LiveCodeBench v5/v6, MATH-500, GPQA, HLE), using normalized string matching and embedding retrieval. Then recompute Table 2 pass@1 on the non-overlapping subset only. If any overlap exists and the non-overlap gains shrink by more than about 5 points on AIME-24, the 'pure RL' generalization claim must be restated as possibly benchmark-contaminated; if overlap is zero, the claim stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Section 5.2 is that pure RL, without cold-start traces, roughly doubles AIME-24 pass@1 (26.8 to 73.6) and lifts LiveCodeBench v5 by ~30 points. This conclusion requires that the evaluation benchmarks are held out from the RL training data. Section 4.1 describes a 699k-to-38k math pool and Section 4.2 a 35k code-contest pool; the curation removes format/difficulty outliers and consensus-disagreeing answers, but the text never reports excluding AIME, LiveCodeBench, GPQA, or MATH from these pools. AIME problems are exactly 'numerical answer' math problems that the verifier is built to grade, and the code pool is drawn from contest data, the same genre as LiveCodeBench. Without a documented decontamination step, the reported AIME-24 gain is consistent with memorization or overfitting rather than transferable reasoning. The Section 4.1 Stage-2 filter—removing problems where the model's majority answer disagrees with the reference—adds further bias: it preferentially deletes hard, model-inconsistent problems that a held-out benchmark would expose. Thus the manuscript's condition for the strong claim is not yet met.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Magistral (arXiv:2506.10910) introduces Mistral's first reasoning models, Magistral Medium and Magistral Small, trained with a scalable reinforcement learning (RL) pipeline built in-house. Medium is trained with pure RLVR on top of Mistral Medium 3, without cold-start reasoning traces, and is reported to raise AIME'24 pass@1 from 26.8 to 73.6 and LiveCodeBench v5 from 29.1 to 59.4. Small is a 24B model first SFT'd on traces from Medium and then further trained with RL; its weights are released under Apache 2.0. The paper details GRPO modifications (removal of KL penalty, loss normalization, relaxed upper clip, filtering of zero-advantage groups), an asynchronous distributed training infrastructure, two-stage difficulty filtering for math and code data, multilingual AIME evaluations, ablations on batch size and advantage normalization, analyses of weight-space dynamics, and evidence that RL on text preserves or improves multimodal and instruction-following capabilities.","tokens_in":17574,"tokens_out":5557,"duration_ms":63888,"significance":"If the reported gains are genuine, the paper provides strong evidence that pure RLVR at scale, without distillation or cold-start traces, can produce large improvements on math and coding benchmarks for a capable instruct model. It also contributes useful negative results (proportional code rewards, entropy bonus terms) and an unusually candid set of ablations, including batch-size and advantage-normalization studies. The open-source release of Magistral Small under Apache 2.0 and the detailed infrastructure description are valuable for reproducibility and follow-up work. The central quantitative claims, however, hinge on the assumption that the evaluation benchmarks were excluded from the training data, and this assumption is not documented in the manuscript. Because the data curation sections describe training pools drawn from exactly the same genres as AIME and LiveCodeBench, the absence of a decontamination analysis is a load-bearing gap that prevents full confidence in the headline numbers.","major_comments":[{"comment":"The manuscript reports that pure RL on Mistral Medium 3 yields a ~50% AIME'24 pass@1 gain (26.8 to 73.6) and a ~30-point LiveCodeBench v5 gain (29.1 to 59.4), but it never states that AIME-24/25, LiveCodeBench, MATH-500, GPQA, or HLE were excluded from the ~700k math and 35k code training pools. The curation in Sections 4.1 and 4.2 filters for format, difficulty, and answer consistency, but no decontamination step (exact-match, near-duplicate, or n-gram overlap analysis) is described. Since AIME problems are exactly numerical-answer math problems, and the code pool is drawn from contest data of the same genre as LiveCodeBench, the reported gains are currently consistent with training-set overlap rather than reasoning generalization. The authors should add a decontamination analysis, reporting the number of training examples matching each benchmark and re-evaluating the models on the deduplicated or held-out subsets.","section":"Sections 4.1, 4.2, and 5.2"},{"comment":"The second-stage filter removes problems where a majority of the RL-graded model's samples agree on a final answer that disagrees with the reference, on the rationale that such problems likely have incorrect ground truths. This criterion also removes hard problems that the model consistently fails, thereby biasing the training distribution toward easier or model-aligned items. If benchmark problems are present in the training pool, this filter would preferentially delete hard benchmark items, inflating measured performance on those exact problems. The paper should quantify how many benchmark problems are affected by this filter and provide evidence (e.g., manual inspection or a validated subset) that the removed problems are indeed mislabeled rather than merely difficult for the current model.","section":"Section 4.1, Stage-2 difficulty filter"},{"comment":"For Magistral Small, the SFT checkpoint used as the initial policy for the subsequent RL stage was selected as 'the best checkpoint on AIME'24' among four training epochs. This selection on the target evaluation benchmark, combined with the reported AIME'24 results for the final SFT+RL model, means the Small AIME'24 numbers are optimistically biased and do not cleanly measure the pipeline's generalization. The authors should select the checkpoint using a separate validation set (e.g., MATH or a held-out subset of the SFT data) and report the AIME'24 results for that checkpoint, or at least disclose the selection procedure explicitly and discuss its impact on the headline AIME'24 comparison.","section":"Section 5.3"},{"comment":"The paper reports averages over 64 runs for AIME and 16 runs for LiveCodeBench but provides no standard errors or significance tests. Several comparative claims are within a plausible noise range: for example, in Table 3, SFT versus SFT+RL on AIME'24 maj@64 goes from 90.0 to 83.3 (a decrease), and GPQA moves from 68.8 to 68.2; in Table 2, Magistral Medium's AIME'24 pass@1 of 73.6 is compared against DeepSeek-R1's 79.8 with no variance information. Adding confidence intervals or at least reporting the per-run dispersion would materially strengthen the validity of the stated comparisons.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The statement that the loss is shown 'with all modifications highlighted in red' is not visible in the printed version; please use a different convention, such as equation annotations or a textual list of the modifications.","section":"Section 2.1, final GRPO equation"},{"comment":"The benchmark is referred to as 'the MATH dataset' in the text and 'MATH-500' in the tables; please use consistent terminology throughout.","section":"Section 5.1"},{"comment":"The multilingual AIME 2024 versions are described only as 'created by translating the questions'; the translation process, the translator model or human efforts, and any quality-control steps should be described, and ideally the translated benchmarks should be released for reproducibility.","section":"Section 5.4"},{"comment":"The code verifier samples 20 tests per problem within each response group, but the paper does not discuss how sensitive the reward signal is to this random test selection; a brief analysis or a fixed-seed policy would be helpful.","section":"Section 2.2.2, code correctness"},{"comment":"The claim that recomputing the key-value cache is unnecessary 'potentially due to off-policy corrections inherent to the loss function' is an empirical assertion without direct evidence; please either cite a supporting ablation or soften the claim.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to attract substantial attention given the strong results and the open-source release of Magistral Small. The decontamination gap is the primary blocking issue: the manuscript's central claim of a 'nearly 50% accuracy increase' on AIME'24 from pure RL is not fully supported unless training-set overlap can be ruled out. I would encourage the editor to treat the requested decontamination analysis as a condition for acceptance rather than a minor revision point. The checkpoint-selection issue for Magistral Small is also important but secondary. The authors' willingness to report failed experiments is a plus, and the infrastructure details appear genuinely useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely informative RLVR paper, but the headline number—73.6 on AIME-24 from pure RL—has a hole in the reporting: Section 4 describes the math and code pools in detail but never says those pools were decontaminated against AIME, LiveCodeBench, MATH, or GPQA. Since AIME problems are exactly the \"numerical answer\" style the verifier is built for, that omission matters. The reader's stress-test note lands; the concern is not manufactured.\n\nWhat's new and good: the paper shows you can take a capable instruct model, run GRPO with a few well-motivated tweaks (no KL, Clip-Higher, length penalty, zero-advantage group filtering), and get a large jump on math and code without any cold-start reasoning traces. The language-consistency reward that forces CoT to match the user's language is simple and looks effective. The infrastructure section is refreshingly concrete, and the unsuccessful experiments (proportional rewards, entropy bonus) are honestly reported. Releasing Magistral Small under Apache 2.0 gives people something to actually run. Those are real contributions.\n\nSoft spots, in order of severity. First, no documented decontamination of the 699k math / 35k code pools. The paper says the pools were filtered by format and difficulty, but never says AIME-24 or LiveCodeBench problems were removed. Given the verifier's reliance on rule-based numeric answers and contest-style code, the reported gains could be partly memorization. This is the single biggest thing I'd want resolved before trusting the numbers. Second, the best SFT checkpoint for Magistral Small was selected on AIME-24; that's a mild selection leak and should be stated as such. Third, tables don't have error bars or significance tests; for AIME at least they do 64 runs, so variance could be reported. Fourth, the multilingual AIME translations were made in-house; a quick external check would help.\n\nThat said, the central empirical pattern—pure RL helping both a large and a 24B model, and text-only RL preserving multimodal abilities—is credible and consistent with the ablations. I don't think the paper is wrong; it's under-documented at exactly the point where a benchmark claim needs documentation.\n\nThis paper deserves a serious referee. Send it out, but require the authors to report decontamination details and add variance information before publication. I'd also suggest they release at least the data pipeline code, not just the model.","headline":"Solid and transparent RLVR report, but the headline AIME gain lacks a documented decontamination step; referee should require it.","tokens_in":18578,"tokens_out":2459,"would_cite":true,"duration_ms":26919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pure reinforcement learning, with no cold-start reasoning traces, lifts a capable instruct model to strong math and coding performance, while preserving or improving other capabilities.","keywords":["reinforcement learning","verifiable rewards","reasoning models","large language models","GRPO","multimodal reasoning","data filtering","language consistency"],"falsifier":"Run the same RL pipeline under two controls: (1) replace the correctness reward with a random binary reward while keeping format, length, and language rewards; and (2) hold out the problems that the consensus filter removed (where the model's majority answer disagreed with the reference) and test the final model on them. If the random-reward run shows similar AIME gains, the reported improvements are not driven by verifiable correctness; if the held-out problems are solved poorly, the filter deleted valid hard problems and data curation, not RL, is carrying the result.","tokens_in":17056,"feed_emoji":"🧠","tokens_out":8814,"duration_ms":84184,"temperature":0.7,"pith_summary":"This paper tries to show that a reasoning model can be built with pure reinforcement learning from verifiable rewards, with no cold-start reasoning traces distilled from an existing reasoning model. The authors report that their RL pipeline alone lifts pass@1 accuracy on the AIME'24 math benchmark from 26.8% to 73.6%, and on LiveCodeBench v5 from 29.1% to 59.4%, starting from a capable instruct model. They also claim that training on text-only problems preserves or improves multimodal understanding, instruction following, and function calling, and that pure RL works for a 24-billion-parameter model, contradicting earlier reports that small models need distillation to compete. If true, the result matters because it makes reasoning capability a product of scalable RL and verifiable rewards rather than of access to proprietary reasoning traces, and it suggests RL on text can strengthen capabilities across modalities.","feed_headline":"Pure RL lifts AIME-24 from 26.8% to 73.6%","feed_subtitle":"Text-only training also improves multimodal and coding results, and works for a 24B model.","key_machinery":"The central object is the verifiable-reward training loop: an online, asynchronous GRPO system in which generators continuously produce completions, a rule-based math verifier (parsers plus SymPy) or a sampled 20-test C++/Python code runner assigns correctness rewards, and a soft length penalty plus a fastText language-consistency reward shape behavior. The algorithm removes the KL divergence penalty, uses a relaxed upper clip threshold ($\\epsilon_{\\text{high}}$) to preserve exploration, normalizes the loss by total generation length, normalizes advantages per minibatch, and drops groups with zero advantage. The argument runs on this loop: staged difficulty filtering (a weaker model first, then an RL-trained grader) and consensus-based ground-truth cleaning supply problems that are hard but solvable, while the RL loop converts increased completion length and reward into benchmark gains.","core_discovery":"The central claim is that an end-to-end online RLVR pipeline, built from scratch with the authors' own models and infrastructure, is sufficient to turn an instruct model into a strong reasoner. On AIME'24, pass@1 rises from 26.8 to 73.6 and majority voting reaches 90.0; on AIME'25 pass@1 rises from 21.2 to 64.9; LiveCodeBench v5 rises from 29.1 to 59.4. The paper attributes these gains to a specific recipe: GRPO with the KL penalty removed, Clip-Higher exploration, length- and advantage-normalized losses, zero-advantage group filtering, staged data difficulty with an RL-trained grader, and a language-consistency reward that keeps thoughts and answers in the user's language. The paper further claims that RL on text preserves or improves multimodal, instruction-following, and function-calling abilities, and that pure RL on a 24B model reaches performance comparable to SFT on distilled traces, with RL on top of SFT giving the best result.","pith_inferences":["If verifiable-reward quality is the true driver, the same recipe should transfer to other domains with automated checkers (e.g., formal verification, unit-test-heavy software engineering, structured data extraction), and improving verifier coverage should compound the gains.","The consensus-based ground-truth cleaning implies a non-trivial fraction of benchmark labels are wrong; a testable consequence is that evaluating on a cleaned label set would raise scores for many RL-trained models, not just this one.","The reported cross-modal transfer (text-only RL improving MMMU/MMMU-Pro) suggests reasoning skill is partly modality-agnostic; injecting a small amount of vision-language verifiable data could test whether the effect saturates or compounds.","The paper's recipe differs from earlier pure-RL reports in several ways (no KL, Clip-Higher, two-stage data filtering, group filtering); ablating these one at a time on a fixed base model would reveal which ingredient is responsible for the small-model gains."],"forward_implications":["Reasoning capability can be trained with verifiable rewards alone, removing the need to license or reproduce proprietary reasoning traces for cold-start.","Text-only RL acts as a general capability booster: multimodal benchmarks (MMMU, MMMU-Pro) improve even though training is text-only.","Pure RL is viable for smaller models, so the recipe transfers beyond frontier-scale checkpoints, and combining distillation SFT with RL yields the strongest small model.","The staged difficulty and consensus-based data cleaning suggest that dataset curation and verifier quality are as important as the RL algorithm itself."],"supporting_citations":[{"why":"Provides the RLVR recipe and the R1-Zero baseline that pure RL is compared against and contradicted.","marker":"[DeepSeek-AI et al., 2025]"},{"why":"Introduces GRPO, the group-relative policy optimization algorithm the pipeline modifies.","marker":"[Shao et al., 2024]"},{"why":"Supplies the Clip-Higher exploration strategy, length penalty, and no-KL practice adopted in the loss.","marker":"[Yu et al., 2025]"},{"why":"Motivates the minibatch advantage normalization used in the final loss.","marker":"[Andrychowicz et al., 2020]"},{"why":"Provides the fastText classifier that enforces the language-consistency reward.","marker":"[Joulin et al., 2016]"}],"fun_headline_variants":["No distillation: pure RL boosts reasoning to 73.6% on AIME-24","Ground-up RL turns instruct model into reasoner: AIME-24 73.6%","RL on text only: AIME-24 jumps to 73.6%, multi-modal intact","From 26.8% to 73.6%: pure RL reasoning without SFT traces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training signal rests on the automated verifiers and the filtered dataset: if the rule-based math parser and SymPy comparison, the 20 sampled code tests, and the consensus-based removal of supposedly wrong ground truths do not track true solution quality, the model optimizes against a distorted target and the benchmark gains will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["No distillation: pure RL boosts reasoning to 73.6% on AIME-24","Ground-up RL turns instruct model into reasoner: AIME-24 73.6%","RL on text only: AIME-24 jumps to 73.6%, multi-modal intact","From 26.8% to 73.6%: pure RL reasoning without SFT traces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4038,"prompt_tokens":898,"completion_tokens":3140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":3041}},"tokens_in":514,"tokens_out":3140,"duration_ms":25139,"temperature":1.0,"reasoning_tokens":3041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:13:56.642180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same RL pipeline under two controls: (1) replace the correctness reward with a random binary reward while keeping format, length, and language rewards; and (2) hold out the problems that the consensus filter removed (where the model's majority answer disagreed with the reference) and test the final model on them. If the random-reward run shows similar AIME gains, the reported improvements are not driven by verifiable correctness; if the held-out problems are solved poorly, the filter deleted valid hard problems and data curation, not RL, is carrying the result.","supporting_citations":[],"review_version":1}