{"id":"3f2c19ac-5da7-4fe8-b6c4-84b0b998e3b4","arxiv_id":"2507.19766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A segment rollout plus dynamic masking of confident positive tokens lets a 30B-A3B reasoning model beat a 235B-A22B model on AIME2025 and BeyondAIME after 128k-token RL training.","lead":"Ultra-long RL training with 128k-token outputs is made practical by splitting decoding into segments and by masking already-mastered positive tokens when entropy drops. On math benchmarks, a 30B-parameter model trained this way outperforms a 235B model, and training runs about 2x faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract and reader's strongest_claim overstate the 128k result: the 85.1/61.9 scores belong to the Yarn-extended 140k variant, not the 128k model.","rationale":"The reader's weakest_assumption identifies the POIS estimator's bias as the key risk. That is a real methodological concern, but it is not the most load-bearing issue for the central claim as stated. Even if POIS is biased, the empirical benchmark results could still hold; the paper presents POIS as a heuristic that works better than SAIS, and the final performance does not hinge on the estimator being unbiased. The more direct threat to the central claim is that the flagship numbers in the Abstract (85.1/61.9) are taken from a variant with Yarn-based length extension, not from the 128k model the Abstract describes. This is an internal inconsistency that can be verified by reading Table 2, and it is the number the reader's strongest_claim repeats. The comparison with Qwen3-235B remains positive even after correction, so the qualitative conclusion survives, but the magnitude is materially smaller than claimed. This strengthens the case for a conditional verdict, not a rejection: the paper should correct the misreported numbers and unify the evaluation protocol before a stronger verdict can be reached.","tokens_in":9912,"tokens_out":5239,"duration_ms":58096,"concrete_test":"Re-read Table 2 and the Abstract, and confirm whether the 85.1/61.9 entries correspond to the row labeled \"UloRL-A3B-128k-Yarn\" (with Yarn length extension to 140k) rather than \"UloRL-A3B-128k\" (82.8/60.2). If so, recompute the comparison against Qwen3-235B using the 128k row; the revised margins (82.8 vs 81.5 on AIME2025, 60.2 vs 59.0 on BeyondAIME) still show a positive gap but with much smaller magnitude. The Abstract and strongest_claim should then be corrected to either cite the 128k row or explicitly state that the 85.1/61.9 results include Yarn extension.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim as stated in the Abstract and repeated in the reader's strongest_claim is that \"RL training with 128k-token outputs\" improves Qwen3-30B-A3B to 85.1% on AIME2025 and 61.9% on BeyondAIME. Table 2, however, shows those exact numbers only for \"UloRL-A3B-128k-Yarn,\" a variant that additionally applies Yarn to extend the output length to 140k (Section 4.2, paragraph 3). The actual 128k result, \"UloRL-A3B-128k,\" is 82.8% on AIME2025 and 60.2% on BeyondAIME. Thus the headline numbers are inflated by 2.3 and 1.7 points, and the claim that the 128k model \"surpasses Qwen3-235B-A22B\" (81.5/59.0) should be evaluated against 82.8/60.2, not 85.1/61.9. This is an internal inconsistency between the Abstract and the experimental table, not a matter of interpretation. The strongest_claim inherits the error. Because this is the paper's flagship quantitative result, the load-bearing comparison is overstated as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UloRL, a collection of RLVR training techniques for long-output reasoning: segment rollout with two importance-sampling variants (SAIS and POIS), dynamic masking of well-mastered positive tokens (DMMPTs), a generative verifier model, data cleaning and transformation, and a simple overlong-punishment rule. Experiments on Qwen3-30B-A3B report a 2.06x training speedup with four segments at 64k output, and Table 2 reports AIME2025/BeyondAIME improvements from 70.9/50.7 to 82.8/60.2 for the 128k model, with 85.1/61.9 for an additional YaRN-extended 140k variant. The abstract and introduction attribute the higher numbers to the 128k model, which is inconsistent with Table 2. The paper also presents entropy-dynamics plots for token masking and one ablation of DMMPTs.","tokens_in":10235,"tokens_out":6741,"duration_ms":68441,"significance":"If the results hold, the paper demonstrates a practical recipe for RLVR at very long output lengths and offers a plausible mechanism for entropy stabilization via masking of high-confidence positive tokens. The direct speedup measurement in Table 1 is a concrete and reproducible contribution, and the planned release of code and models would help the community verify and build on the work. The main caveats are the abstract/table mismatch, the unanalyzed bias of POIS, and the mixed evaluation protocol; none of these destroys the core empirical finding, but they must be addressed before the claims can be accepted as stated.","major_comments":[{"comment":"The abstract and Section 1 state that RL training with 128k-token outputs raises AIME2025 from 70.9% to 85.1% and BeyondAIME from 50.7% to 61.9%. Table 2, however, reports those exact numbers only for the UloRL-A3B-128k-Yarn variant, which additionally extends the output length to 140k via YaRN (Section 4.2, paragraph 3). The plain 128k model, UloRL-A3B-128k, scores 82.8% and 60.2%. The headline comparison to Qwen3-235B-A22B is therefore stated with inflated numbers, even though the 128k values still surpass the baseline. Please correct the abstract and introduction, and make the 128k vs. 140k distinction explicit wherever the result is summarized.","section":"Abstract; §1; §4.2, Table 2"},{"comment":"The POIS estimator sets π_θ_old to the current rollout policy for all segments, forcing every importance ratio to 1. For segments generated by earlier policies, this is not an on-policy estimator; it is an unweighted off-policy update with no importance correction and no bias analysis. The paper attributes the entropy stability and accuracy advantage of POIS over SAIS to the benefits of on-policy training, but the comparison conflates the estimator bias with the rollout scheme. Since POIS is the only importance-sampling variant used in the final UloRL experiments, please provide a bias bound, a correction term, or an experiment that isolates the estimator (for example, comparing POIS against SAIS at the same 128k setting, or measuring the gradient difference on held-out segments).","section":"§3.1.2, POIS"},{"comment":"The evaluation protocol is not uniform across the rows used for the headline comparison. The table footnote indicates that metrics marked with an asterisk are from the authors' evaluation, while unmarked metrics are from official reports; for the SOTA baselines this mixes official AIME-2025 numbers with self-evaluated BeyondAIME numbers, whereas all UloRL numbers are produced by the authors' pipeline. Because the claim that UloRL surpasses Qwen3-235B-A22B depends on this comparison, please either rerun all baselines under the same avg@32 protocol or restrict the claim to comparisons that use a single protocol.","section":"§4.2, Table 2"},{"comment":"The contribution attribution is underdetermined. Table 2 ablates only DMMPTs (UloRL-A3B-w/o-DMMPTs), while the generative verifier, data cleaning/transformation, and overlong-punishment rule are never individually ablated. The POIS-vs-SAIS comparison in Figure 2 is also run at 4k, 32k, and 64k output lengths, not at the final 128k configuration. Please add at least one ablation for the verifier and one for the data-cleaning pipeline, or soften the conclusion that each component is necessary for the reported gains.","section":"§4.2, Table 2; §3.3–§3.5"}],"minor_comments":[{"comment":"The introduction contains several typos: 'Pesudo On-Policy Importance Sampling' should be 'Pseudo', 'welll-Mastered' should be 'well-Mastered', and 'duiring' should be 'during'.","section":"§1"},{"comment":"In the SAIS importance ratio, the denominator is written as π_{θ_old^{f(i)}}(o_{i,t} | q, o_{i,<|s_i|}); the context subscript appears inconsistent with the token index t and with Equation (6). Please define f(i) and the context notation precisely.","section":"Eq. (8)"},{"comment":"The column header 'A VG' appears broken; it should read 'Avg' or 'Average'.","section":"Table 2"},{"comment":"The algorithm does not specify how many gradient updates are performed per rollout step, how the experience pool is sampled for the mini-batch of size 1024, or how samples with different segment counts are weighted in the loss. Please add these details for reproducibility.","section":"§3.1.1, Algorithm 1"},{"comment":"The hardware, distributed configuration, and total training steps/tokens are not reported; these details are needed to contextualize the 2.06x speedup and the overall cost of the 128k runs.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The abstract/table mismatch and the POIS bias question are the two points that most need addressing before I could recommend acceptance. If the authors can provide a uniform evaluation protocol and an ablation or analysis for POIS, the empirical contribution is likely to be solid. I do not see a circularity problem: the hyperparameters τ and σ are hand-chosen and the main results are external benchmark measurements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll skip the pleasantries. The useful part of this paper is the segment rollout for long-output RL: dividing 128k-token decoding into 16k segments, letting finished samples train immediately, and reporting a 2.06x speedup with four segments. That measurement is direct and I have no reason to doubt it. The DMMPTs entropy-control idea is also worth taking seriously; the entropy plots show the mechanism does what it claims, and the \"don't train tokens you already know\" reasoning is clear.\n\nWhat is not solid is the paper's headline comparison. The abstract says RL with 128k-token outputs gets 85.1 on AIME2025 and 61.9 on BeyondAIME, and the reader's strongest claim repeats that. Table 2 shows those are the Yarn-extended 140k variant's numbers. The plain UloRL-A3B-128k row is 82.8 and 60.2. That still beats Qwen3-235B-A22B (81.5/59.0), but the margin is roughly half what the abstract implies. This is an internal inconsistency, not a matter of interpretation; it needs a correction.\n\nThe other soft spots are more standard. POIS, as the paper admits, is an off-policy estimator that pretends all earlier segments are on-policy, setting the importance ratio to 1 and removing clipping for those tokens. No bias bound is given. It might be fine in practice, but the paper's own entropy argument depends on it, so it deserves an analysis or at least an empirical check against SAIS at final benchmark numbers. The evaluation protocol mixes official and self-evaluated baselines, so the cross-model comparisons should be re-run under one protocol. Only one ablation is reported (removing DMMPTs), and there is no sensitivity analysis for tau=0.99 or sigma=0.2. The generative verifier is introduced but never evaluated in isolation. Code and model are promised but the paper doesn't let us check them.\n\nWho gets value from this? Anyone building long-output RL systems: the segment rollout speedup is directly usable, and the MPT observation is a plausible contribution to the entropy-collapse discussion. It deserves serious refereeing, but the authors need to fix the abstract/table mismatch, run a uniform evaluation, and provide proper ablations before I'd trust the benchmark claims.","headline":"The segment rollout speedup and DMMPTs are the real contributions; the abstract's headline 128k numbers are actually the Yarn variant's, not the plain 128k model's.","tokens_in":10742,"tokens_out":3175,"would_cite":true,"duration_ms":37700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on 128k-token outputs with segmented rollouts and entropy masking lifts a 30B model's reasoning scores above a 235B model's.","keywords":["ultra-long output reinforcement learning","segment rollout","pseudo on-policy importance sampling","dynamic masking of well-mastered positive tokens","entropy collapse","generative verifier","mathematical reasoning benchmarks","Qwen3-30B-A3B"],"falsifier":"Run the identical 128k-token recipe on the same 30B model but replace POIS with a consistent off-policy estimator that keeps each segment's true behavior policy, holding all other hyperparameters fixed; if the AIME2025 and BeyondAIME gains over the base model shrink substantially, or the entropy curves stop matching the on-policy case, the reported benefits are attributable to the biased estimator rather than to the segment-rollout scheme itself.","tokens_in":9700,"feed_emoji":"🧠","tokens_out":10775,"duration_ms":106227,"temperature":0.7,"pith_summary":"The paper argues that the practical ceiling on reinforcement learning for reasoning is not the model but the training harness: ultra-long outputs bottleneck batches and the policy collapses into repetition. Its proposed recipe, UloRL, splits 128k-token decoding into 16k-token segments, uses a pseudo on-policy importance-sampling estimator to keep training stable, and masks tokens the model already masters when entropy dips. If the reported experiments hold, these changes make 128k-token RL training about twice as fast per step and lift a 30B-parameter model's AIME2025 score from 70.9% to 85.1%, above a 235B-parameter model.","feed_headline":"128k-token RL lifts 30B model past a 235B one","feed_subtitle":"Segment rollout and masking already-mastered tokens make ultra-long reasoning outputs trainable","key_machinery":"The machinery is a staged training loop. Segment rollout divides the 128k-token maximum into eight 16k-token chunks, so samples that finish early enter the experience pool immediately while unfinished ones continue in later iterations; this is what yields the 2.06x speedup at four segments. Pseudo On-policy Importance Sampling (POIS) then recomputes every token's importance weight as if all segments came from the current policy, forcing every ratio to 1 and disabling clipping, which the paper says stabilizes entropy. Dynamic Masking of well-Mastered Positive Tokens (DMMPTs) masks any token with predicted probability above $\\tau=0.99$ in a reward-1 sample whenever the current sequence entropy falls below a target $\\sigma=0.2$. A generative verifier judges whether a predicted answer is equivalent to the reference, and a data-cleaning pipeline removes multi-part, trivial, or mislabeled questions; together these supply the reward signal.","core_discovery":"The central claim is that entropy collapse in long-output RL comes primarily from overtraining positive-sample tokens the model already predicts with high probability, and that the right fix is to mask those tokens dynamically rather than add an entropy bonus or reweight samples. Combined with segment rollout, which lets finished samples train immediately instead of waiting for the longest samples in a batch, and with a generative verifier that replaces brittle string matching for answer equivalence, the paper reports that a 30B model trained with 128k-token responses surpasses a 235B reference model on AIME2025 and BeyondAIME. The same ablation shows that removing the dynamic masking drops average accuracy by about 3.6 points, and increasing the output cap from 32k to 128k produces a monotone climb in benchmark scores.","pith_inferences":["Because POIS replaces the true behavior-policy probabilities of earlier segments with the current policy, its importance ratios are not unbiased estimates of the segment-rollout objective; the paper does not bound this bias, so a direct comparison against a consistent off-policy estimator would tell whether POIS's stability is a real property of the objective or an artifact of the estimator.","The two DMMPTs thresholds, $\\tau=0.99$ and $\\sigma=0.2$, are presented as fixed choices; transferring the recipe to other model families or reward schemes would likely require re-tuning them, and the paper does not characterize that sensitivity.","The headline gain bundles segment rollout, POIS, DMMPTs, the generative verifier, and data cleaning; separate ablations of the verifier and data pipeline would reveal how much of the 14-point AIME improvement is due to the RL algorithm alone.","If the monotone length-performance trend continues, context extension beyond 128k may give further gains, but the paper gives no evidence about where the curve saturates."],"forward_implications":["A 4-segment rollout makes each RL update 2.06x faster than a single-segment 64k rollout, so ultra-long outputs become affordable to train on.","Length itself appears to be a training axis: average accuracy on AIME2025 and BeyondAIME climbs from 62.9% at 32k outputs to 71.5% at 128k outputs.","DMMPTs is a load-bearing component: without it the same recipe averages 67.9% instead of 71.5%.","A 30B-parameter model trained this way reports 82.8% on AIME2025 against 81.5% for a 235B-parameter model, and the context-extended variant reaches 85.1%.","Parameter count is not the only route to stronger reasoning: the smaller trained model surpasses the much larger reference on both benchmarks."],"supporting_citations":[{"why":"Introduced partial rollouts for long outputs, the prior technique that segment rollout adapts.","marker":"Team et al., 2025"},{"why":"Defines GRPO, the group-relative clipped surrogate that UloRL starts from.","marker":"Shao et al., 2024"},{"why":"Supplies DAPO's dynamic sampling, token-level gradient loss, clip-higher, and overlong handling that UloRL adopts or modifies.","marker":"Yu et al., 2025"},{"why":"Argues positive-sample training drives entropy collapse, the claim against which UloRL's MPT hypothesis is framed.","marker":"Zhu et al., 2025"},{"why":"Reports that on-policy training has more stable entropy than off-policy training, motivating POIS.","marker":"He et al., 2025"},{"why":"Introduces generative verifiers as next-token-prediction reward models, which UloRL uses for answer-equivalence judgments.","marker":"Zhang et al., 2024"},{"why":"Documents the Qwen3 base models that UloRL trains and evaluates.","marker":"Yang et al., 2025"},{"why":"Provides the recipe UloRL follows to apply context extension for output lengths up to 140k.","marker":"An et al., 2025"},{"why":"Defines the YaRN context-extension method used in the 140k-token variant.","marker":"Peng et al., 2023"}],"fun_headline_variants":["Segment rollout and token masking unlock 128k-token RL training","Dynamic masking prevents entropy collapse in long-output RL","30B model tops 235B using 128k-token RL","Masking mastered tokens stabilizes ultra-long RL training","Ultra-long RL: mask mastered tokens to boost reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that pretending earlier rollout segments were produced by the current model, instead of by the older models that actually produced them, does not distort the training signal enough to change the outcome.","fun_headline_variants_meta":{"raw":{"variants":["Segment rollout and token masking unlock 128k-token RL training","Dynamic masking prevents entropy collapse in long-output RL","30B model tops 235B using 128k-token RL","Masking mastered tokens stabilizes ultra-long RL training","Ultra-long RL: mask mastered tokens to boost reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3297,"prompt_tokens":955,"completion_tokens":2342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2259}},"tokens_in":571,"tokens_out":2342,"duration_ms":17951,"temperature":1.0,"reasoning_tokens":2259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:01:19.495452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical 128k-token recipe on the same 30B model but replace POIS with a consistent off-policy estimator that keeps each segment's true behavior policy, holding all other hyperparameters fixed; if the AIME2025 and BeyondAIME gains over the base model shrink substantially, or the entropy curves stop matching the on-policy case, the reported benefits are attributable to the biased estimator rather than to the segment-rollout scheme itself.","supporting_citations":[],"review_version":1}