{"id":"4a72ada7-9f10-4265-b415-2f655d9ea277","arxiv_id":"2507.12507","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Prolonged RL with decoupled clipping, KL regularization, and periodic reference-policy resets improves a 1.5B reasoning model by 14.7% on math, 13.9% on coding, and 54.8% on logic puzzles relative to DeepSeek-R1-Distill-Qwen-1.5B.","lead":"This report trains a 1.5B language model with a long run of reinforcement learning across math, code, STEM, logic puzzles, and instruction following, and reports large gains over the DeepSeek-R1-Distill-1.5B baseline. A generalist might read it to see if a small model can improve with training recipe tricks, though the gains are tuned on validation sets drawn from the evaluation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation/evaluation overlap is the load-bearing risk: the Section 4.2 validation blend includes final benchmarks, and for logic puzzles the 100-per-task 'test' set is numerically the same size as the 9600 validation set; held-out evaluation is needed.","rationale":"I share the reader's assessment. The paper is a useful engineering report: the model is released, the verifiers are rule-based, and the KL/clipping ablations in Section 5 provide some controlled evidence for those components. The reset is less cleanly isolated, but the dominant threat to the central claim is the validation/evaluation overlap. Section 4.2 states plainly that the validation blend includes AIME2024, Codeforces, GPQA-diamond, IFEval, and graph_color, and the run log shows that resets and hyperparameter changes were made when validation metrics significantly degrade or plateau. That means the final model and the reported deltas are the output of a selection procedure whose objective includes the test sets. For logic puzzles, the arithmetic is exact: 96 tasks x 100 samples = 9600, so the 'reserved test' set of Section 4.3 has the same size as the '9600 validation samples' of Section 2.4; if these are the same items, the +54.8% logic gain is not a held-out number at all. This is not an internal inconsistency in the RL algorithm, but it is a correctness risk for the headline empirical claims. The public model makes a held-out check feasible, so the condition in the reader's verdict is appropriate. I would keep the CONDITIONAL verdict, with the condition being a held-out, validation-disjoint evaluation and, ideally, a final ablation that isolates the reference-policy reset from the other changes across Runs 1-8.","tokens_in":11859,"tokens_out":10148,"duration_ms":113244,"concrete_test":"Using the released checkpoint, run a held-out evaluation disjoint from the Section 4.2 validation blend: regenerate Reasoning Gym instances with fresh seeds not used in the 37K/9600 sets, and score AIME2025 and a later Codeforces window (neither in the validation blend) under the paper's settings (T=0.6, top_p=0.95, 16 samples). Recompute the reported deltas versus DeepSeek-R1-Distill-Qwen-1.5B; if the logic or math improvements shrink by more than ~30% relative, the headline gains are contaminated. A cheaper static check: diff the 9600 validation files from Section 2.4 against the 100-per-task 'test' files from Section 4.3; identical file sets would make the +54.8% a validation result by construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline improvements are not yet established as generalization results because the training recipe was selected using a validation blend (Section 4.2) that explicitly includes subsets of the final evaluation benchmarks: AIME2024, Codeforces, GPQA-diamond, IFEval, and graph_color. Every run-level decision—when to hard-reset the reference policy, when to change rollout count, context length, or reward shaping—is described as being triggered by this validation signal, so the reported deltas on those benchmarks are at least partly a selection artifact. The overlap is most direct for logic puzzles: Section 2.4 reports 9600 validation samples across 96 tasks, and Section 4.3 reports that 100 samples per task were reserved as test sets; 96 x 100 = 9600, so the +54.8% logic-puzzle gain appears to be measured on the same set used to monitor training and trigger resets. Since AIME2024, Codeforces, GPQA-diamond, and IFEval are also in the validation blend and the aggregate math/coding/STEM/instruction numbers are computed over them, the central quantitative claims cannot be separated from validation-set fitting without a held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on prolonged reinforcement learning of a 1.5B language model (DeepSeek-R1-Distill-Qwen-1.5B) across math, code, STEM, logical puzzles, and instruction-following tasks. The authors propose controlled KL regularization, decoupled clipping and dynamic sampling from DAPO, and periodic resets of the reference policy and optimizer states. They report average gains over the base model of +14.7% on math, +13.9% on coding, +54.8% on logic puzzles, +25.1% on STEM, and +18.1% on instruction-following, and competitive performance with domain-specialized 1.5B models. The paper includes ablation studies on temperature, clipping, dynamic sampling, entropy mitigation strategies, and reset.","tokens_in":12163,"tokens_out":8579,"duration_ms":83745,"significance":"If the results hold, this is a useful engineering contribution: it demonstrates that a 1.5B model can be trained for a long horizon on diverse verifiable-reward tasks with careful tuning, and it provides detailed training dynamics (KL, entropy, response length). The open-sourced model and the stage-by-stage recipe are valuable for reproducibility. The main weakness is that the training recipe is selected using a validation blend that overlaps with the final evaluation benchmarks, so the reported quantitative improvements are not yet established as generalization results.","major_comments":[{"comment":"The validation blend described in Section 4.2 includes subsets of AIME2024, Codeforces, GPQA-diamond, IFEval, and graph_color from Reasoning Gym, and Section 4.2 explicitly states that this validation signal is used to decide hard resets, rollout-count changes, context-window changes, and reward shaping. These same benchmarks (or subsets thereof) are used in the final evaluation in Section 4.3, so the reported gains on them are at least partly a selection artifact rather than evidence of generalization. The overlap is most direct for logic puzzles: Section 2.4 reports 9600 validation samples across 96 tasks (100 per task), and Section 4.3 reports reserving 100 samples per task as test sets, so the two sets are likely identical. Please report results on benchmarks that were not used for any training decision, and disclose explicitly which evaluation sets overlap with the validation blend.","section":"Section 4.2, 4.3"},{"comment":"The claim that periodic reference-policy resets are a critical component is not backed by a controlled ablation. The evidence in Figure 6 is a single extended training run that degrades on Codeforces, followed by a reset that also involves reinitializing optimizer states; in Runs 2, 4-7, resets are accompanied by changes in rollout count, context length, and reward shaping. This confound means the reader cannot attribute the recovery to the reset itself. A matched-compute comparison with and without reset, holding all other factors fixed, is needed to support the 'critical component' wording.","section":"Section 5.3"},{"comment":"All quantitative results come from a single training trajectory, and no error bars, confidence intervals, or multiple seeds are reported. The pass@1 estimates are computed from 16 samples per prompt, which on small benchmarks (e.g., AIME2024 has 30 problems) yields substantial variance. Without per-benchmark numbers with uncertainty, the aggregate improvements in the abstract and conclusion cannot be assessed for statistical significance.","section":"Section 4.4"},{"comment":"The ablations of decoupled clipping and dynamic sampling in Figure 5 appear to be performed on a small-scale AIME2024 validation setting, not on the final multi-stage recipe. The paper should state whether these conclusions transfer to the full recipe, and should provide error bars or multiple seeds for the reported 'clear improvements', since Figure 5(c) in particular shows only a single trajectory.","section":"Section 5.2"}],"minor_comments":[{"comment":"The reported math improvement is inconsistent: the abstract and conclusion state +14.7%, while Section 4.4 says 'average improvement of 15.7%'. Please reconcile.","section":"Abstract/Conclusion vs Section 4.4"},{"comment":"Reference [31] is cited for IFEval, but [31] is 'Online difficulty filtering for reasoning oriented reinforcement learning' and is not the IFEval paper; IFEval is [9]. Please correct the citation.","section":"Section 4.3"},{"comment":"There are several typos: 'levearge' should be 'leverage', 'penaly' should be 'penalty', and '4 8 x NVIDIA-H100-80GB nodes' should likely read '48 x NVIDIA-H100-80GB nodes'. Section 4.4 has 'context windo' for 'context window', and Section 3 intro has 'instatility' for 'instability'.","section":"Section 4.1"},{"comment":"Please clarify how pass@1 is estimated from 16 samples; state whether an unbiased estimator is used or whether it is the mean per-prompt success rate.","section":"Section 4.4"},{"comment":"The text says 'approximately 100 tasks' while Section 4.3 says 96 tasks; consider stating '96 tasks' consistently in both places.","section":"Section 2.4"},{"comment":"In the phrase 'properly higher temperatures enable broader behavioral exploration', 'properly' is unclear; consider 'appropriately higher temperatures'.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The validation-overlap issue is the main reason for major revision. The authors may be able to address it by reporting per-benchmark results on the non-overlapping benchmarks (e.g., AIME2025, AMC, MATH, Minerva, OlympiadBench, APPS, Codecontests, TACO, HumanevalPlus, LiveCodeBench) with confidence intervals. I would not reject outright because the engineering details and model release are valuable, and the overlap is disclosed (though not acknowledged as a limitation)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Reference-policy reset is the one genuinely new thing here, and it is worth taking seriously. The idea is simple: when KL divergence against a fixed reference spikes or validation plateaus, hard-reset the reference policy and optimizer states, and training continues to improve. The ablation in Section 5.3 shows that extending Run 1 without the reset degrades Codeforces validation scores, and a reset recovers them. That is concrete evidence, and the mechanism is plausible and new.\n\nThe rest of the recipe is a validated combination of known pieces—GRPO, DAPO-style decoupled clipping and dynamic sampling, a small KL penalty, high rollout temperature, and multi-domain data. The model is released, so the claims are checkable. To their credit, the authors are upfront that the model remains weak on ARC, cognition, and games, and they note that the temperature findings may be model-dependent.\n\nThe soft spot is the validation/evaluation overlap, and it is load-bearing. Section 4.2 says the validation blend includes subsets of AIME2024, Codeforces, GPQA-diamond, IFEval, and graph_color. The training narrative is intervention-driven: every stage change, reset, or hyperparameter adjustment is described as triggered by validation metrics. So the final numbers on those benchmarks are at least partly a selection artifact. The logic-puzzle case is the worst: 100 samples per task held out across 96 tasks is exactly 9600 samples, the same size as the 9600-sample validation set. That makes the +54.8% logic-puzzle gain indistinguishable from validation-fitting.\n\nSmaller issues: no error bars, a single training trajectory, and pass@1 estimated from 16 samples on small benchmarks is noisy. The abstract quotes +14.7% on math while Section 4.4 says 15.7%; minor but sloppy. The self-cited Nemotron instruction-following data is not a real problem since the central claim does not depend on it.\n\nNone of this kills the paper. The reset mechanism is novel and plausibly useful independent of the exact benchmark numbers, and the public model is valuable. But the headline improvements are not established as generalization results until there is a held-out evaluation on benchmarks disjoint from the validation blend. I would send this to peer review, but with a clear request to evaluate on disjoint benchmarks and report variance. It is an engineering report, not a proof.","headline":"The reference-policy reset is a genuinely useful idea, but the validation/evaluation overlap means the headline gains are not yet established as generalization.","tokens_in":12751,"tokens_out":3196,"would_cite":true,"duration_ms":29170,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that prolonged reinforcement learning, stabilized by a small KL penalty and periodic reference-policy resets, lifts a 1.5B model's reasoning across math, code, puzzles, STEM, and instruction-following to near-specialist…","keywords":["reinforcement learning","GRPO","verifiable rewards","KL regularization","reference policy reset","entropy collapse","reasoning models","small language models"],"falsifier":"Train the same 1.5B model with the same budget but monitor and schedule resets using a validation set disjoint from all final evaluation benchmarks; if the reported gains on math, code, and logic puzzles disappear or shrink to noise, the central claim that these techniques unlock long-term improvement is not supported.","tokens_in":11696,"feed_emoji":"🧠","tokens_out":6938,"duration_ms":74956,"temperature":0.7,"pith_summary":"This paper argues that prolonged reinforcement learning can keep improving a small language model across many reasoning domains, rather than plateauing early, if the training recipe includes a small KL-divergence penalty and periodic resets of the reference policy. Starting from a 1.5B reasoning checkpoint, the authors train on verifiable-reward tasks in math, code, STEM, logic puzzles, and instruction following, and report average gains over the base model of 14.7% on math, 13.9% on coding, 54.8% on logic puzzles, 25.1% on STEM, and 18.1% on instruction following. The result matters because it suggests that scaling training time on a modest model, with careful stabilization, can bridge the gap to domain-specialized systems.","feed_headline":"Prolonged RL lifts a 1.5B model's reasoning by up to 55%","feed_subtitle":"A 1.5B model trained for ~16k GPU-hours gains on math, code, puzzles, STEM, and instruction-following.","key_machinery":"The load-bearing mechanism is the reference-policy reset: every so often the KL anchor $\\pi_{\\text{ref}}$ is hard-reset to a recent snapshot of the online policy, and optimizer states are reinitialized. This prevents the KL penalty, whose coefficient is small ($\\beta = 10^{-4}$), from eventually dominating the loss and freezing learning, while still preserving the entropy-stabilizing effect of KL regularization against a fixed anchor. The other machinery is a GRPO objective with decoupled clipping bounds ($\\epsilon_{low}=0.2$, $\\epsilon_{high}=0.4$) and dynamic sampling that filters prompts where all rollouts succeed or all fail, which raises reward-signal density.","core_discovery":"The paper's central claim is that the limiting factor in RL for reasoning models is not model size but training stability over long horizons. With verifiable rewards, decoupled clipping, dynamic sampling, a small controlled KL penalty ($\\beta = 10^{-4}$), and periodic hard resets of the reference policy and optimizer states, a 1.5B model continues to improve across diverse tasks for roughly sixteen thousand GPU-hours. Compared with its starting checkpoint, the model gains 14.7% on math, 13.9% on coding, 54.8% on logic puzzles, 25.1% on STEM, and 18.1% on instruction-following, and remains competitive with two domain-specialized 1.5B models trained for single domains.","pith_inferences":["A fully blind reset schedule is the obvious next test: if the same gains appear without validation-guided intervention, the recipe is robust; if not, the evaluation blend is doing much of the work.","The staged training runs with human monitoring resemble an adaptive curriculum, so automating the detection of KL spikes or entropy collapse could make the method more reproducible.","Reference-policy reset is analogous to warm restarts in optimization and may reduce the need for very high rollout temperatures late in training."],"forward_implications":["Small-scale models can serve as test beds for RL recipes that later transfer to larger models, since stability techniques rather than scale drive the gains.","Removing KL regularization entirely, as some recent recipes do, may be wrong when starting from a strong chain-of-thought checkpoint; a small penalty can be needed.","Reference-policy resets give a practical intervention when validation plateaus or KL spikes, extending useful training without changing data.","Diverse verifiable-reward data across domains produces a generalist that matches single-domain specialists at the same size."],"supporting_citations":[{"why":"Establishes the verifiable-reward RL paradigm and the reasoning checkpoint lineage that supplies the base model.","marker":"[2]"},{"why":"Supplies the math training data, verifier, and the context-window-scaling training recipe the paper compares against.","marker":"[3]"},{"why":"Supplies decoupled clipping and dynamic sampling, the GRPO modifications adopted in the final recipe.","marker":"[4]"},{"why":"Supplies the coding RL dataset and the coding validation benchmarks used for evaluation.","marker":"[5]"},{"why":"Supplies the logic puzzle environments and their verifiers for both training and evaluation.","marker":"[7]"},{"why":"Defines GRPO, the base algorithm whose objective and group-relative advantage are used.","marker":"[10]"},{"why":"Serves as a domain-specialized coding baseline and is cited among recipes that remove the KL penalty.","marker":"[13]"}],"fun_headline_variants":["Prolonged RL lifts small model's reasoning by up to 55%","Training small models longer unlocks diverse reasoning gains","Stable RL training boosts 1.5B model's logic puzzles by 55%","Small model, long RL: 55% jump on logic puzzles, +14% math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training recipe's resets and hyperparameter choices were monitored against a validation blend that includes slices of the final evaluation benchmarks, so the reported gains may depend on peeking at the test distribution.","fun_headline_variants_meta":{"raw":{"variants":["Prolonged RL lifts small model's reasoning by up to 55%","Training small models longer unlocks diverse reasoning gains","Stable RL training boosts 1.5B model's logic puzzles by 55%","Small model, long RL: 55% jump on logic puzzles, +14% math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2353,"prompt_tokens":905,"completion_tokens":1448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1365}},"tokens_in":521,"tokens_out":1448,"duration_ms":12608,"temperature":1.0,"reasoning_tokens":1365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:45:08.129009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 1.5B model with the same budget but monitor and schedule resets using a validation set disjoint from all final evaluation benchmarks; if the reported gains on math, code, and logic puzzles disappear or shrink to noise, the central claim that these techniques unlock long-term improvement is not supported.","supporting_citations":[{"cited_title":"Tang, Manan Roongta, Colin Cai, Jef- frey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica","cited_arxiv_id":null,"evidence_quote":"Supplies the math training data, verifier, and the context-window-scaling training recipe the paper compares against."},{"cited_title":"Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the logic puzzle environments and their verifiers for both training and evaluation."},{"cited_title":"Deepcoder: A fully open-source 14b coder at o3-mini level","cited_arxiv_id":null,"evidence_quote":"Serves as a domain-specialized coding baseline and is cited among recipes that remove the KL penalty."}],"review_version":1}