{"id":"8418c99c-9d4e-4db1-bc71-755f368d096a","arxiv_id":"2608.06243","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DASH reweights token-level supervision in on-policy self-distillation using sequence-adaptive propagation gates, improving math reasoning accuracy on AIME 2024, AIME 2025, and HMMT at all three tested Qwen3 scales.","lead":"This paper introduces DASH, a method that re-weights token-level supervision in on-policy self-distillation of reasoning models using the temporal sequence of teacher-student divergences. It reports consistent accuracy gains over standard OPSD on three math benchmarks at three model scales, with no additional teacher or student forward passes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-within-200-step checkpoint selection on the test benchmarks is the load-bearing weak point; verify with a fixed or held-out checkpoint protocol.","rationale":"The reader's weakest assumption matches my own: the reported improvements rely on selecting, for each seed, the checkpoint that maximizes test-benchmark accuracy, which is in-sample selection. The manuscript itself flags this in Section 5.1, Appendix B.3, and Appendix C, where it explicitly distinguishes the protocol from held-out validation and disclaims statistical significance. Those self-flagged limitations are exactly the load-bearing issue. The method has independent support that should be credited: a parameter-free derivation, released code, four-seed means for OPSD and DASH, and ablations including fixed-lambda, inverse-gate, and scale-matched controls. None of this removes the selection-bias risk, because the reported scores are maxima over ten checkpoints and the DASH-specific gains at 4B and 8B are small: 1.40 and 1.60 macro-average points, with per-benchmark deltas as low as 0.6 points. The proposed check is cheap because checkpoints are already saved and requires no new training. The verdict should remain conditional: accept the mechanism and the ablation evidence, but require a fixed-checkpoint or held-out-validation comparison before treating the uniform-improvement claim as established.","tokens_in":20648,"tokens_out":4551,"duration_ms":43247,"concrete_test":"Recompute all Table 1 entries from the already-saved checkpoints using the step-200 checkpoint with no selection, and report DASH minus OPSD for each of the nine benchmark-model settings together with the distribution of selected steps t*_s for both methods. If the uniform nine-setting improvement shrinks or flips under this fixed-checkpoint rule, the claimed gains are an artifact of best-within-200-step reporting; if DASH still improves in the large majority of settings and the gains exceed seed-level noise, the checkpoint-selection concern does not land.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.1 and Appendix B.3 define the reporting protocol: for each seed, the checkpoint in {20, 40, ..., 200} that maximizes the unweighted average on AIME 2024, AIME 2025, and HMMT February 2025 is selected, and all three benchmark scores are taken from that single checkpoint. Because this is also the final evaluation set, the protocol is in-sample model selection with ten candidates per seed. The central claim that DASH improves over matched vanilla OPSD on every benchmark at all three scales depends on this selection not systematically favoring DASH. That is not guaranteed: DASH changes the training loss and therefore can have different checkpoint-to-checkpoint variance and different peak timing than OPSD. Under argmax selection, a method with noisier checkpoints or later peaks receives a larger upward bias, and with per-benchmark gains as small as 0.6 points (Qwen3-4B HMMT, Qwen3-8B AIME 2025) and only four training seeds, the uniform direction could flip under a predetermined checkpoint rule. The paper is transparent about this choice, explicitly calling it 'best-within-200-step reporting rather than held-out validation selection,' but transparency does not remove the load: the headline comparison is presented as a fixed-training-budget improvement, and the selection protocol is part of that comparison. Appendix C also states that no statistical significance is claimed and does not report paired seed-level OPSD-versus-DASH differences, so it is impossible to tell from the paper whether DASH wins in most seeds or only in the selected-checkpoint means.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DASH (Divergence-Adaptive Supervision Horizons), a modification of on-policy self-distillation (OPSD) for training reasoning language models. Standard OPSD averages token-level forward-KL distillation losses with uniform coefficients; DASH replaces these with sequence-conditioned coefficients. After clipping each local divergence, DASH computes a sigmoid gate from the gap between the local clipped signal and its sequence-level mean, then applies a backward recursion A_t = r_t + λ_t A_{t+1} to produce a weighted loss whose token coefficients depend on the realized discrepancy history. The authors provide a fixed-horizon gradient decomposition (Prop. 1) as structural motivation, clarifying that DASH does not estimate the score-function term in that decomposition. Experiments on Qwen3-1.7B/4B/8B on AIME 2024, AIME 2025, and HMMT February 2025 report that DASH improves over the authors' matched OPSD reruns on every benchmark at every scale, with ablations separating the benefit of adaptive gates from fixed multi-step aggregation, gate direction, coefficient scale, divergence choice, sensitivity κ, and vocabulary support. The paper is transparent about its experimental protocol: checkpoints are selected within the 200-step budget by maximizing the unweighted average on the three test benchmarks, which the authors call 'best-within-200-step reporting rather than held-out validation selection.'","tokens_in":21084,"tokens_out":4769,"duration_ms":42957,"significance":"If the reported improvements are robust, DASH is a valuable, low-cost contribution: it requires no additional teacher or student forward pass, is simple to implement on top of OPSD, and its ablations (Inverse-gap, fixed λ, scale-matched controls, normalized DASH) provide credible evidence that the gains come from the sequence-conditional coefficient allocation rather than from trivial scale effects. The formal gradient decomposition is correct and clearly scoped, and the release of code plus four-seed matched OPSD reruns is a strength. However, the central quantitative claim of uniform improvement over OPSD rests on a checkpoint-selection rule that uses the test benchmarks as the selection set, and the main hyperparameter κ is also chosen on the same test benchmarks. Because several per-benchmark gains are small (0.6–1.4 points) relative to the four-seed standard deviations (0.3–1.3 points), the in-sample selection can materially bias the comparison. The paper is transparent about this protocol, but transparency does not remove the need for a fixed-checkpoint or held-out-validation check.","major_comments":[{"comment":"The reporting protocol selects, for each seed, the checkpoint in {20, 40, ..., 200} that maximizes the unweighted average on the three final benchmarks (Eqs. 38–40). Because these benchmarks are also the evaluation set, every reported score is a maximum over ten candidates. The abstract's claim that DASH 'improves over our matched vanilla OPSD reruns on every benchmark at all three scales' therefore depends on this argmax selection not systematically favoring DASH. That is not guaranteed: if DASH has higher checkpoint-to-checkpoint variance or a later performance peak than OPSD, the selection bias can differ between methods. With per-benchmark gains as small as 0.6 points (Qwen3-4B HMMT 2025, Table 1), a fixed checkpoint rule (e.g., step 200) or a held-out-validation rule could plausibly flip the direction on individual benchmarks. Please report the OPSD-vs-DASH comparison at a predetermined checkpoint and under a validation-selected checkpoint, with seeded paired differences.","section":"Section 5.1 and Appendix B.3"},{"comment":"The propagation sensitivity κ is selected as the value (κ=5) achieving the highest macro-average on the three test benchmarks. Because the same benchmarks are used for final reporting, this is in-sample hyperparameter tuning on the test set. The claim that every tested κ outperforms OPSD is robust to this issue, but the headline configuration's advantage over OPSD is inflated by the fact that the hyperparameter was chosen to maximize the reported target. The paper should either fix κ a priori, select it on a held-out validation split, or show the OPSD comparison for each κ value (Table 8 partially does this) and state explicitly that κ was not tuned on the test set.","section":"Section 5.4 (Figure 4a)"},{"comment":"The four-seed results are reported as per-benchmark means and standard deviations, but no paired seed-level OPSD-versus-DASH differences are given. The central 'every benchmark at every scale' claim relies on uniformity across nine benchmark-scale settings, and the standard deviations (0.30–1.30) are comparable to several per-benchmark gains (e.g., HMMT 2025 on 4B: +0.6 points; AIME 2025 on 8B: +0.6 points). Without a seed-level paired table or a paired summary (e.g., the number of seeds for which DASH beats OPSD on each benchmark, or a sign test), the reader cannot assess whether the uniform improvement is consistent or driven by a subset of seeds. Please add the per-seed paired deltas and a simple paired analysis.","section":"Appendix C"}],"minor_comments":[{"comment":"The metric Avg@12 is first used in Section 5.1 without a definition; define it in the preliminaries or before its first use in the experimental setup.","section":"Section 3 / Section 5.1"},{"comment":"The clipped signal r_t is used in Eq. (9) and described in Algorithm 1, but it would improve readability to define r_t explicitly before Eq. (9), since the main text introduces d_t in Eq. (2) and the clipped variant only appears later.","section":"Section 4.2, Eq. (9)"},{"comment":"The comment that individual forward-KL summands can be negative and hence r_t is not guaranteed non-negative is important; consider highlighting this in the main text where clipping is introduced, because readers may assume r_t is a standard non-negative KL divergence.","section":"Appendix A.4"},{"comment":"The caption states that the top three distinct results are highlighted with 1st/2nd/3rd superscripts, but these superscripts are not visible in the text rendering of the table; please ensure they appear in the published artifact or replace them with explicit bold/color formatting.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its checkpoint-selection protocol and provides a serious ablation suite, which is commendable. The key reservation is that the headline quantitative claim is not yet supported at a fixed training budget: the best-within-200-step selection on the test benchmarks, combined with κ selected on the same benchmarks, introduces an unknown amount of selection bias that could alter the uniform-improvement conclusion. If the authors add a fixed-checkpoint or held-out-validation comparison and the improvement persists, the contribution would be solid for this venue. The external baselines from Zhao et al. should be treated as context, not as a matched comparison, and the text mostly respects that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine, honestly reported incremental method. The headline claim—DASH beats matched OPSD everywhere—is probably overstated because the checkpoint selection is done on the test set, but the ablation structure suggests the effect is real, not a selection artifact.\n\nWhat's new: DASH replaces uniform token-level KL weights in OPSD with gates computed from the divergence gap relative to the sequence mean, using a backward recursion. The fixed-horizon gradient decomposition (Prop. 1) is a clean, correct statement: the score-function term retains only future divergences via the zero-conditional-mean property. They use it honestly as motivation only. The ablation suite is the paper's best feature: fixed-λ controls, an inverse-gap control that reverses the gate sign, and a 2×2 separating coefficient allocation from average scale. Inverse-gap underperforming DASH by ~3 points under the same selection rule is good evidence the direction of the mapping matters, and the allocation-vs-scale split shows the gains come primarily from the relative profile. Four seeds, stds, released code, and transparent description of the checkpoint protocol all help.\n\nThe soft spots: the best-within-200-step selection on the three test benchmarks, with ten candidates per seed, is in-sample model selection. Gains as small as 0.6 points (8B HMMT, 4B AIME25) and only four seeds mean the uniform 'beats OPSD everywhere' claim is fragile; a fixed checkpoint rule or held-out validation could shift it. The same test set is used for κ and τ (though κ-robustness is shown). EOPD/AVSD/PW-OPSD are single-seed reruns, so those comparisons are display. No significance testing is claimed, which is honest, but it limits the strength of the headline.\n\nI partly disagree with the stress-test framing: if the selection protocol merely rewarded noisier checkpoints, inverse-gap and fixed-λ—run under the identical rule—should see similar upward bias. They don't. So the method is likely not a mirage; its magnitude is just optimistic. The paper deserves a serious referee. I'd ask for the selected checkpoint step per seed and one held-out validation run before trusting the numbers.","headline":"A genuine, honestly reported incremental method with a strong ablation suite; test-set checkpoint selection makes the headline gains look optimistic, but the effect survives the obvious controls and the paper deserves a serious referee.","tokens_in":21524,"tokens_out":3695,"would_cite":true,"duration_ms":32013,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DASH reweights token-level distillation signals by the realized divergence history along each rollout, and the authors report it improves over their matched OPSD reruns on every benchmark at all three model scales without any additional…","keywords":["on-policy self-distillation","temporal credit assignment","token-level supervision weighting","adaptive propagation gates","sequence-relative divergence gaps","mathematical reasoning benchmarks","reinforcement learning with verifiable rewards","Qwen3"],"falsifier":"Rerun the matched OPSD-versus-DASH comparison on Qwen3-1.7B selecting checkpoints on a validation set disjoint from AIME 2024, AIME 2025, and HMMT February 2025, or at a fixed step budget, and compare four-seed macro-averages; the central claim fails if DASH's advantage over the matched OPSD rerun falls below roughly one point or inverts. A supporting diagnostic is to compare the per-seed variance and the distribution of selected checkpoint step indices: if DASH's selected checkpoints are systematically later or more variable across seeds than OPSD's, the best-within-200-step rule is the likely source of the reported gain.","tokens_in":20456,"feed_emoji":"⚖️","tokens_out":11067,"duration_ms":78525,"temperature":0.7,"pith_summary":"This paper argues that on-policy self-distillation (OPSD), which trains reasoning models with dense token-level teacher–student supervision, leaves the temporal structure of each rollout unused because it averages local divergences with a uniform coefficient. The same local divergence value $d_t$ can follow very different discrepancy histories, and standard OPSD cannot tell those contexts apart. The proposed method, DASH, converts each token's divergence gap from the sequence mean into an adaptive propagation gate and aggregates backward, so that supervision weights depend on how divergences evolved before each token. The authors report that on three mathematical reasoning benchmarks and three Qwen3 model scales, DASH improves over their matched OPSD reruns on every setting, raising four-seed macro-averages by 1.4 to 3.2 points with no additional teacher or student forward pass. If the result holds, dense distillation can be made markedly more sample-efficient by reweighting signals that are already being computed.","feed_headline":"Sequence-aware reweighting beats OPSD on all nine settings","feed_subtitle":"DASH shapes each token's supervision weight by its divergence history, at zero extra forward-pass cost","key_machinery":"The carrying mechanism is the adaptive propagation gate $\\lambda_t$ together with backward multi-step aggregation. At each position, the local forward-KL divergence $d_t = D_{\\mathrm{KL}}(\\pi_t^T \\| \\pi_t^S)$ between the privileged teacher and the student is capped per vocabulary item at $\\tau = 0.05$ to form the local signal $r_t$; DASH centers this signal by the sequence mean $\\bar{r}$ and converts the gap into a detached gate $\\lambda_t = \\operatorname{sg}[\\sigma(-\\kappa(r_t - \\bar{r}))]$ with $\\kappa = 5$. The backward recursion $A_T = r_T$, $A_t = r_t + \\lambda_t A_{t+1}$ makes later signals decay multiplicatively through the intervening gates, so each position's effective coefficient follows $c_1 = 1$, $c_k = 1 + \\lambda_{k-1} c_{k-1}$ and is a function of the realized divergence path that precedes it. Because the gates are detached, gradients flow only through the local signals, and the added work is a scalar backward scan under 1% of step time: the method reweights the direct distillation gradient rather than introducing a new learning signal.","core_discovery":"The central claim is that the uniform coefficient profile of vanilla OPSD is a genuine bottleneck and that a sequence-conditioned profile improves reasoning accuracy on every benchmark and scale tested. The paper first establishes a motivating fact about on-policy autoregressive generation: at a reasoning fork, two rollouts can share the same local divergence and then diverge sharply in future divergence and in terminal outcome, so the local scalar alone cannot determine how much supervision that token should carry. DASH's response is to clip the local forward-KL contributions into a signal $r_t$, compute the gap $g_t = r_t - \\bar{r}$ to the sequence mean, and map it through a detached sigmoid gate $\\lambda_t = \\sigma(-\\kappa g_t)$ that stays more open after below-average divergence and more closed after above-average divergence, before running the backward recursion $A_t = r_t + \\lambda_t A_{t+1}$. The component ablations attribute the gain primarily to this discrepancy-conditioned allocation: fixed multi-step aggregation, the reversed gate direction, and uniform coefficient scaling each recover only part of the improvement. The authors state that DASH reuses the teacher and student distributions OPSD already computes, so the reported gains require no additional teacher or student forward pass.","pith_inferences":["The gate mechanism only requires a per-token scalar trajectory, so the same sequence-relative-gap reweighting could be transplanted onto other dense signals, such as process-reward scores, per-token advantages, or entropy-based weights, without re-deriving the method.","Because the headline numbers use best-within-200-step reporting on the test benchmarks, the gain available under a strict held-out protocol is probably smaller than 3.20 points; quantifying that gap would separate the value of the reweighting policy from the value of favorable checkpoint selection.","The paper's motivating observation, identical local divergence with divergent futures, implies that the local KL is not a sufficient statistic for downstream mismatch; a gate that conditions on a short divergence history instead of just the deviation from the sequence mean is a natural testable next step.","The appendix's GRPO hybrid shows the aggregation idea transfers to outcome-level RL with scale-dependent results (67.30 vs. 66.40 at 8B), suggesting DASH composes with existing reward-weighting schemes rather than replacing them."],"forward_implications":["On Qwen3-1.7B, DASH raises the four-seed OPSD macro-average from 41.87 to 45.07, with gains of 1.40 points at 4B and 1.60 points at 8B, and it beats the matched OPSD rerun on every individual benchmark at every scale.","The gain comes with no extra forward passes: DASH reuses the teacher and student distributions OPSD already computes, adding only a scalar backward aggregation that costs under 1% of step time.","The improvement is not explained by fixed multi-step aggregation, reversed gate direction, or uniform coefficient scaling, so the discrepancy-conditioned allocation itself is the operative ingredient.","Forward KL is the effective local signal for the gates: symmetric JSD and reverse KL score 6.84 and 3.60 macro-average points lower on Qwen3-1.7B, and compressing the vocabulary to the top-100 teacher tokens plus a tail loses only 0.70 points while a top-1 approximation loses 10.24 points."],"supporting_citations":[{"why":"Supplies the on-policy distillation framework and forward-KL local loss that DASH extends.","marker":"Agarwal et al. 2024"},{"why":"Defines the OPSD protocol and privileged-teacher conditioning, and provides the Base, SFT, and GRPO reference numbers DASH is compared against.","marker":"Zhao et al. 2026"},{"why":"Frames the on-policy autoregressive mismatch that motivates treating divergence histories as informative.","marker":"Ross, Gordon, and Bagnell 2011"},{"why":"Establishes the scheduled-sampling contrast for how supervision of on-policy generation should be allocated.","marker":"Bengio et al. 2015"},{"why":"Provides the temporal-difference lineage for the backward propagation of local signals.","marker":"Sutton 1988"},{"why":"Supplies the multi-step backward aggregation structure that DASH's recursion mirrors.","marker":"Schulman et al. 2016"},{"why":"Grounds the KL-distillation supervision signal the method reweights.","marker":"Hinton, Vinyals, and Dean 2015"},{"why":"PW-OPSD is the strongest competing baseline, using predefined position-dependent weights that DASH contrasts with sequence-adaptive weights.","marker":"Liu et al. 2026a"},{"why":"Supplies the OpenThoughts-Math-30K training set used in all reruns.","marker":"Guha et al. 2025"},{"why":"Provides the GRPO outcome-reward baseline and the RLVR paradigm DASH builds on.","marker":"Shao et al. 2024"}],"fun_headline_variants":["DASH: history-aware distillation reweights tokens, no extra passes","Adaptive divergence gates fix OPSD sparsity, beat it everywhere","Sequence-conditioned supervision weights outperform vanilla OPSD","Zero-cost adaptive reweighting lifts reasoning across all scales"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported uniform improvements rely on a checkpoint-selection protocol under which, for each training seed, the checkpoint that scores best on the three test benchmarks themselves is chosen; if that in-sample selection helps DASH more than OPSD, the gains may not hold under a held-out selection protocol at the same training budget.","fun_headline_variants_meta":{"raw":{"variants":["DASH: history-aware distillation reweights tokens, no extra passes","Adaptive divergence gates fix OPSD sparsity, beat it everywhere","Sequence-conditioned supervision weights outperform vanilla OPSD","Zero-cost adaptive reweighting lifts reasoning across all scales"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1611,"prompt_tokens":1113,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":729,"tokens_out":498,"duration_ms":4920,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:33:53.948334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the matched OPSD-versus-DASH comparison on Qwen3-1.7B selecting checkpoints on a validation set disjoint from AIME 2024, AIME 2025, and HMMT February 2025, or at a fixed step budget, and compare four-seed macro-averages; the central claim fails if DASH's advantage over the matched OPSD rerun falls below roughly one point or inverts. A supporting diagnostic is to compare the per-seed variance and the distribution of selected checkpoint step indices: if DASH's selected checkpoints are systematically later or more variable across seeds than OPSD's, the best-within-200-step rule is the likely source of the reported gain.","supporting_citations":[],"review_version":2}