{"id":"8395624c-b15a-4e87-9fa0-97d6de729798","arxiv_id":"2608.09447","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"WDL-OPD co-trains an anchor and an auxiliary policy through a shared geometric-mixture reverse-KL loss to a frozen teacher, yielding stronger Qwen3 math and code checkpoints and avoiding entropy-collapse failures seen in single-policy on-policy distillation.","lead":"This paper introduces WDL-OPD, a method that co-trains two language-model policies so a frozen teacher can guide training on the student's own outputs without the usual instability of on-policy distillation. On Qwen3 models it reports stronger math and code checkpoints at 1.7B and 4B scale, while explicitly noting the experiments are not fully controlled.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains are not yet attributable to mixture co-training: each scale–domain cell has a curriculum, initialization, compute, or checkpoint-selection confound, so the stabilization claim awaits the Appendix B matched control.","rationale":"The paper is unusually transparent: it states its limitations explicitly, reports negative runs, provides the exact algorithm, and scopes the claim as a stabilization hypothesis rather than a universal causal claim. The theoretical content, especially Proposition 1 and the frozen-auxiliary limit, is correct as far as it goes. The load-bearing concern is precisely that the experimental comparison does not isolate the proposed mechanism. The reader's weakest-assumption analysis identifies this same issue: the reported gains in Tables 3–6 could be driven by curriculum, initialization, compute, or checkpoint selection rather than by mixture-constrained co-training. I agree with that assessment. The proposed mechanism—that the moving auxiliary absorbs part of the correction—is plausible but untested, and the paper itself says endpoint accuracy alone cannot identify it. The Appendix B controlled matrix is the correct and concrete way to settle the question. Since the reader already arrived at CONDITIONAL and my analysis does not move that verdict, UNCHANGED is the appropriate recommendation.","tokens_in":9275,"tokens_out":3165,"duration_ms":29494,"concrete_test":"Run the Appendix B controlled matrix at 4B math: hold prompts, teacher, anchor initialization (the single-OPD checkpoint), rollout generator, top-k support, optimizer, step count, and total FLOPs fixed; compare standard OPD, frozen-auxiliary mixture, fully trainable WDL-OPD, and two independently trained policies without mixture coupling. Record anchor and auxiliary checkpoints at every evaluation step, with per-branch gradient norms and entropy traces. If WDL-OPD's best branch beats the frozen-auxiliary row and the independent two-policy row at matched budget, the attribution concern is resolved. If the independent two-policy control matches WDL-OPD, the gain is attributable to capacity or initialization, not mixture coupling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that mixture-constrained co-training produces the strongest student checkpoint in all four scale–domain cells—rests on the assumption that the shared-mixture loss, not the unmatched conditions, explains the gains. That assumption is untested because every cell contains a confound. Section 4.2 and Table 2 show that the 1.7B math single-policy stage uses a 5,662-prompt sweet-zone curriculum while the two-policy stage switches to the full 104,935-prompt pool. Table 2 and Appendix A.1 show that code anchors are initialized from already-trained checkpoints (GRPO step 120 at 1.7B; a trained code checkpoint at 4B), while the single-policy comparisons start from different points. Algorithm 1 trains two policies with separate optimizers, so capacity, optimizer state, and FLOPs are not matched. Section 5.3 and A.2 report the 4B code result at step 90, selected by the health rule, while later steps develop repetition; the stabilization claim is therefore partially a checkpoint-selection effect. Section 6 explicitly concedes that extra trainable capacity, different initializations, implicit ensembling in the loss, or checkpoint selection may account for part of the gain, and Section 7 states that these limitations 'prevent a universal causal claim.' The proposed optimization-buffer mechanism is never directly measured: no per-branch gradient norms, cosine similarities, parameter displacements, or state-distribution drifts are reported. Thus the strongest table claim is descriptive, not mechanistic. This is a real condition, not a fatal flaw, because Appendix B specifies the controlled matrix that would resolve the attribution question.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WDL-OPD, a two-policy on-policy distillation method in which an anchor policy generates rollouts, an auxiliary policy is trained on the same visited states, and the geometric mixture of their token distributions is matched to a frozen teacher by reverse KL. The authors derive the gradient structure of the mixture loss, prove that freezing the auxiliary yields an anchor-plus-contrast extrapolated target related to OPD2 and W2S-OPD, and argue that joint training creates branch-level degrees of freedom that can act as a stabilization buffer. The experimental record covers Qwen3-1.7B and Qwen3-4B on math and code, reporting that WDL-OPD produces the strongest recorded student checkpoint in each of four scale-domain cells, alongside negative single-policy OPD runs. The paper is explicitly scoped as supporting a stabilization hypothesis rather than a universal causal claim, and it provides an algorithm, a proof, and a list of the controlled ablations needed to test the hypothesis.","tokens_in":9479,"tokens_out":3293,"duration_ms":33152,"significance":"If the stabilization hypothesis is correct, this is a conceptually useful extension of on-policy distillation: it introduces a trainable second policy as an optimization buffer and shows that a static model-difference target cannot express the resulting branch-level freedom. The paper has several genuine strengths: Proposition 1 is stated and proved cleanly; Algorithm 1 and the implementation identity in Appendix A.3 make the exact loss and gradient paths checkable; the authors report negative runs and pathological trajectories instead of only favorable endpoints; and the development-set numbers are independently re-evaluated from preserved checkpoints. The honest limitation sections and the explicit list of required matched controls are also a credit. The main weakness is that the headline empirical claim rests on comparisons that differ in curriculum, initialization, compute, and checkpoint selection, so the stabilization mechanism itself is not yet evidenced. The paper is a well-framed report of a method and a hypothesis, not yet a controlled demonstration that the shared-mixture co-training, rather than confounds, explains the gains.","major_comments":[{"comment":"The 1.7B math comparison is not matched: the single-policy OPD stage uses a 5,662-prompt sweet-zone curriculum while the two-policy WDL-OPD stage switches to the full 104,935-prompt pool. Because RQ1 and the abstract claim 'strongest student checkpoint in each of four scale-domain settings' depend on this cell, the 6.4-point MATH500 improvement cannot be attributed to mixture-constrained co-training. The authors should either run WDL-OPD on the same 5,662-prompt curriculum, run single-policy OPD on the full pool, or remove the causal-sounding comparison from the central claim.","section":"Section 4.2, Table 2"},{"comment":"The 4B code claim is confounded by checkpoint selection: the reportable value is the step-90 checkpoint chosen by the health rule, while steps 100-109 develop repetition pathology, and the independent re-evaluation of the preserved checkpoint gives 0.637 versus an online peak of 0.669. The stabilization hypothesis cannot be distinguished from a checkpoint-selection story without reporting the full trajectory, the predeclared selection rule applied uniformly to single- and two-policy runs, and the terminal-checkpoint comparison. This is load-bearing for RQ3 because the same failure signature appears in the two-policy run after the selected checkpoint.","section":"Section 5.3, Appendix A.2"},{"comment":"The optimization-buffer mechanism is never directly measured. The paper lists the required diagnostics in Section 6 (per-branch gradient norms, cosine similarity, parameter displacement, entropy, top-k support mass, state-distribution drift) but reports none of them. Since the central claim is a stabilization hypothesis, and Section 6 concedes that extra trainable capacity, different initializations, implicit ensembling, or checkpoint selection may account for part of the gain, the absence of these measurements leaves the mechanism unsupported. The authors should provide at least a subset of these diagnostics in the revised version, or explicitly reframe the contribution as an empirical recipe without the stabilization mechanism.","section":"Section 6, Algorithm 1"},{"comment":"The compute and capacity budget is not matched between single- and two-policy runs: Algorithm 1 steps two optimizers, and the code anchor is initialized from an already-trained GRPO checkpoint at 1.7B and from a trained code checkpoint at 4B, while the single-policy comparisons start differently. The dual-compute control proposed in Section 6, in which two independent policies are trained toward the teacher without a shared mixture at matched FLOPs, is essential to attribute any gain to mixture coupling. Without that control, the four-cell performance claim could be explained by extra capacity and initialization alone; this is a load-bearing gap for the paper's central claim.","section":"Section 4.3, Appendix A.1"},{"comment":"The paper correctly identifies the needed controlled matrix but does not execute it, and Appendix B restates the same matrix as a plan rather than reporting results. For a journal submission whose title and abstract present WDL-OPD as a method, the absence of any matched row (standard OPD, OPD2, W2S-style proxy, frozen-auxiliary mixture, trainable WDL-OPD, and independent two-policy control under identical prompts, trajectories, initialization, and compute) means the central empirical claim is not yet testable from the reported data. The revision should include at least one scale-domain cell of this matrix, or the paper should be recast as a preliminary study rather than a method demonstration.","section":"Section 6, 'Controlled baseline matrix'"}],"minor_comments":[{"comment":"The phrase 'strongest student checkpoint in each of four scale-domain settings' should be qualified as 'strongest in the recorded, not fully matched, runs' even earlier in the abstract and in the Table 3 caption, since the later limitations already concede the confounds.","section":"Abstract and Table 3"},{"comment":"The dashes for missing anchor evaluations at steps 40-290 should be accompanied by a caption note that the anchor was not retained at those steps; Appendix A.2 explains this, but the main-text table is ambiguous at first glance.","section":"Table 4"},{"comment":"The sentence 'They point in the same mixed-logit direction' is accurate for the logit-space gradient, but it may mislead readers into thinking the parameter updates are aligned; a clarifying clause that the parameter gradients differ through the Jacobians of the two policies would improve precision.","section":"Equation (10)"},{"comment":"The table caption states that single- and two-policy stages use different training pools, but the main-text discussion of Table 5 should explicitly say that per-benchmark gains cannot be causally assigned to WDL-OPD because of the pool difference; this is already in Section 4.2 but the results section should restate it when interpreting the table.","section":"Table 5"},{"comment":"The first limitation sentence should be moved into the experimental design section, where the curriculum mismatch is first introduced, so that readers do not need to reach the limitations section to learn that the 1.7B math comparison is not matched.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest, well-structured, and technically sound in its derivation, but its empirical core is a set of unmatched comparisons followed by a plan for the matched matrix that the authors themselves identify as necessary. The revision must supply at least one controlled comparison cell or substantially narrow the claims to a preliminary record. I would not reject the paper, because the confounds are fixable within the project's scope and the authors have already specified the required experiment. However, acceptance in its current form would set a weak evidentiary bar for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nWhat should you know? This paper proposes a genuine variant of on-policy distillation: two trainable policies, an anchor that generates rollouts and an auxiliary that shares the same geometric-mixture reverse-KL loss to a frozen teacher. The moving auxiliary is the new piece; the paper explicitly shows that freezing it reduces to a known proxy-target form (OPD2/W2S-OPD). That is honest scoping, not a flaw.\n\nWhat's done well: the algorithm is completely specified, the algebra on the mixture loss and gradient split is correct, and negative runs are reported rather than hidden. The authors state their confounds plainly — different curriculum for 1.7B math, pre-trained anchors for code, two optimizers vs one, and a health-rule checkpoint selection — and they provide the controlled matrix that would resolve attribution. That is exactly how a stabilization hypothesis should be presented.\n\nSoft spots: the evidence does not yet support the load-bearing claim that mixture co-training, rather than confounds, drives the gains. Every scale–domain cell has at least one unmatched axis: curriculum, initialization, compute, or checkpoint selection. The proposed optimization-buffer mechanism is untested — no gradient norms, no per-branch diagnostics. The paper itself concedes this. So the contribution right now is a well-specified method with promising but non-attributable results. That is a conditional, not a fatal, situation.\n\nMy take: this deserves a serious referee. The idea has genuine novelty (a co-adapted auxiliary that can diverge from the anchor), the theory is clean, and the authors are unusually straight about what is and isn't established. The controlled matrix in Appendix B is the obvious next step; if it holds, the method is a real contribution. Until then, the reader should treat the headline numbers as recorded outcomes, not as evidence for the mechanism.\n\nRecommendation: send to peer review. It is an honest, well-scoped paper with a falsifiable central hypothesis and a concrete path to test it. I'd cite it for the formulation and the honest limitations section.","headline":"A clean, honestly scoped method paper: the moving auxiliary is new, the algebra is right, but the headline gains are not yet attributable to co-training — worth a serious referee, not a desk reject.","tokens_in":10187,"tokens_out":1539,"would_cite":true,"duration_ms":13773,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that making the auxiliary policy trainable, rather than freezing it as a static contrast, is what stabilizes on-policy distillation, and it reports the strongest recorded student checkpoint in every tested scale-domain…","keywords":["on-policy distillation","knowledge distillation","co-training","reverse KL","mixture-constrained loss","policy stability","language model post-training","Qwen3"],"falsifier":"Run the paper's required controlled matrix with prompts, trajectories, teacher, initialization, support, optimizer, update count, and total compute fixed, comparing fully trainable WDL-OPD against two independently trained policies with no shared mixture: if the independent-two-policy control matches or beats WDL-OPD, mixture coupling is not the cause of the gain. A cheaper check on the existing record: rerun the 1.7B math single-policy baseline on the full 104,935-prompt pool with the two-policy optimizer; if the 6.4-point gap closes, the curriculum switch explains the result.","tokens_in":8980,"feed_emoji":"🤖","tokens_out":10780,"duration_ms":83040,"temperature":0.7,"pith_summary":"On-policy distillation trains a student on its own rollouts, so every update changes the states where the next teacher query happens, and that loop can become unstable. This paper tries to show that the instability can be countered by co-training a second policy: an anchor generates all rollouts, an auxiliary sees the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL, with gradients flowing to both. In Qwen3 experiments at 1.7B and 4B scale, the recipe produces the strongest recorded student checkpoint in each of four math and code settings, lifting MATH500 from 0.630 to 0.685 at 4B and from 0.521 to 0.585 at 1.7B while all seven recorded single-policy code runs degrade. The authors present this as a stabilization hypothesis rather than a universal causal claim, because several comparisons differ in curriculum or initialization, and they specify the controlled matrix needed to test the mechanism.","feed_headline":"Two co-trained policies beat single-policy OPD in all four tests","feed_subtitle":"A second trainable policy absorbs part of each teacher correction, stabilizing math and code distillation at both scales.","key_machinery":"The machinery is the mixture-constrained co-training loop in Algorithm 1: the anchor samples every rollout; at each visited state the top-$k$ support is taken from the anchor; the anchor, auxiliary, and frozen teacher are evaluated on that support; a geometric mixture of the anchor and auxiliary log-scores is renormalized; and the loss is the reverse KL $D_{\\mathrm{KL}}(q_M \\| q_T)$ on that support, back-propagated into both policies. The load-bearing identity is the invariance $\\lambda\\Delta z_A + (1-\\lambda)\\Delta z_U = c(s_t)\\mathbf{1}$, which shows the objective identifies the mixture more strongly than either branch. Proposition 1 supplies the frozen-auxiliary limit and connects the method to static-delta methods, so the paper's claimed novelty is trainability of the auxiliary plus the shared mixture loss, not logit extrapolation.","core_discovery":"The central claim is that the new object is a moving auxiliary policy, not a frozen teacher delta. When the auxiliary is frozen, exact mixture matching to the teacher reduces algebraically to an extrapolated anchor-plus-contrast target of the form $z_A^\\ast = z_T + \\frac{1-\\lambda}{\\lambda}(z_T - z_U)$, closely related to the proxy targets used by OPD2 and W2S-OPD. When both branches are trainable, the loss identifies the geometric mixture $q_M = q_A^\\lambda q_U^{1-\\lambda}$ rather than either branch alone, and Equation (11) shows that perturbations satisfying $\\lambda\\Delta z_A + (1-\\lambda)\\Delta z_U = c(s_t)\\mathbf{1}$ leave the mixture unchanged. That branch-level null space lets the two policies divide a teacher correction along different Jacobians, optimizer states, and rollout roles, which the paper names the optimization-buffer hypothesis: part of a sharp correction may be absorbed by the non-rollout branch so the future state distribution moves less violently. The paper is explicit that endpoint accuracy alone cannot identify this mechanism and that the current evidence is descriptive, not a matched factorial study.","pith_inferences":["If the optimization-buffer mechanism is real, per-branch gradient norms and parameter displacement should show the auxiliary absorbing more of the update on steps where the anchor's rollout distribution would otherwise shift sharply; the paper lists these diagnostics but does not report them.","A sweep over the mixture weight $\\lambda$ and support size $k$ would test the mechanism's signature: the co-training advantage over a frozen auxiliary should shrink as $\\lambda \\to 1$ or $k \\to 1$, where the mixture collapses onto the anchor.","The paper's health-rule early stopping suggests a practical corollary it leaves implicit: co-training should be paired with validation-based checkpoint selection, not run to a fixed step count.","The branch-dynamics result hints that deployment can pick the non-rollout branch without paying the cost of generating training rollouts at serving time; this is implied by the method but not tested."],"forward_implications":["Either branch can be served at inference, so the second trainable policy adds training cost but no extra serving cost.","Both branches must be validated, because the better branch is not fixed in advance: the auxiliary is the best artifact in 4B math and the anchor in 1.7B math and code.","Two-policy co-training can avoid the specific code failure signature of entropy growth and repetition that degraded all six recorded 4B single-policy configurations, though the 4B run later develops repetition, so the method delays rather than eliminates pathology.","The frozen-auxiliary limit means WDL-OPD is not claiming novelty for teacher-minus-base deltas or logit extrapolation; the recoverable claim is that a trainable auxiliary sharing the loss creates degrees of freedom a static contrast cannot express."],"supporting_citations":[{"why":"Defines on-policy distillation as training on student-generated sequences, the setting WDL-OPD modifies.","marker":"(Agarwal et al., 2024)"},{"why":"Supplies the conditions for stable OPD and the small-token-support observation that motivates the top-k shared support.","marker":"(Li et al., 2026)"},{"why":"Defines OPD2's teacher/base delta, the frozen-delta method whose form reappears in the frozen-auxiliary limit.","marker":"(Heo et al., 2026)"},{"why":"Defines W2S-OPD's anchor-plus-contrast proxy target, the other reference point for the frozen-auxiliary equivalence.","marker":"(Yu et al., 2026)"},{"why":"Introduces Weak-Driven Learning's joint-logit principle that WDL-OPD transfers to anchor-generated on-policy states.","marker":"(Chen et al., 2026)"},{"why":"Shows teacher-compatible warm-up can rescue OPD, a confound the paper says must be shared across comparison rows.","marker":"(Liu et al., 2026)"},{"why":"Provides the Qwen3 model family used in all recorded experiments.","marker":"(Yang et al., 2025)"},{"why":"Supplies the LiveCodeBench external evaluation that grounds the code results.","marker":"(Jain et al., 2024)"}],"fun_headline_variants":["Two co-trained policies beat single-policy OPD","Co-training with a second policy stabilizes OPD","Trainable auxiliary policy tames OPD instability","A moving auxiliary policy stabilizes on-policy distillation","Joint co-training wins at 1.7B and 4B scales"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the recorded gains come from the shared-mixture co-training rather than from the mismatches the paper itself lists: different curricula between the 1.7B math stages, a pre-trained GRPO code anchor, extra trainable capacity, implicit ensembling in the loss, or checkpoint selection.","fun_headline_variants_meta":{"raw":{"variants":["Two co-trained policies beat single-policy OPD","Co-training with a second policy stabilizes OPD","Trainable auxiliary policy tames OPD instability","A moving auxiliary policy stabilizes on-policy distillation","Joint co-training wins at 1.7B and 4B scales"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4671,"prompt_tokens":1091,"completion_tokens":3580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":3500}},"tokens_in":707,"tokens_out":3580,"duration_ms":23749,"temperature":1.0,"reasoning_tokens":3500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:24:23.588112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's required controlled matrix with prompts, trajectories, teacher, initialization, support, optimizer, update count, and total compute fixed, comparing fully trainable WDL-OPD against two independently trained policies with no shared mixture: if the independent-two-policy control matches or beats WDL-OPD, mixture coupling is not the cause of the gain. A cheaper check on the existing record: rerun the 1.7B math single-policy baseline on the full 104,935-prompt pool with the two-policy optimizer; if the 6.4-point gap closes, the curriculum switch explains the result.","supporting_citations":[],"review_version":2}