{"id":"971bc56f-6c81-4c7c-9180-6ac5348f2db4","arxiv_id":"2608.08726","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PAST adapts a self-distillation teacher using complete correct and failed student trajectories, then distills the adapted teacher back to the prefix-only student, gaining 5.6pp Avg@12 over Vanilla OPSD.","lead":"This paper introduces PAST, a training method that lets a teacher model see a student's completed answer and its correctness before teaching the student from the original prefix only. It reports a 5.6 percentage point average accuracy improvement over vanilla on-policy self-distillation on three math reasoning benchmarks, along with a theoretical result explaining why privileged teacher success alone does not guarantee student improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 5.6pp gain has a real CI, but the causal attribution to trajectory-conditioned adaptation rests on a single-seed 2x2 factorial; a three-seed interaction replication should settle it.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and I do not think it should move: the three-seed main comparison is real evidence for the performance claim, the theory in Appendix C is internally consistent, and the paper is transparent about limitations. The weakest link I would emphasize is different from the reader's chosen weakest_assumption. The reader flags the unproven gamma_minus margin in Corollary 6; that is an honest gap, but a negative finding there would weaken the paper's theoretical narrative without contradicting the empirical gain. The factorial attribution is more directly load-bearing for the stated central claim, because the paper explicitly says the gain is attributed to teacher adaptation with matching trajectories, and that attribution rests on a single-seed 2x2 table with no confidence interval. The interaction contrast is the right statistic: the full-cell gain is far larger than the sum of the two single-factor cells, so the paper's complementary-mechanism claim is exactly a large positive interaction. A three-seed replication of just the four factorial cells is feasible with the released protocol and would settle whether the mechanism attribution is empirical fact or a single-seed artifact. Until that is run, CONDITIONAL remains the appropriate verdict, and the attribution part of the strongest claim should be read as provisional. I agree with the reader that no code or full hyperparameter set is released, which further supports not upgrading to ACCEPT, but that is a reproducibility caveat rather than an identified flaw in the argument.","tokens_in":16860,"tokens_out":12214,"duration_ms":141426,"concrete_test":"Rerun the four cells of Table 2 (Vanilla OPSD, adaptation-only, trajectory-only, full PAST) with the same three seeds (17, 29, 43) and the same 100-update protocol, and compute a paired bootstrap 95% CI for the interaction contrast I = (PAST - Vanilla) - (TrajectoryOnly - Vanilla) - (AdaptationOnly - Vanilla). If the CI excludes 0 and the point estimate stays near 4.97pp, the mechanism attribution is supported. If the CI includes 0, the strong claim should be reduced to 'PAST improves over Vanilla OPSD' without the claim that complete-trajectory teacher adaptation is the cause. This is a single, decisive check because the interaction is exactly the quantity that distinguishes additive factor effects from the claimed complementary mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim has two parts: PAST improves Avg@12 over Vanilla OPSD by 5.617pp, and the factorial plus perturbation diagnostics attribute that gain to teacher adaptation using the matching complete trajectory. The first part is supported by a three-seed paired-bootstrap CI of [2.592, 8.333]. The second part is not supported with the same standard. Table 2 is the only direct evidence that full PAST (48.426) is more than the sum of trajectory access alone (43.364) and adaptation alone (42.901); the implied interaction is about 4.97pp. If Table 2 is single-seed, as the reader states, the apparent superadditivity has an unquantified sampling error. The 95% CI for the PAST-versus-Vanilla contrast does not cover the interaction contrast, which has its own variance. Figure 4a shows only that the teacher's own success and output distribution change when the matching trajectory is removed or shuffled; it does not show that this teacher-side change is what improves the student. The theoretical result in Corollary 6 is also not verified for the implemented estimator: the paper never demonstrates that gamma_minus = g_k(q_-;H) > 0 at enough prefixes, and the success-masked, retry-conditioned GRPO estimator is acknowledged to be biased. So if the factorial interaction is not reproducible, there is no direct evidence left for the causal attribution, only the raw improvement. This is a load-bearing gap because the title and contributions claim a mechanism, not just a performance increase.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PAST, a method for on-policy self-distillation (OPSD) that adapts a privileged teacher using complete student rollouts (both correct and failed) before distilling the adapted teacher onto the student's original prefixes. The theoretical analysis characterizes forward-KL projection to a prefix-only student, showing that distillation transfers the conditional arithmetic mean of the trajectory-conditioned teacher and providing sufficient conditions for positive value transfer. Empirically, on three mathematical reasoning benchmarks with Qwen3-1.7B, PAST reports a macro-average Avg@12 of 48.426 versus 42.809 for Vanilla OPSD, a 5.617-point improvement with a three-seed paired-bootstrap 95% CI [2.592, 8.333]. A 2x2 factorial study and trajectory-perturbation diagnostics are offered as evidence that the gain comes from trajectory-conditioned teacher adaptation.","tokens_in":17176,"tokens_out":4571,"duration_ms":47583,"significance":"If the empirical and attribution claims hold, PAST is a simple and practical extension of OPSD: it reuses rollouts that are already collected, requires no additional verifier labels beyond final-answer correctness, and is accompanied by a rigorous projection analysis (Propositions 1-5, Corollaries 2-6) with complete proofs in Appendix C. The paper also includes explicit limitations, detailed reproducibility information (checkpoint lifecycle, RNG states), and numerical protocol checks (Table 4). The headline three-seed result is well supported by a paired bootstrap interval. The main weakness is that the causal mechanism claim — that the gain is specifically due to trajectory-conditioned teacher adaptation — is supported by evidence that has not been given the same statistical standard.","major_comments":[{"comment":"The factorial cells appear to be single-seed point estimates; no seed-level variance is reported for trajectory-only (43.364), teacher-only (42.901), or the PAST-minus-sum interaction of about 4.97 points. The paper's central attribution claim — that PAST combines trajectory access and teacher adaptation — rests on this table. The interaction contrast needs uncertainty quantification (e.g., a paired bootstrap across seeds) or a multi-seed replication. Without it, the superadditivity that distinguishes PAST from either factor alone is an unquantified point estimate.","section":"Section 5.3, Table 2"},{"comment":"The positive-transfer theorem requires gamma_minus = g_k(q_-;H) > 0 at enough prefixes, but the paper does not demonstrate that the implemented estimator achieves this condition. The Limitations section and Appendix C.9 explicitly acknowledge that the success-masked, retry-conditioned GRPO estimator is biased relative to the population objective. To make the theoretical support load-bearing, the authors should measure the actual frozen-student value margin (or a lower bound) on held-out prefixes, or otherwise show that the regularization in Eq. (14) keeps the teacher's successful continuations compatible enough with the frozen student for Corollary 6 to apply.","section":"Section 4, Corollary 6 and Section 3.3"},{"comment":"The trajectory-perturbation diagnostics measure effects on teacher success and output JSD, not on the final student. The claim that the adapted teacher uses the matching trajectory is established, but the further claim that this teacher-side dependence is what improves the student is not. A direct test would distill from perturbed teachers and compare final student outcomes, or regress the factorial interaction on the perturbation effect. As it stands, the student-side attribution is an inference from separate pieces of evidence, not a measured causal link.","section":"Section 5.4, Figure 4a"}],"minor_comments":[{"comment":"The joint-vs-single-branch comparison (full PAST at 48.426 versus 45.864 for failed-only and 44.908 for correct-only) is also presented without seed variance; this ablation should be explicitly labeled as single-seed or accompanied by uncertainty, consistent with the paper's otherwise careful treatment of seeds.","section":"Section 5.5 / Appendix E.2, Table 6"},{"comment":"The 95% CI is reported only for the macro-average PAST-vs-Vanilla contrast; task-level leads (AIME24, AIME25, HMMT25) are point estimates across seeds. The paper should clarify which comparisons carry intervals and which do not, to prevent overreading of the individual-task results.","section":"Section 5.2 and Table 1"},{"comment":"The E11 trace is presented as illustrative, but the manuscript does not state that it is a single teacher/student cycle and not an ensemble statistic; please add this qualifier in the text or caption.","section":"Section 5.4, Figures 4b-4c"},{"comment":"The expected-sample-count identity assumes independent continuations with a constant success probability s, while Figure 6 shows that the success rate varies over time; the text should note this limitation when interpreting the 17.6% sampling reduction.","section":"Appendix C.9, Eq. (53)"},{"comment":"Equation (44) defines the trajectory KL as the expectation over C~Q_U of a sum of token-level KLs; the sentence immediately after the proposition says 'the same tokenwise divergence' with respect to Eq. (13), but Eq. (13) is success-conditioned while Eq. (44) is not — please reconcile this wording.","section":"Appendix C.5, Proposition 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest and the core empirical result is credible, but the mechanism claim is the intellectual center of the paper (title, abstract, contributions) and it is currently supported by single-seed factorial evidence plus a theoretical sufficient condition that is not verified for the implemented estimator. The stress-test concern is therefore real. I would recommend major revision rather than rejection: the raw improvement is well supported, and the missing pieces (multi-seed factorial, a direct check of gamma_minus, or a perturbation-to-student experiment) are feasible within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: PAST does something new — it uses the complete student rollout as teacher-side privilege, preserves correct-branch behavior, adapts the teacher on failed branches with a student-proximity penalty, and then distills onto the original prefixes. The headline result is real: a 5.617pp macro Avg@12 gain over Vanilla OPSD at 1.7B, with a 95% CI [2.592, 8.333], and it beats SD-Zero and H2SD. The projection theory is also solid as far as I checked: forward-KL distillation lands on the conditional arithmetic mean of the trajectory-conditioned teacher, and Proposition 4 is a clean counterexample showing a perfect privileged teacher can leave the student unchanged. Those are genuine contributions.\n\nThe paper is honest in the right places. It flags that the 4B extension is one-seed, that the verifier only checks final answers, and that the failed-branch estimator is a biased finite-sample version of the population objective. That last admission matters because the paper's mechanism claim — that teacher adaptation using the matching trajectory causes the gain — depends on Corollary 6, but the paper never shows the implemented estimator attains the positive margin γ− that the corollary requires. So the theory gives a sufficient condition, not proof that this update satisfies it.\n\nThe weak spot is attribution. The 5.6pp gain is well-supported, but the 2x2 factorial in Table 2, the branch ablations, and the objective comparisons appear to be single-seed point estimates. The implied interaction is about 5pp, and that number has no error bar. The stress-test note is right: if the interaction doesn't reproduce, the causal story loses its direct evidence, even though the raw improvement stands. I would not call this fatal — the main effect is solid and the method is well motivated — but the paper's title and contributions claim a mechanism, and that mechanism is currently supported by one unquantified factorial. No code or full hyperparameter set is released either, so exact reproducibility is not yet assessable.\n\nWho gets value: anyone working on self-distillation or hindsight training for reasoning LLMs. The theory sections are short and worth reading; the empirical design is mostly careful. I'd send it to a serious reviewer. The fix is straightforward: multi-seed factorial and branch ablations, or at minimum an explicit statement that they are single-seed and should not be read as conclusive. As it stands, it's a strong CONDITIONAL.","headline":"PAST's 5.6pp average gain over Vanilla OPSD is real and well-supported by a 3-seed CI, but the causal-attribution story leans on a single-seed factorial that needs replication before the mechanism claim is solid.","tokens_in":803,"tokens_out":745,"would_cite":true,"duration_ms":32747,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAST claims a reasoning model's completed rollouts can be repurposed as teacher-side privilege.","keywords":["on-policy self-distillation","privileged information","complete student trajectories","teacher adaptation","forward-KL distillation","mathematical reasoning","GRPO","hindsight supervision"],"falsifier":"Compute the frozen-student value margin $\\gamma_-(H)=\\sum_a (q_-(a|H)-p_k(a|H))Q_k(H,a)$ on failed prefixes from held-out problems. If the mean margin is non-positive while PAST still outperforms Vanilla OPSD, the proposed transfer mechanism fails.","tokens_in":16568,"feed_emoji":"🧮","tokens_out":11809,"duration_ms":103484,"temperature":0.7,"pith_summary":"PAST asks whether the ending of a student rollout, known after sampling but hidden at inference, can be used as privileged information rather than discarded. The paper's answer is yes, provided the ending is used to adapt the teacher before distillation: correct responses train the teacher to preserve the student's distribution, failed responses train it toward verifier-verified success under proximity regularization, and the student still learns only from its original prefixes. If correct, this gives a way to turn a model's own failures into dense next-token supervision without ever showing the student future context. The reported 5.6-point Avg@12 macro improvement over Vanilla OPSD on three mathematical reasoning benchmarks, with attribution to both matching trajectories and teacher adaptation, is the concrete stake.","feed_headline":"Complete-student hindsight adds 5.6 points to self-distillation","feed_subtitle":"Failed rollouts teach the teacher; correct ones stay unchanged; three math benchmarks improve.","key_machinery":"The load-bearing object is the trajectory-conditioned teacher $q_\\phi(a\\mid H_t, U)$ with $U=(Y,Z)$, and the causal projection $q(a\\mid H_t)=\\mathbb{E}[q_U(a\\mid H_t)\\mid H_t]$ that forward-KL distillation implements. Proposition 1 shows the KL to any prefix-only student decomposes into privileged variation $I(A;U\\mid H_t)$ plus distillable shift $D_{\\mathrm{KL}}(q\\parallel p_k)$, which explains why a perfect privileged teacher can leave the student unchanged (Proposition 4). The failed branch is driven by GRPO on teacher continuations plus the student-proximity KL $\\frac{1}{|C_j|}\\sum_s D_{\\mathrm{KL}}(q_\\phi(\\cdot\\mid G_{j,s},U_i)\\parallel p_k(\\cdot\\mid G_{j,s}))$ with $\\beta_{\\mathrm{KL}}=0.05$; Corollary 6 gives the transfer margin $\\gamma_-(H)$ needed for a positive frozen-student value update.","core_discovery":"The central claim is that a completed student rollout is not just a sample to score; it can be turned into teacher-side privilege. PAST freezes the student, samples fresh responses, and adapts one teacher: on correct trajectories it matches the frozen student's full next-token distribution so correct behavior is preserved, and on failed trajectories it runs verifier-seeking GRPO with a student-proximity KL penalty ($\\beta_{\\mathrm{KL}}=0.05$) so the teacher's successful continuations stay compatible with the student. The adapted teacher is then distilled once on the original student prefixes. Across AIME 2024, AIME 2025, and HMMT 2025 with a 1.7B reasoning model, PAST raises the Avg@12 macro average to 48.426 from Vanilla OPSD's 42.809, a 5.617-point gain (95% CI [2.592, 8.333]); the 2x2 factorial shows neither complete-trajectory input alone nor teacher adaptation alone reproduces the gain, and trajectory removal or shuffling degrades the teacher's success and distributional closeness.","pith_inferences":["A direct extension the paper leaves open: methods that distill with reverse KL would project toward the geometric mean $\\exp(\\mathbb{E}[\\log q_U])$ rather than the arithmetic mean, so PAST's decomposition and its failure mode may not carry over to reverse-KL distillations.","The acknowledged bias in the finite-sample estimator (success masking, adaptive group sizes, one retry, all-failure skipping) means a clean test of the population objective is still missing; an unbiased estimator would separate the distillation mechanism from retry dynamics.","The one-seed 4B result with a lower AIME24 score suggests the claimed gain may be scale- or task-dependent; a multi-seed scale sweep would test whether the 5.6-point margin survives."],"forward_implications":["If PAST is right, a reasoning model can be improved from its own failed rollouts without ever giving the deployed policy access to the future: the teacher carries the hindsight, the student keeps a causal interface.","The projection theorem means forward-KL distillation of a trajectory-conditioned teacher can transfer no more than the conditional arithmetic mean over trajectories; trajectory-specific variation is exactly the part a prefix-only student cannot learn.","Because correct trajectories have the frozen student as their ideal fixed point, the correct branch's role is preservation rather than improvement; fitting only sampled correct tokens would not carry the same guarantee.","Teacher success alone is not evidence of student improvement: Proposition 4 constructs a teacher that succeeds with probability one while distillation leaves the student unchanged."],"supporting_citations":[{"why":"It defines on-policy distillation on student-generated states, the training interface PAST preserves.","marker":"Agarwal et al., 2024"},{"why":"It establishes the student-prefix teacher-supervision setup for on-policy distillation.","marker":"Gu et al., 2024"},{"why":"It defines Vanilla OPSD, the privileged self-teacher baseline PAST extends and beats.","marker":"Zhao et al., 2026a"},{"why":"It supplies GRPO, the group-relative policy signal PAST uses to adapt the teacher on failed trajectories.","marker":"Shao et al., 2024"},{"why":"It provides the divergence measures behind the forward-KL projection identity that separates privileged variation from mean shift.","marker":"Lin, 1991"},{"why":"It supplies Bregman divergence projection results supporting the arithmetic-mean causal target of distillation.","marker":"Banerjee et al., 2005"},{"why":"It is the SD-Zero baseline that conditions a reviser on the full response and outcome, a hindsight self-improvement comparison.","marker":"He et al., 2026"},{"why":"It is the H2SD baseline routing successful and failed responses through distinct hindsight mechanisms.","marker":"Cai et al., 2026"},{"why":"It is RSTG, which provides teacher guidance to failed zero-variance RL groups, the closest failed-trajectory guidance baseline.","marker":"Han et al., 2026"}],"fun_headline_variants":["Failed rollouts become teacher's advantage in self-distillation","Self-distillation gets a boost from the student's own failures","Trajectory-aware teacher lifts self-distillation by 5.6 points","PAST: turn student failures into teacher successes for math reasoning","Student's failed trajectories become teacher's edge: +5.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the failed-branch update makes the projected teacher's next-token distribution improve the frozen student's expected verifier value ($\\gamma_-(H)>0$) at enough prefixes; Corollary 6 gives sufficient conditions for that transfer, but the paper does not prove the implemented estimator achieves them, and the finite-sample estimator is acknowledged to be biased.","fun_headline_variants_meta":{"raw":{"variants":["Failed rollouts become teacher's advantage in self-distillation","Self-distillation gets a boost from the student's own failures","Trajectory-aware teacher lifts self-distillation by 5.6 points","PAST: turn student failures into teacher successes for math reasoning","Student's failed trajectories become teacher's edge: +5.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001367,"raw_usage":{"total_tokens":5582,"prompt_tokens":1026,"completion_tokens":4556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":4466}},"tokens_in":642,"tokens_out":4556,"duration_ms":28977,"temperature":1.0,"reasoning_tokens":4466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:27:02.711302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the frozen-student value margin $\\gamma_-(H)=\\sum_a (q_-(a|H)-p_k(a|H))Q_k(H,a)$ on failed prefixes from held-out problems. If the mean margin is non-positive while PAST still outperforms Vanilla OPSD, the proposed transfer mechanism fails.","supporting_citations":[],"review_version":1}