Pith. sign in

REVIEW 3 major objections 6 minor 38 references

When the teacher wants to say 'Wait' but the student wants to continue, reasoning has gone wrong; Relay-OPD detects that moment, lets the teacher correct it, and lifts distilled math accuracy by 5.7 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:42 UTC pith:NKOALLLD

load-bearing objection A practical, well-executed distillation method with a genuinely new trigger signal; the core mechanism is plausible but not directly validated, and the evaluation needs more seeds and a cleaner tuning story. the 3 major comments →

arxiv 2607.26057 v1 pith:NKOALLLD submitted 2026-07-28 cs.CL cs.AI

Pass the Baton: Trajectory-Relayed On-Policy Distillation

Haolei Xu , Xiaowen Xu , Haiwen Hong , Zixuan Ni , Hongxing Li , Yiwen Qiu , Weiming Lu , Yongliang Shen This is my paper
classification cs.CL cs.AI
keywords on-policy distillationprefix failureteacher-student continuation asymmetryhandoff triggerreflection tokenstrajectory interventionmathematical reasoningknowledge distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On-policy distillation trains a small student on its own trajectories, but once the student commits to a wrong reasoning direction, every later token builds on the error and the teacher's supervision becomes unreliable. This paper claims that the failure is visible the moment it happens: on a failed prefix, the teacher's most likely next token is a reflection word (Wait, But, However), while the student's top choices contain none—so the two models disagree about whether to redirect. Relay-OPD turns that asymmetry into a handoff trigger, lets the teacher write a short corrective leg at the detected point, then returns control to the student. With a 4-billion-parameter teacher and 0.6B/1.7B students on eight math benchmarks, it beats standard on-policy distillation by 5.73 points on average for the larger student and cuts training trajectory length by more than half. The paper argues that early, local intervention—not longer or later correction—is what fixes prefix failure.

Core claim

The central claim: prefix failure—the moment a student commits to a wrong reasoning direction—is visibly marked by a teacher–student asymmetry: the teacher's top token is a reflection word (Wait, But, …) while the student's top-K contains none. This is observable online, with no label or verifier, and marks where reasoning went wrong. Relay-OPD uses it as a handoff trigger: the teacher briefly takes over (one reflection token plus L paragraphs, at most M times), then the student resumes. Training on these relay trajectories with a reverse-KL objective beats standard OPD by +5.73% (1.7B student) and +3.01% (0.6B) across eight math benchmarks, and cuts training trajectory length by more than h

What carries the argument

The handoff trigger φ(h)=1[a_T(h)∈R]·1[K_S(h)∩R=∅], where R is a fixed list of reflection tokens, a_T is the teacher's argmax token, and K_S is the student's top-K support. It turns the observation 'teacher redirects, student continues' into an online, label-free failure detector. Around it, Relay-OPD builds relay trajectories: the teacher writes one reflection token plus L paragraphs at up to M triggers, then the student resumes; a reverse-KL single-sample objective (Eq. 9) trains on the relay tokens. A single speculative-decoding engine makes the alternation exact and cheap, reusing teacher logits for both verification and trigger detection.

Load-bearing premise

The load-bearing premise is that 'the teacher wants to say a reflection word while the student's top choices contain none' reliably marks the point where reasoning has actually gone wrong; if that correlation is weak, the trigger fires on noise and the reported gains may come mostly from shorter trajectories rather than from genuine correction.

What would settle it

Annotate a few hundred student rollouts by locating the first point at which the final answer becomes unrecoverable (using a verifier or human judgment), then measure the handoff criterion's precision and recall at those points against random prefixes. If the trigger is not significantly better than chance, or if replacing each teacher leg with an equally short random sentence at the same detected positions preserves the accuracy gain, then the asymmetry is not a failure marker and the method's benefit is largely early termination.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Prefix failure in on-policy distillation can be corrected online without external verifiers, reward models, or process labels.
  • Training becomes cheaper: average rollout length drops by over 50% (50.7% for the 1.7B student, 63.9% for 0.6B) and the best checkpoint is reached earlier.
  • A very small amount of teacher writing—around 0.35% of tokens in the intervention study, settling at 2–3% during training—is enough to produce most of the correction.
  • The gains hold across eight benchmarks and both student scales, with best or second-best results everywhere; pass@k also improves.
  • The value of teacher takeover is front-loaded: shifting intervention from early to late triggers cuts accuracy sharply, so early positions are where correction matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The handoff criterion is effectively a cheap, online error detector for long reasoning chains; a natural extension is to use it at inference time to decide when a small model should yield to a larger one, not just during training.
  • The paper's evidence that the trigger marks genuine failures is indirect (one detailed case, budget-exhaustion curves, final accuracy); a direct precision/recall study against labeled failure points would separate real correction from the benefit of stopping early.
  • Because later takeovers are far less effective, a training schedule that shrinks the relay budget as the student improves—matching the observed drop in budget exhaustion—could preserve the gains with even fewer teacher tokens.
  • The reflection-token vocabulary is model-family-specific, as the paper itself notes; transferring the method to a model whose correction style avoids Wait/But would require re-deriving the set, and that is likely where the method would break first.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Relay-OPD, an on-policy distillation method that detects 'prefix failure' through a teacher–student continuation asymmetry: when the teacher's top-1 next token is a reflection word (Wait, But, ...) while the student's top-K contains no reflection token, the teacher takes over for a short leg (L paragraphs, up to M takeovers) before the student resumes. The student is optimized on the resulting relay trajectory with a clipped reverse-KL-style advantage. On eight math benchmarks with Qwen3-4B teacher and 0.6B/1.7B students, the method reports average improvements over standard OPD (+5.73% for 1.7B) and over FastOPD (+1.49%), with over 50% shorter training trajectories. The method is implemented in a single speculative-decoding engine.

Significance. The central idea—using an online, label-free teacher-student divergence as a handoff trigger for local correction—is interesting and potentially practical. If the trigger is a valid failure detector, the paper provides a strong empirical demonstration: the ablations (teacher-leg vs. trigger-stop, objective choice, sensitivity to M/L/K) are informative, and the reported training-length reduction is substantial. The paper is honest about limitations (reflection-token transfer, teacher-gap assumption, budget tuning on 1.7B). Strengths include the clear problem framing, the explicit speculative-decoding implementation, the rich ablation suite, and the availability of code via a GitHub link. The main weaknesses are the unvalidated Eq. (6) trigger and the lack of uncertainty quantification in the headline comparisons.

major comments (3)
  1. [§3.2, Eq. (6); Appendix A.1; Table 2] The paper's central claim is that the handoff criterion in Eq. (6) is a label-free indicator of where the student's reasoning has gone wrong. This claim is not directly tested. The only direct illustration is one case (Appendix E.1); the rest of the evidence is indirect (final accuracy, teacher-token ratio, and Table 2). The Trigger-Stop ablation (M=1, no teacher leg, average 43.48) is worse than FastOPD's fixed 4,096-token truncation (45.47, Table 7), so stopping at Eq. (6) is not shown to be a better failure detector than a fixed cutoff. The reflection-token set in Appendix A.1 is asserted to transfer to Qwen3 with no false-positive measurement on correct reasoning. Because Eq. (6) is load-bearing, please report precision/recall (or trigger-rate vs. final correctness) on a labeled/held-out sample, and measure false-positive rates of R on correct prefixes.
  2. [Tables 1, 6, 7; §4.5] The headline comparisons rest on a single training run for each configuration, with no error bars or significance tests. Hyperparameters (M, L, K in §4.5) and the FastOPD truncation length (Table 7) were selected on the same eight evaluation benchmarks, and the best checkpoint is chosen on those benchmarks (Table 1 subscripts). This creates selection bias: the reported +5.73%/+1.49% averages are maxima over the swept configurations, not typical performance. At minimum, provide multiple seeds (or bootstrap CIs) for the main comparison and a held-out validation set for checkpoint/hyperparameter selection, or clearly state that the reported numbers are tuned on the test set.
  3. [§3.3, Eqs. (10)–(12)] The efficiency claim that 'teacher logits computed during verification simultaneously provide a_T and the trigger criterion at no additional cost' appears inconsistent with the state-dependent target. In student legs, π_tgt = π_barθ, so speculative verification does not compute teacher logits; yet Eq. (6) requires teacher logits on every student-leg prefix. The paper does not explain how teacher logits are obtained there without an extra teacher forward pass. This matters because the single-engine speculative-decoding implementation is presented as a contribution. Please clarify the implementation or qualify the 'no additional cost' statement, and ideally report wall-clock training time.
minor comments (6)
  1. [Figure 1 and text after abstract] There are garbled token-probability snippets ('So 50.6% - 16.4% Now 21.1% But 74.4% Wait 7.8% So 10.1%...') between the abstract and introduction. Ensure the figure/caption and surrounding text are legible and the numbers match Table 8.
  2. [§3.2, Eq. (7)–(9)] The term 'reverse-KL-style' for Eq. (7)–(9) is imprecise: the objective is a clipped importance-weighted policy-gradient surrogate, not a reverse-KL divergence. Consider renaming or explaining the relationship.
  3. [§3.2, §3.3] The notation h^z_t is used with different index conventions in Eq. (10) and Algorithm 1. Define it once before first use and keep the indexing consistent.
  4. [Table 1 caption] The caption should define the subscripts in the Avg column (training step of the best checkpoint) and explicitly state that values are single runs with no error bars.
  5. [Figure 2(a)] The metric 'mean@4' is used without definition; clarify whether it is the same 4-sample mean used in the evaluation protocol.
  6. [Eq. (11)–(12)] The subscript on π_tgt is dropped in Eq. (12); use consistent notation for readability.

Circularity Check

0 steps flagged

No significant circularity: the derivation chain is self-contained and externally benchmarked.

full rationale

Relay-OPD's derivation does not reduce to its own inputs at any equation-level step. The handoff trigger (Eq. 6) is an operational condition on teacher and student token distributions; it is not defined in terms of final accuracy or the benchmark outcomes it is used to predict. Its usefulness is established by external intervention experiments and downstream benchmark results, which are independent of the trigger's definition. The relay objective (Eq. 7) is the same single-sample RKL-style advantage as standard OPD applied to the actually generated relay tokens; it contains no fitted term that is renamed as a prediction. The only hyperparameters (K, M, L) are selected via the reported sensitivity analysis on the same evaluation benchmarks, which is a form of test-set selection and a limitation, but it is not a constructional identity: the reported gains could in principle fail under other settings, and the paper shows neighboring settings with lower accuracy. The reflection-token set is borrowed from external sources (Guo et al.; Muennighoff et al.) and its transfer to Qwen3 is asserted rather than measured, a validity concern, but not circularity. No load-bearing self-citation appears: cited foundations (speculative decoding, OPD, SKD, FastOPD) are external prior work. The paper's own limitations section notes the budget was tuned on the 1.7B student and reused for 0.6B, which again is an experimental-design caveat, not a derivation that equates an output to an input. Therefore no specific reduction of the kind required by the circularity pass is present.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on four main inputs: three tuned hyperparameters (K, M, L), a paper-specific reflection-token trigger, an unproven objective design for teacher legs, and standard speculative decoding. No new physical or conceptual entities are introduced beyond the training procedure itself. The free parameters were selected on the same evaluation benchmarks, which is the largest external-evidence weakness.

free parameters (3)
  • K (handoff top-K) = 5
    Controls the divergence threshold for triggering a handoff; swept over {1,5,10,|V|} in §4.5 and selected for best average accuracy on the evaluation set.
  • M (max teacher takeovers) = 2
    Relay budget cap; swept over {1,2,3,4} in §4.5; M=2 chosen.
  • L (teacher leg paragraphs) = 3
    Length of teacher takeover in paragraphs; swept over {0,...,5} in §4.5 and §4.4; L=3 chosen, with L=4 nearly tied.
axioms (4)
  • ad hoc to paper The reflection-token set R (Wait, But, Hmm, ... and variants) reliably indicates teacher redirect intent, and its absence from student top-K marks a reasoning failure worth correcting.
    The entire trigger mechanism in Eq. (6) depends on this assumption. It is supported by one detailed case and aggregate training dynamics, but no precision/recall against ground-truth failures is provided.
  • domain assumption Teacher-leg tokens can be optimized with the same reverse-KL-style single-sample objective as student-leg tokens, without explicit off-policy correction.
    Teacher-leg tokens are sampled from π_T, not π_θbar, so the expectation of A_t in Eq. (7) is not the standard reverse-KL estimator used in student legs. The paper does not justify this design theoretically.
  • standard math Speculative decoding with the student as draft model and teacher as target model exactly reproduces the relay process described in §3.2.
    The paper invokes the correctness of speculative sampling (Leviathan et al., 2023) without reproving it; the deterministic leg-initial token is a special case that needs the stated acceptance semantics to preserve exactness.
  • domain assumption The DAPO-Math-17K training distribution and the eight evaluation benchmarks are representative and uncontaminated for measuring transfer to math reasoning.
    No contamination analysis is provided, and benchmark names include recent AIME/HMMT editions that may appear in web-scale training data.

pith-pipeline@v1.3.0-alltime-deepseek · 18734 in / 13460 out tokens · 119119 ms · 2026-08-01T00:42:55.421795+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.

Figures

Figures reproduced from arXiv: 2607.26057 by Haiwen Hong, Haolei Xu, Hongxing Li, Weiming Lu, Xiaowen Xu, Yiwen Qiu, Yongliang Shen, Zixuan Ni.

Figure 1
Figure 1. Figure 1: (a) A Relay-OPD case: at the detected handoff trigger, the teacher prefers the reflection token (But, 74.4%) while the student would continue along the current direction (So, 50.6%); a brief teacher leg corrects the reasoning and the student resumes to the correct answer. (b) Difference between OPD and Relay-OPD. (c) Overall performance. ∗ Equal contribution. † Project leader. ‡ Corresponding author. 1 arX… view at source ↗
Figure 2
Figure 2. Figure 2: Preliminary Trajectory Intervention Experiments. Qwen3-4B-Instruct-2507 (teacher) and Qwen3-1.7B-Non-Thinking (student) on 128 DAPO-Math-17K English samples; the handoff criterion matches the main experiments. (a) Accuracy (mean@4) and teacher token ratio as the number of teacher-generated paragraphs L varies; a periodic delay of r skips r consecutive valid triggers before the next takeover. (b) Teacher–st… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Relay-OPD. Unlike standard OPD (top), which trains on student-only tra￾jectories that continue misdirected after prefix failure, Relay-OPD constructs relay trajectories (mid￾dle): when the handoff trigger (bottom) detects that the teacher would redirect the reasoning while the student would not, the teacher briefly takes over before the student resumes. The entire rollout runs in a single specu… view at source ↗
Figure 5
Figure 5. Figure 5: Inference response length of Relay-OPD vs. FastOPD. improves by +3.01% over OPD and surpasses FastOPD by +0.62%. Relay-OPD further achieves significantly higher pass@k than standard OPD across different sampling budgets ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training Dynamics of Relay-OPD. Using Qwen3-4B-Instruct-2507 as teacher and Qwen3-1.7B-Non-Thinking as student. Left: fraction of trajectories exhausting the relay budget; Middle: teacher token ratio relative to all effective response tokens; Right: policy entropy of Relay￾OPD, OPD, and FastOPD. Relay-OPD teaches the student to redirect reasoning before deviations compound further, yielding more accurate a… view at source ↗
Figure 7
Figure 7. Figure 7: Hyperparameter Sensitivity of Relay-OPD. Using Qwen3-4B-Instruct-2507 as teacher and Qwen3-1.7B-Non-Thinking as student. Left: relay budget (M, L); Right: handoff top-K, where |V| denotes the vocabulary size. The Teacher FKL variant takes the teacher’s top-k support set Kt = TopKk (πT (· | h z t )) with k = 128. The teacher probability is renormalized over Kt as q k T (v | h z t ) = πT (v | h z t ) P u∈Kt … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 18 linked inside Pith

  1. [1]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp.\ 21246--21263, 2024

    2024

  2. [2]

    Aime 2024

    AI-MO. Aime 2024. https://huggingface.co/datasets/AI-MO/aimo-validation-aime, 2024 a

    2024

  3. [3]

    Amc 2023

    AI-MO. Amc 2023. https://huggingface.co/datasets/AI-MO/aimo-validation-amc, 2024 b

    2023

  4. [4]

    Matharena: Evaluating llms on uncontaminated math competitions, February 2025

    Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena.ai/

    2025

  5. [5]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, volume 2024, pp.\ 32694--32717, 2024

    2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024

  10. [10]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  12. [12]

    u botter, Frederike L \

    Jonas H \"u botter, Frederike L \"u beck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Trajectory-refined distillation

    Li Jiang, Haoran Xu, Yichuan Ding, and Amy Zhang. Trajectory-refined distillation. arXiv preprint arXiv:2606.08432, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp.\ 1317--1327, 2016

    2016

  16. [16]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp.\ 611--626, 2023

    2023

  17. [17]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  18. [18]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations, volume 2024, pp.\ 39578--39601, 2024

    2024

  20. [20]

    On-policy distillation

    Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi:10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation

    doi:10.64434/tml.20251026 2025

  21. [21]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 20286--20332, 2025

    2025

  22. [22]

    Unlocking on-policy distillation for any model family

    Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Frere, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Unlocking on-policy distillation for any model family. https://huggingface.co/spaces/HuggingFaceH4/on-policy-distillation, 2025

    2025

  23. [23]

    Privileged information distillation for language models

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Self-distillation enables continual learning

    Idan Shenfeld, Mehul Damani, Jonas H \"u botter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp.\ 1279--1297, 2025

    2025

  27. [27]

    Mimo-v2-flash technical report

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    On the position bias of on-policy distillation

    Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang. On the position bias of on-policy distillation. arXiv preprint arXiv:2606.22600, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    Deepseek-v4: Towards highly efficient million-token context intelligence

    Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026

    arXiv 2026

  30. [30]

    Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling

    Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In International Conference on Learning Representations, volume 2025, pp.\ 64616--64646, 2025

    2025

  31. [31]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    Pith/arXiv arXiv 2025

  32. [32]

    Distilling rule-based knowledge into large language models

    Wenkai Yang, Yankai Lin, Jie Zhou, and Ji-Rong Wen. Distilling rule-based knowledge into large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 913--932, 2025 b

    2025

  33. [33]

    Black-box on-policy distillation of large language models

    Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643, 2025

    arXiv 2025

  34. [34]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38: 0 113222--113244, 2026

    2026

  35. [35]

    Glm-5: from vibe coding to agentic engineering

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026

    Pith/arXiv arXiv 2026

  36. [36]

    Fast and effective on-policy distillation from reasoning prefixes

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pp.\ 25553--25569, 2026

    2026

  37. [37]

    Self-distilled reasoner: On-policy self-distillation for large language models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026

  38. [38]

    Less is more: Early stopping rollout for on-policy distillation

    Zhou Ziheng, Jiaqi Li, Huacong Tang, Ying Nian Wu, and Demetri Terzopoulos. Less is more: Early stopping rollout for on-policy distillation. arXiv preprint arXiv:2605.27028, 2026

    Pith/arXiv arXiv 2026