Pith. sign in

REVIEW 2 major objections 5 minor 51 references

In on-policy distillation, the reward that works best is the gap between a teacher model and its own pre-reasoning base checkpoint, not the teacher's raw token preferences.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:56 UTC pith:PB6EH4S2

load-bearing objection Solid empirical paper: using teacher-minus-base as the primary distillation reward consistently beats OPD and ExOPD across Qwen3 and Gemma4, but the 'reasoning trace' interpretation is a conjecture, not yet established. the 2 major comments →

arxiv 2607.15161 v1 pith:PB6EH4S2 submitted 2026-07-16 cs.LG cs.CL

On-Policy Delta Distillation

Byeongho Heo , Jaehui Hwang , Sangdoo Yun , Dongyoon Han This is my paper
classification cs.LG cs.CL
keywords on-policy distillationdelta signalknowledge distillationreasoning transferpost-trainingteacher-base modelLLM reasoningOPD2
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that when a strong teacher supervises a smaller student through on-policy distillation, the most effective training signal is the difference between the teacher's token probabilities and the probabilities of the teacher's base checkpoint before reasoning tuning. This 'delta signal' is meant to isolate the knowledge acquired specifically during reasoning tuning, rather than transferring the teacher's general style and conversational habits. The proposed method, OPD2, adds a centering step and a gating condition to this signal, and reports consistent gains over standard on-policy distillation and the extrapolation-based ExOPD across Qwen3 models of 1.7B, 4B, and 8B, plus Gemma4-E4B, on math, science, and code benchmarks. A sympathetic reader would care because on-policy distillation is a cheaper alternative to reward-model reinforcement learning, and this result suggests that a simple teacher-minus-base difference can transfer reasoning skill in a short post-training run.

Core claim

The central claim is that the standard on-policy distillation reward, log pi* - log pi_student, should be replaced by the delta signal R_delta = log pi* - log pi*_base, where pi*_base is the teacher's checkpoint before instruction tuning for reasoning. The authors argue that this difference represents the teacher's learning trace from reasoning tuning and therefore provides a more direct channel for transferring reasoning ability. To make this signal trainable, they subtract the expected reward under the student's sampling distribution (centering) and apply a joint condition that zeroes the advantage whenever the delta signal and the standard distillation advantage disagree in sign. The resu

What carries the argument

The key object is the delta signal, the token-level difference in log-probability between the reasoning-tuned teacher and its base model prior to reasoning tuning. It is paired with a gated advantage function that combines the delta advantage with the standard on-policy distillation advantage through a sign-agreement condition, turning off the update when the two disagree. The delta signal is what carries the reasoning-specific knowledge; the gate prevents the student from drifting away from the teacher's distribution or collapsing to a one-hot token.

Load-bearing premise

The method assumes the teacher's base checkpoint differs from the teacher only in the reasoning-tuning knowledge, so the log-probability gap isolates reasoning skill rather than also encoding format, style, alignment, or data-distribution differences.

What would settle it

Train the same student with a delta signal built from a teacher-base checkpoint that has been fine-tuned on chat style or general instruction following but not on reasoning. If OPD2's gains persist with that non-reasoning delta, the signal is not specifically the reasoning trace; if the gains vanish, the reasoning-trace interpretation is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If OPD2 is correct, teacher-base checkpoints become reusable assets for post-training: a single extra forward pass through the base model turns any teacher into a reasoning-transfer supervisor without needing a reward model.
  • The reported 4B OPD2 model outperforms the 8B model trained with either OPD or ExOPD on math, implying that delta-signal distillation can substitute for some model-scale gains.
  • The method improves already strong reasoning models in thinking mode, where standard on-policy distillation typically hurts, suggesting the delta signal is more selective about what to change.
  • Because all reported numbers are at the final training step rather than a selected peak, the advantage is not an artifact of early-training checkpoint selection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The delta signal's effectiveness may depend on how cleanly the teacher-base checkpoint isolates reasoning tuning; if the base also differs in chat formatting, alignment, or data distribution, part of the gain may come from suppressing those differences rather than from reasoning knowledge per se.
  • Editorial inference: One testable extension is to build the delta signal from a base checkpoint that was fine-tuned on non-reasoning instructional data; if OPD2's gains persist, the 'reasoning trace' interpretation would need revision.
  • Editorial inference: The gating condition resembles a variance-reduction or trust-region device; a natural variant would be a soft gate or a learned threshold, which might yield further gains on code benchmarks where the method currently preserves rather than improves the original model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes On-Policy Delta Distillation (OPD2), a modification of on-policy distillation (OPD) in which the standard token-level reward log π* − log π_θ is replaced by a delta signal log π* − log π*_base, where π*_base is the teacher's pre-reasoning-tuning base checkpoint. The method also introduces a centered advantage (Eq. 7) and a gating condition (Eq. 8) that combines the delta advantage with the original OPD advantage. Experiments across Qwen3-1.7B/4B/8B and Gemma4-E4B on 14 math, science, and code benchmarks are reported, with OPD2 consistently outperforming OPD and ExOPD. An ablation study (Table 8) indicates that the delta signal is the main contributor to the gains.

Significance. If the empirical ranking holds, the paper makes a useful and simple contribution to on-policy distillation: a fixed, parameter-free reward modification that improves post-training performance across model families and reasoning domains, while adding only modest compute. Strengths include the breadth of evaluation (three domains, two model families, multiple sizes, both thinking and non-thinking modes), the ablation isolating the delta signal, and a transparent computational-cost comparison. The central conceptual claim, however, is that the delta signal isolates the 'learning trace' of reasoning tuning; this interpretation is not yet backed by a controlled test, and the ablation shows the delta signal is precisely the component responsible for the gains, so the interpretation is load-bearing for the paper's narrative. The main empirical results also lack variance or confidence intervals, which weakens the statistical support for the consistency claim.

major comments (2)
  1. [§2.2, Eq. (5)] The central interpretive claim—that R_delta = log π* − log π*_base 'represents the learning trace of the teacher model from its own base model'—assumes the base checkpoint is a valid counterfactual for reasoning tuning. In practice, the base and post-trained checkpoints also differ in chat template, special tokens, alignment, and stylistic preferences. The word-cloud and token-level analyses (§2.2, Figs. 2–3, Table 1) are correlational and do not establish that the delta isolates reasoning content. Since Table 8 shows the delta signal is the main source of the performance gain, this assumption is load-bearing. Please add a controlled test: for example, compare R_delta against the difference between the teacher and an alternative base/pretrained checkpoint of similar capability, estimate the contribution of format/template tokens to R_delta, or test whether R_delta predicts reasoning corr
  2. [§3.2, Tables 2–7] All main results are pass@1 means averaged over repetitions, but no variance, confidence intervals, or number of independent training runs are reported. Several head-to-head differences are small (e.g., Qwen3-4B thinking CodeForces: 36.1 vs. 36.3 in Table 5; Table 8 shows no-condition non-thinking Math 55.8 vs. OPD2 54.6). Since the central claim is that OPD2 consistently outperforms both baselines, this lack of uncertainty quantification makes it difficult to rule out noise for the smaller per-benchmark gaps. Please report standard errors or confidence intervals for the main tables (and ideally per-repetition values), and state the number of training runs per method.
minor comments (5)
  1. [§3.4, Table 8] The text says the condition and centering 'generally contribute to the overall performance,' but the table shows the no-condition variant improves non-thinking Math (55.8 vs. 54.6) and the no-centering variant improves thinking Math (63.1 vs. 62.7). Please either reconcile this with the text or soften the claim, and consider reporting variance for the ablation as well.
  2. [§2.3, Eq. (8)] The gating condition is presented as addressing a convergence-point issue, but the argument only shows that the gate is zero when π_θ = π*. It does not demonstrate that the condition prevents the one-hot collapse or training instability described in §2.3. An empirical check (e.g., entropy or gradient-norm statistics during training) would make the motivation concrete.
  3. [§3.1] The paper states that code will be available but does not provide a working link or exact checkpoint versions. For reproducibility, please include precise model identifiers, dataset versions, and evaluation-harness versions in the appendix.
  4. [§2.2, Fig. 3] The token-level examples in Fig. 3 use fixed synthetic inputs rather than student rollouts. Since Eq. (3) is explicitly on-policy, please clarify in the caption or text whether these visualizations are intended only as intuition and not as a simulation of the training-time signal distribution.
  5. [Throughout] Minor presentation issues: 'logity_t' in §2.2 appears to be a typo for 'logit y_t'; the author affiliation line has an odd spacing ('NA VER AI Lab'); and the LaTeX formatting of R_delta and OPD2 is inconsistent in places.

Circularity Check

0 steps flagged

No significant circularity: the delta-signal reward is defined directly from the teacher and its base checkpoint, and the reported gains are measured against external benchmarks rather than derived from a fitted quantity.

full rationale

The paper's central reward, R_delta_t = log pi*(y_t|x,y<t) - log pi*_base(y_t|x,y<t) (Eq. 5), is a fixed functional of two teacher checkpoints; it contains no parameter fitted to the evaluation benchmarks. The advantage formulations (Eqs. 6-8) are also deterministic functions of teacher, base, and student probabilities, with no free constants selected to force the reported results. The word-cloud and token-level analyses are motivating illustrations, not fitting procedures, and the held-out benchmark comparisons to OPD and ExOPD are external evidence that does not presuppose the method's success. The ablation (Table 8) attributes the gain to the delta signal empirically rather than by construction. The interpretative claim that the delta signal 'represents the learning trace' is an empirical/causal assumption about the base checkpoint being a valid counterfactual; while this is not rigorously validated, it is not a circularity because the benchmark predictions do not rely on that assumption being true. No load-bearing self-citation chain is present: the related ExOPD work is cited as prior art and the paper differentiates its objective rather than importing a uniqueness or ansatz from the same authors. The derivation chain is therefore self-contained with respect to the formal equations; any concern about the causal meaning of the delta signal belongs to external validity, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central method has no physical or ontological invented entities; the delta signal is a constructed training objective, not a new entity. The main free parameters are standard training hyperparameters plus the top-k expectation truncation and reward scale. The load-bearing axioms concern the interpretation of the teacher-base difference and benchmark cleanliness.

free parameters (5)
  • reward scale (gradient multiplier) = 0.1
    All methods uniformly scale rewards by 0.1 to avoid excessive gradient clipping; chosen by hand and can affect training dynamics.
  • top-k for centering expectation = 1024
    The expected reward in Eqs. 6-7 is approximated over the 1024 highest-probability student tokens; the full-vocabulary expectation is not computed.
  • training steps = 100
    All methods are trained for 100 optimization steps to represent a short post-training regime; results are reported at the final step, not the peak.
  • sampling temperature = 0.7
    Rollout temperature is set to 0.7, which changes the on-policy sampling distribution and therefore the rewards.
  • ExOPD extrapolation coefficient lambda = 1.25
    The ExOPD baseline is run with lambda = 1.25, taken from the original ExOPD paper; the comparison depends on this choice.
axioms (5)
  • domain assumption The teacher's base checkpoint is the exact pre-reasoning-tuning ancestor of the teacher.
    Eq. 5 requires that pi*_base differs from pi* only by post-training so that the delta can be interpreted as the reasoning-tuning trace. Invoked in Sec. 2.2 and model setup in Table 10.
  • domain assumption The delta signal primarily captures reasoning-related knowledge rather than style, format, or alignment artifacts.
    The paper's central motivation is that log pi* - log pi*_base isolates reasoning ability. This is supported by word clouds and token examples, but not by a controlled causal test.
  • domain assumption One completion per question with token-level rewards gives a valid on-policy gradient estimate.
    The training setup samples one rollout per question and applies token-level REINFORCE-style updates; this assumes the resulting gradient is a sufficient estimator for the optimization objective.
  • domain assumption Evaluation benchmarks are not contaminated by the training datasets.
    Training uses questions from OpenMathReasoning, OpenScienceReasoning-2, and OpenCodeReasoning; overlap with evaluation benchmarks such as AIME, MATH500, or CodeContests is not discussed.
  • standard math The KL-divergence partial derivative justifies the token-level reward form in Eq. 2.
    Eq. 2 is derived from the partial derivative of KL(pi_theta || pi*) with respect to the sampled-token probability; this is a standard calculus step and is not itself in question.

pith-pipeline@v1.3.0-alltime-deepseek · 19477 in / 10517 out tokens · 79258 ms · 2026-08-01T23:56:14.397732+00:00 · methodology

0 comments
read the original abstract

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

Figures

Figures reproduced from arXiv: 2607.15161 by Byeongho Heo, Dongyoon Han, Jaehui Hwang, Sangdoo Yun.

Figure 1
Figure 1. Figure 1: Comparison of On-Policy Distillation (OPD) and our OPD2 . While conventional on-policy distillation trains students to follow the teacher’s outputs, our OPD2 utilizes the delta signal, the difference between the teacher and its base model, to focus on the knowledge of the learning trajectory required for reasoning capability. RL, the learning signal is applied only to the sampled tokens; thus, the student’… view at source ↗
Figure 2
Figure 2. Figure 2: Word clouds for OPD, Base and ∆. The figure illustrates distillation signal strengths in the form of word clouds. Qwen3-1.7B and Qwen3-4B are used for student and teacher, respectively, on 10k math questions. Compared with OPD, ∆ emphasizes reasoning-connective words such as hence, however, and instead, while suppressing exploratory and verification-related words such as see, try, and verify. where π ∗ pre… view at source ↗
Figure 3
Figure 3. Figure 3: Token-level distillation signals on simple reasoning examples. Blue and red indicate promoting and suppressing signals, respectively, with values clipped to ±6. Compared with OPD, ∆ more consistently suppresses tokens associated with incorrect reasoning. Token-level signals. The most representative characteristics of on-policy distillation are the token￾level rewards and supervision. It is hard to achieve … view at source ↗
Figure 4
Figure 4. Figure 4: Training curve for various domains. The figures show the reasoning performance changes during on-policy distillation training. We report the average performance on AIME24 and AIME25 for Math, CodeContests for Code, and GPQA for Science. All OPD variants exhibit similar training dynamics, while OPD2 achieves the strongest performance. Note that, in all tables, we report performance at the final training ste… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 19 linked inside Pith

  1. [1]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 1

    2018

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1

    1901

  3. [4]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016. 1

    2016

  4. [5]

    Bild: Bi-directional logits difference loss for large language model distillation

    Minchong Li, Feng Zhou, and Xiaohui Song. Bild: Bi-directional logits difference loss for large language model distillation. InProceedings of the 31st International Conference on Computational Linguistics, pages 1168–1182, 2025. 2

    2025

  5. [6]

    Pre-training distillation for large language models: A design space exploration

    Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, and Juanzi Li. Pre-training distillation for large language models: A design space exploration. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3603–3618, 2025. 2

    2025

  6. [7]

    Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026

    Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406, 2026. 2

    Pith/arXiv arXiv 2026

  7. [8]

    An emulator for fine-tuning large language models using small language models.arXiv preprint arXiv:2310.12962, 2023

    Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models.arXiv preprint arXiv:2310.12962, 2023. 2

    Pith/arXiv arXiv 2023

  8. [9]

    Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset.arXiv preprint arXiv:2504.16891, 2025

    Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset.arXiv preprint arXiv:2504.16891, 2025. 3, 4, 6, 8

    Pith/arXiv arXiv 2025

  9. [10]

    Opensciencereasoning-2

    NVIDIA Corporation. Opensciencereasoning-2. https://huggingface.co/datasets/nvidia/ OpenScienceReasoning-2, 2025. Hugging Face dataset. 3, 6, 8

    2025

  10. [11]

    Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025

    Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jo- celyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025. 3, 6, 8

    Pith/arXiv arXiv 2025

  11. [12]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  12. [13]

    Gemma 4 technical report.arXiv preprint arXiv:2607.02770, 2026

    Gemma Team. Gemma 4 technical report.arXiv preprint arXiv:2607.02770, 2026. 3, 7

    Pith/arXiv arXiv 2026

  13. [14]

    Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026. 3, 8, 13

    Pith/arXiv arXiv 2026

  14. [15]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Pith/arXiv arXiv

  15. [16]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992. 6

    1992

  16. [17]

    Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999. 6 14

    1999

  17. [18]

    Asynchronous methods for deep reinforcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PmLR, 2016. 6

    1928

  18. [19]

    TRL: Transformers reinforcement learning, 2020

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers reinforcement learning, 2020. Open-source toolkit for transformer model post-training. 8

    2020

  19. [20]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 8

    Pith/arXiv arXiv 2017

  20. [21]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pages 611–626,

  21. [22]

    Evalchemy, November 2024

    Etash Guha, Negin Raoff, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad-Falcon, Caroline Choi, Kushal Arora, Mike Merrill, Yichuan Deng, Ashima Suvarna, Hritik Bansal, Marianna Nezhurina, Reinhard Heckel, Seewong Oh, Tatsunori Hashimoto, Jenia Jitsev, Yejin Choi, Vaishaal Shankar, Alex Dimakis, Mahes...

    2024

  22. [23]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    2024

  23. [24]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. InAmerican Invitational Mathematics Examination - AIME 2024, February 2024. 8

    2024

  24. [25]

    American invitational mathematics examination - aime

    MAA. American invitational mathematics examination - aime. InAmerican Invitational Mathematics Examination - AIME 2025, February 2025. 8

    2025

  25. [26]

    American mathematics competition - amc

    MAA. American mathematics competition - amc. InAmerican Mathematics Competition - AMC. 8

  26. [27]

    Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms

    Jasper Dekoninck, Nikola Jovanovi´c, Tim Gehrunger, Kári Rögnvaldsson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms

  27. [28]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. 8

    2021

  28. [29]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems,

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems,

  29. [30]

    Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025

    Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025. 8

    2025

  30. [31]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, P...

    2022

  31. [33]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024. 8

    Pith/arXiv arXiv 2024

  32. [34]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst conference on language modeling, 2024. 8 15

    2024

  33. [35]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025

    M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tianshun Xing, Ming Xu, Zhenzhu Ya...

    2025

  34. [36]

    Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating College-Level Scientific Problem- Solving Abilities of Large Language Models. InProceedings of the Forty-First International Conference on Machine Learning, 2024. 8

    2024

  35. [37]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015. 13

    Pith/arXiv arXiv 2015

  36. [38]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016. 13

    2016

  37. [39]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models, 2024. 13

    2024

  38. [40]

    Openthoughts: Data recipes for reasoning models

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025. 13

    Pith/arXiv arXiv 2025

  39. [41]

    Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015. 13

    2015

  40. [42]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024. 13

    2024

  41. [43]

    Minillm: On-policy distillation of large language models, 2026

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: On-policy distillation of large language models, 2026. 13

    2026

  42. [44]

    Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898, 2024

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898, 2024. 13

    Pith/arXiv arXiv 2024

  43. [45]

    On-policy distillation.Thinking Machines Lab: Connectionism,

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connectionism,

  44. [46]

    Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026. 13

    Pith/arXiv arXiv 2026

  45. [47]

    Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026. 13

    arXiv 2026

  46. [48]

    Self- distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self- distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Pith/arXiv arXiv

  47. [49]

    Crisp: Compressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433, 2026

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433, 2026. 13

    Pith/arXiv arXiv 2026

  48. [50]

    Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472, 2026

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472, 2026. 13 16

    Pith/arXiv arXiv 2026

  49. [51]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026. 13

    Pith/arXiv arXiv 2026

  50. [52]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026. 13 17 A Appendix Tables 10 and 11 summarize the model-specific configurations and common optimization settings used in our experiments. We use NVIDIA H100 GPUs for all experiments. Qwen3-1.7B is trained on a single 8-GPU node, w...

    Pith/arXiv arXiv 2026

  51. [2025]

    https://thinkingmachines.ai/blog/on-policy-distillation. 13