Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

A repetition-gated budget recovers pruned LLM generation at 9x with 71% fewer rollout tokens

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:07 UTC pith:D5S3VJRV

load-bearing objection The controller saves compute but not final quality; the paper needs a fixed-short baseline and a loop-token ablation to support the lossless-savings claim. the 3 major comments →

arxiv 2607.13124 v1 pith:D5S3VJRV submitted 2026-07-14 cs.LG cs.AIcs.CL

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

Qingyu Zhang , Qianhao Yuan , Hongyu Lin , Yaojie Lu , Xianpei Han , Le Sun , Xiang Li , Ming Xu
show 2 more authors
Jiarui Li Xiuyin Zhao
This is my paper
classification cs.LG cs.AIcs.CL
keywords structured pruningon-policy distillationLLM recoveryrepetition detectionrollout budget controlknowledge distillationgeneration qualitypost-compression retraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Structured pruning removes layers from a large language model to make it cheaper to serve, but the pruned model often still passes multiple-choice tests while failing to generate coherent text. This paper argues that the lost ability is not erased, only demoted, and that the right repair is to train the pruned model on its own sampled outputs, using the original unpruned model as a frozen teacher at every token. The paper's contribution is a budget controller that notices when early rollouts collapse into repetitive loops, shortens the rollout length to skip those low-information suffixes, and gradually lengthens it again as the model recovers. On math, code, and open-ended tasks, this raises the pruned model's average score to about nine times its unrecovered value, beating standard distillation baselines and matching a much more expensive fixed 8192-token rollout schedule within two points. A sympathetic reader would care because it makes structured pruning viable for actual deployment, not just benchmark recognition tasks.

Core claim

The discovery is that compressed language models retain correct generation trajectories in their sampling distribution, but greedy decoding fails and on-policy rollouts initially degrade into repetition; therefore recovery works by dense token-level distillation on the student's own states, and the efficiency bottleneck is the repetitive suffix. ShortOPD's controller detects teacher-confirmed loops, reduces the per-step rollout budget to an EMA of the usable prefix, and grows it when truncation signals that longer rollouts are clean, delivering an average score of 48.5 versus 5.7 unrecovered and within 1.7 points of a fixed 8192-token horizon at 71% fewer rollout tokens.

What carries the argument

The repetition-gated, truncation-aware horizon controller, which sits around the on-policy distillation objective. A terminal periodic-loop detector finds severe suffix repetition, the existing distillation loss and teacher NLL refine the effective length, and two exponential moving averages smooth batch statistics and budget changes. This controller decides whether to shrink, hold, or grow the per-step rollout budget, while the underlying training loss is a generalized Jensen-Shannon divergence with top-100-plus-tail aggregation and clipped importance weighting.

Load-bearing premise

The load-bearing premise is that near-agreement on the repetitive tail means those tokens teach nothing, so truncating the rollout at the loop onset throws away no useful signal.

What would settle it

Run a version of ShortOPD that keeps full-length rollouts or trains explicitly on loop tokens under the same wall-clock budget; if it matches or beats ShortOPD's average score, the premise that repeated suffixes are information-free is wrong. A second check is to compute per-token gradients inside the aggregated tail bin on loop states—the appendix proves only that binned agreement gives zero gradient, so nonzero hidden tail-bin gradients would also falsify the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Recovery after structured pruning should be treated as distributional repair: the compressed model's own rollouts are the right training states, not fixed offline sequences.
  • Dense token-level teacher distributions beat sparse rewards when the damaged model rarely samples a correct answer, as shown by the near-zero GSM8K gains of PPO and GRPO on the same prompts.
  • The budget savings are large and concrete: ShortOPD matches a fixed 8192-token horizon within 1.7 average points while using 250M vs 869M rollout tokens and 8.5 vs 35.9 training hours.
  • Recovery continues to improve with more exposure, rising from 48.5 to 55.4 average score over three epochs, so the recipe is scalable rather than a one-shot fix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If loop tokens truly carry no learning signal, the same repetition-gated budget idea could speed up other on-policy training regimes where early rollouts degenerate, such as RLHF or self-improvement loops, not just post-pruning distillation.
  • The paper's demonstration that recognition and generation recover along partly independent axes suggests evaluation suites for compression should treat free-form generation as a first-class benchmark; the paper implies this but does not itself design such a suite.
  • A testable extension is to use the repetition detector as a diagnostic for recoverability: the paper observes that repetition is the characteristic failure mode only until roughly 25% of layers are removed, so the detector might mark how much compression a given model family can tolerate before light recovery stops working.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ShortOPD, a repetition-gated dynamic rollout budget for on-policy distillation (OPD) used to recover the free-form generation quality of structurally pruned LLMs. The method detects teacher-confirmed repetitive suffixes, truncates the effective rollout length, and adaptively grows the budget as repetition subsides. On a 25%-pruned Qwen3-4B-Instruct, the authors report that ShortOPD restores about two-thirds of the dense teacher's average generation score and, compared with a fixed 8192-token horizon, uses 71% fewer rollout tokens and ~4x less wall-clock time while staying within two average points. The paper also includes ablations of the learning signal and recovery-corpus domain coverage.

Significance. If the efficiency claim holds, ShortOPD is a practically valuable recipe for post-pruning recovery, a step beyond the common recognition-benchmark validation of structured pruning. The paper is well positioned against prior work, provides reproducible code links, and includes several controlled comparisons (off-policy baselines, sparse-reward RLVR, leave-one-domain-out corpus ablations, and a multiple-choice sanity check). The on-policy self-distillation premise is well motivated by the pass@k and repetition observations. However, the central efficiency contribution hinges on the controller's causal role and on the premise that repetitive-suffix tokens carry no useful supervision; both need sharper evidence.

major comments (3)
  1. [Section 4.3 / Figure 5b] The contributions (Section 1) claim ShortOPD 'improves recovery over fixed short and fixed long horizons,' but Figure 5b shows Avg 48.5 for ShortOPD vs 49.6 for fixed H=2048 and 50.2 for fixed H=8192. ShortOPD is 1.1 points below the fixed short horizon. The abstract's 'matches within two points' is accurate, but the stronger claim is contradicted. Please correct the wording or provide multiple seeds/error bars to show the difference is within noise.
  2. [Section 4.3 / Algorithm 1] The causal role of the repetition-gated controller is not isolated. The comparison against fixed H=2048 and H=8192 differs in total token budget and wall-clock. There is no matched-token ablation, e.g., fixed H=1024 with the same step count or a non-adaptive short-to-long schedule, so the efficiency gain could be due simply to generating fewer tokens rather than to the controller's gating strategy. This is load-bearing for the paper's unique contribution and should be tested directly.
  3. [Section 1 / Appendix C] The premise that repetitive-suffix tokens carry 'no useful distillation signal' is insufficiently supported. Appendix C proves only a conditional, per-token stationary point; it does not show that the full OPD objective is insensitive to loop-state supervision. The aggregate JSD/NLL statistics (0.0014 vs 0.051; 3e-5 vs 0.68) are consistent with low gradient, but tail-bin differences could later compound, and the on-policy coupling means removing loop states changes gradients at earlier tokens. An ablation that explicitly trains on loop tokens or masks them under a matched rollout-token budget is needed to justify the lossless-cut assumption.
minor comments (5)
  1. [Section 1 / Contributions] The bullet states that ShortOPD 'improves recovery over fixed short and fixed long horizons (Figure 1).' Figure 1 does not contain fixed-horizon baselines; this reference appears to be incorrect and should be changed to Figure 5 or removed.
  2. [Table 4] The '3 epochs / step' row reads 2127, which should be 2130 (710×3). Please verify.
  3. [Algorithm 1] The notation uses the symbol '⊮' for the indicator function in some places and '1[...]' in others (e.g., Eq. 3). Please standardize.
  4. [General] The paper header includes 'Work in Progress'; this is unusual for a journal submission and should be removed or explained in the cover letter.
  5. [References] Several references are dated 2026 (e.g., [16], [24], [26]) and may be arXiv preprints. Please ensure they are accessible and correctly cited.

Circularity Check

0 steps flagged

Central efficiency claim is tested against external fixed-horizon and off-policy baselines; the only self-citations (ShortGPT, Vision-OPD) are building blocks, not load-bearing derivations. No prediction reduces to a fitted input.

full rationale

ShortOPD's central claims are evaluated against independent external baselines: fixed H=2048 and H=8192 OPD, SFT w/o KD, SeqKD, KD, and RLVR. The controller is not fitted to the target metric: the rollout budget is updated from measured repetition/truncation statistics (Eq. 4), while the reported Avg comes from downstream evaluation tasks that never enter the controller. Appendix C is a genuine conditional per-token gradient derivation, explicitly labeled 'a conditional, per-token statement rather than a claim that every small scalar loss implies a small full-objective gradient,' so it is not overclaimed. The only self-citations are ShortGPT [5] as the pruning building block and Vision-OPD [24] as the OPD recipe; both are cited as external tools with public implementations, and the novel repetition-gated budget control is not justified by those citations. No self-definitional step exists: effective length is defined from the terminal-periodic detector (Algorithm 2) and loss/NLL thresholds, not from the final evaluation score. The noted limitation — no ablation explicitly training on loop tokens under matched budgets, and no fixed-short baseline isolating the controller — is a completeness/identifiability concern about the efficiency claim, not circularity: it does not make any stated prediction reduce to its inputs by construction. Therefore no circular step meets the evidentiary bar, and the score is 1.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claim rests on the controller's hand-chosen thresholds and on the assumption that repeated-suffix tokens are low-information; no new entities are introduced. The paper inherits the OPD base recipe and the pruning method as background, but the novel contribution (adaptive budget) depends on several ad hoc constants.

free parameters (7)
  • repetition-rate thresholds = ρ_low=0.20, ρ_high=0.45
    Gate thresholds for shrink/grow in Eq. 4; chosen by hand, no sensitivity analysis.
  • clean truncation threshold = τ=0.10
    Threshold for growth gate when clean truncation rate is high.
  • headroom margin = λ=1.15
    Multiplier on effective length for shrink target.
  • growth rate = γ↑=1.25
    Budget growth factor in Eq. 4.
  • EMA decay = β=β_H=0.7
    Smoothing decays for statistics and budget updates.
  • detector hyperparameters = W=512, P=10, A=32, η=0.9, C=3, L_min=64, L_sev=128, φ=0.30
    Terminal periodic-loop detector settings; hand-selected without robustness study.
  • loss thresholds for effective-length refinement = OPD-loss threshold max(0.05, 0.25·d_bar_base); teacher-NLL threshold max(0.5, 0.25·u_bar_base)
    Used to refine the loop onset; relative to baseline windows, chosen ad hoc.
axioms (5)
  • domain assumption The pre-compression model is a valid teacher for the pruned student; its distribution remains the correct target.
    Recovery distills from the frozen original model; if pruning shifted capabilities such that the original distribution is not the right target, the method could misguide the student. Section 3.1.
  • domain assumption Block-Influence depth pruning at 25% is representative of structured pruning recoverability.
    All main experiments use one pruning method and one ratio; generality to other structured pruning methods is assumed. Section 1 and Limitations.
  • domain assumption Terminal periodic-loop detection correctly identifies low-information suffixes; teacher-student agreement implies vanishing update signal.
    Controller assumes loop tokens carry negligible training signal; Appendix C proves gradient vanishes at exact agreement but empirically relies on binned agreement thresholds. Section 3.2, Appendix C.
  • ad hoc to paper The 25% compression point is the recoverable critical point; at stronger compression the model becomes incoherent and the method is not intended.
    The testbed is chosen based on the repetition-vs-depth sweep (Appendix E); the paper does not claim applicability beyond this regime. Section 1.
  • domain assumption GPT-5.5 judge scores are a reliable proxy for generation quality.
    Open-ended tasks rely on an LLM judge; judge biases are not analyzed. Section 4.1 and Appendix D.

pith-pipeline@v1.3.0-alltime-deepseek · 17134 in / 15191 out tokens · 139095 ms · 2026-08-02T06:07:39.287123+00:00 · methodology

0 comments
read the original abstract

Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires. Two observations trace this gap. First, greedy \textsc{pass}@$1$ nearly vanishes after compression, yet \textsc{pass}@$k$ recovers substantially under repeated sampling: useful generations are demoted, not erased. Second, the recoverable regime fails mainly through suffix repetition. Recovery should therefore train on the compressed model's own on-policy states with dense token-level supervision, which On-Policy Distillation (OPD) provides by reusing the pre-compression model as a frozen teacher. However, long on-policy rollouts spend early recovery budget on low-information repetitive suffixes, delaying loss descent. To mitigate this waste, we propose \textbf{\shortopd}, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes, treats the surviving prefix as each rollout's effective length, and allocates future rollout budgets to the effective lengths the policy can currently use. Across math, code, and open-ended generation, \shortopd\ raises the compressed model's score to about $9\times$ its unrecovered value and $1.6$--$4.4\times$ standard recovery recipes (SFT w/o KD, KD, and SeqKD), and it matches a fixed $8192$-token rollout horizon within two points using a quarter of the training time ($8.5$ vs.\ $35.9$ hours) and $71\%$ fewer rollout tokens. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, a step closer to deployment-ready generation quality.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

    cs.LG 2026-07 conditional novelty 6.0

    Adaptive FastOPD expands the OPD rollout horizon only when boundary-region teacher–student progress plateaus and the horizon is utilized, cutting training time 49–71% while matching or exceeding accuracy.

  2. On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment

    cs.AI 2026-07 conditional novelty 6.0

    Source-routed dual-teacher top-K KL distillation realigns misaligned LLMs with less template dependence and less task collapse than rollback, RESTA, soft-SFT, and SSRD.

Reference graph

Works this paper leans on

44 extracted references · 13 linked inside Pith · cited by 2 Pith papers

  1. [1]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  4. [4]

    Slicegpt: Compress large language models by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. InInternationalConference on Learning Representations (ICLR), 2024

    2024

  5. [5]

    Shortgpt: Layers in large language models are more redundant than you expect

    Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. InFindings of the Association for Computational Linguistics: ACL 2025, 2025

    2025

  6. [6]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  7. [7]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mo- hammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  8. [8]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML), 2023

    2023

  9. [9]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  10. [10]

    Awq: Activation-aware weight quantization for llm compression and acceler- ation

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceler- ation. In Proceedings of Machine Learning and Systems (MLSys), 2024

    2024

  11. [11]

    Fluctuation-based adaptive structured pruning for large language models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. InAAAI Conference on Artificial Intelligence (AAAI), 2024

    2024

  12. [12]

    Shortened llama: Depth pruning for large language models with comparison of retraining methods.arXiv preprint arXiv:2402.02834, 2024

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: Depth pruning for large language models with comparison of retraining methods.arXiv preprint arXiv:2402.02834, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    The unreasonable ineffectiveness of the deeper layers

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  14. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  15. [15]

    Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics (ACL), 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InAnnual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  16. [16]

    The benchmark illusion: Pruned llms can pass multiple choice but fail to answer.arXiv preprint arXiv:2606.17609, 2026

    Rui Wen, Lu Sun, Jiayang Liu, Zesheng Xu, Tianshuo Cong, and Zheng Li. The benchmark illusion: Pruned llms can pass multiple choice but fail to answer.arXiv preprint arXiv:2606.17609, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    Sequence level training with recur- rent neural networks

    Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recur- rent neural networks. InInternational Conference on Learning Representations (ICLR), 2016. 13

    2016

  20. [20]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024

    2024

  21. [21]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Empirical Methods in Natural Language Processing (EMNLP), 2016

    2016

  24. [24]

    Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740, 2026

    Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020

    2020

  26. [26]

    Learning to break the loop: Analyzing and mitigating repetitions for neural text generation

    Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  27. [27]

    Laco: Large language model pruning via layer collapse

    Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. InFindings of the Association for Computational Linguistics: EMNLP 2024, 2024

    2024

  28. [28]

    Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers

    Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 329–339, 2025

    2025

  29. [29]

    Everybody prune now: Structured pruning of llms with only forward passes.arXiv preprint arXiv:2402.05406, 2024

    Lucio Dery, Steven Kolawole, Jean-François Kagey, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of llms with only forward passes.arXiv preprint arXiv:2402.05406, 2024

    arXiv 2024

  30. [30]

    Llm pruning and distillation in practice: The minitron approach

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Maha- baleshwarkar, Gerald Shen, et al. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408.11796, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  32. [32]

    Autoregressive knowledge distillation through imitation learning

    Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. InEmpirical Methods in Natural Language Processing (EMNLP), 2020

    2020

  33. [33]

    Minillm: On-policy distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: On-policy distillation of large language models. In International Conference on Learning Representations (ICLR), 2024

    2024

  34. [34]

    f-divergence minimization for sequence-level knowledge distillation

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023

    2023

  35. [35]

    Distillm: Towards streamlined distillation for large language models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. InInternational Conference on Machine Learning (ICML), 2024

    2024

  36. [36]

    Tulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, et al. Tulu 3: Pushing frontiers in open language model post-training. InConference on Language Modeling (COLM), 2025

    2025

  37. [37]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, et al. Dapo: An open-source llm reinforcement learning system at scale. InAdvancesin Neural Information Processing Systems (NeurIPS), 2025

    2025

  38. [38]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  39. [39]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS Datasets and Benchmarks, 2021. 14

    2021

  40. [40]

    Opencodeinstruct.https://huggingface.co/datasets/nvidia/OpenCodeInstruct, 2024

    NVIDIA. Opencodeinstruct.https://huggingface.co/datasets/nvidia/OpenCodeInstruct, 2024

    2024

  41. [41]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  42. [42]

    Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality.https://lmsys.org/blog/2023-03-30-vicuna/, 2023

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality.https://lmsys.org/blog/2023-03-30-vicuna/, 2023

    2023

  43. [43]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. InEmpirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  44. [44]

    ).").")

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023. 15 Appendix A Training...

    2023