Pith. sign in

REVIEW 3 major objections 6 minor 45 references

Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an easy-to-hard curriculum schedule with rising temperature consistently improves white-box knowledge distillation of large language models.

desk verdict The paper's own ablations show the curriculum ordering is not the causal ingredient—the rising temperature schedule is—so the headline claim is overstated, but the empirical package is real and worth a referee's time. read the letter →

arxiv 2506.05695 v1 pith:T55LJSKL submitted 2025-06-06 cs.CL cs.AIcs.LG

Lingyuan Liu , Mengxiang Zhang This is my paper
classification cs.CLcs.AIcs.LG
keywords knowledgedistillationcurriculumlearninglargelanguagemodelswhite-boxdifficultyrankingtemperatureschedulinginstructionfollowingreciprocalrankfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that ordering the data used in white-box knowledge distillation from easy to hard, a curriculum it calls POCL, makes distilled language models consistently better on five instruction-following benchmarks, across two model families and six distillation losses. The framework ranks each training sample using a fusion of the student model's ROUGE-L and cross-entropy scores, partitions the data into four difficulty tiers, and feeds the tiers to the student progressively while the distillation temperature rises from 1 to 2. Because POCL is a wrapper around the existing KD loss, the authors argue it is cheap and portable: it does not change the loss or the teacher, only the order and schedule of training. The larger point is that data structure inside distillation is itself a performance lever.

What carries the argument

The load-bearing machinery is POCL, a two-part framework: a difficulty measurer and a training scheduler. The difficulty measurer computes two rankings per training sample, one by ROUGE-L between student output and ground truth and one by student cross-entropy, and merges them through reciprocal rank fusion, $FR_{\mathrm{score}} = \sum_i 1/(k + r_i)$ with $k = 60$, partitioning the 12.5K training samples into four subsets from easiest to hardest. The scheduler, called Baby Step, starts training on the easiest subset and adds the next subset at fixed intervals or convergence, while the distillation temperature $\tau$ grows linearly from 1 to 2 and the SFT ratio $\alpha$ falls from 0.3 to 0 for off-policy methods. The framework's role is to control the order and sharpness of the distributions the student is matched against, which the paper argues prevents abrupt distribution shifts during distillation.

What would settle it

Train the same student on random-order sample subsets while keeping the identical rising-temperature and alpha schedules; if ROUGE-L matches POCL, the easy-to-hard ordering contributes nothing. A cheaper check is already in the paper: POCL with temperature and ratio removed scores 23.21 versus 23.49 for plain KLD on DollyEval, so an experiment that fixes the temperature at 1 while varying only the order would determine whether the ordering claim is right.

Watch

Extended reading notes

Core claim

The central claim is that a curriculum schedule, not a new loss function, is what stabilizes and improves white-box KD of LLMs. In the paper's terms, the difficulty measurer ranks samples by reciprocal rank fusion over ROUGE-L and cross-entropy; the Baby Step scheduler then trains on the easiest subset first and adds harder subsets at successive stages, while the distillation temperature is linearly raised from 1 to 2 and off-policy methods shift their SFT weighting from 0.3 down to 0. Across KLD, RKL, JSD, TVD, SKL, SRKL, and GKD, POCL raises average ROUGE-L relative to plain baselines, with average gains between roughly 0.5 and 2.6 points depending on method and student model. The authors interpret this as evidence that progressive overload reduces teacher-student distribution shift, mitigates catastrophic forgetting and mode collapse, and eases training-inference mismatch.

Load-bearing premise

The load-bearing premise is that easy-to-hard sample ordering is the active ingredient, but the paper's own Table 3 shows that ordering without the temperature schedule does not beat plain KLD, so the mechanism attributed to the curriculum is not cleanly isolated.

Editorial extensions

If this is right

  • Adding POCL to any of the tested white-box KD losses raises average ROUGE-L across five instruction-following benchmarks for both the GPT-2 0.1B and OPT 0.3B students.
  • Easy-to-hard ordering outperforms hard-to-easy ordering, although hard-to-easy still improves over plain training in most settings.
  • The rising temperature component is necessary for the reported gains; removing it leaves POCL at or below the unmodified baseline.
  • Because POCL is a wrapper, it can be dropped onto existing distillation loops without changing the loss or the teacher, at roughly the same total training steps.
  • Staged training reaches higher validation ROUGE-L earlier than the corresponding KD baseline, indicating faster convergence as well as better final performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the active ingredient may be the temperature schedule rather than the curriculum order, since the paper's own Table 3 shows ordering-only POCL loses to plain KLD on DollyEval; a randomized-order control with the same temperature schedule would settle this.
  • A directly testable extension is to hold the rising-temperature schedule fixed and randomize sample order within stages; if ROUGE-L is unchanged, the easy-to-hard ordering is not carrying the effect.
  • Because POCL's difficulty measure uses the student model's own confidence, the ranking is capacity-dependent; as student models grow, the perceived difficulty spread may narrow and the gains may shrink, a possibility the paper itself raises in its limitations.
  • The same staging idea could transfer to black-box distillation by substituting a proxy difficulty signal such as teacher output entropy, but the paper does not demonstrate that transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes POCL, a curriculum-learning plug-in for white-box knowledge distillation of LLMs. POCL ranks training samples by difficulty using reciprocal rank fusion of the student's Rouge-L and cross-entropy scores, partitions the data into n difficulty subsets, and then trains the student in n stages, adding harder subsets while linearly increasing the distillation temperature and (for off-policy methods) decreasing the SFT ratio. The authors evaluate POCL on GPT-2 and OPT with KLD, RKL, JSD, TVD, SKL, SRKL, and GKD, reporting Rouge-L improvements on five instruction-following benchmarks over a plain-SFT/SeqKD/white-box-KD baseline set.

Significance. If the full POCL recipe is robust, it would be a simple, plug-in training schedule that improves white-box KD across several loss functions and two model families, and the public code is a strength. The paper's breadth of base KD methods and its component-wise ablations are useful. However, the paper's own ablation isolating the curriculum-ordering component does not beat plain KLD, so the significance as a curriculum-learning contribution is not currently established; the paper would need to be reframed around the adaptive-temperature schedule or supported by a controlled experiment showing that difficulty ordering alone helps.

major comments (3)
  1. [Section 5.2, Tables 3 and 6] The variant that isolates the curriculum-ordering component—KLD+POCL (w/o temp. & ratio)—does not improve over plain KLD on DollyEval (23.21 vs 23.49), S-NI (18.97 vs 19.70), or UnNI (21.85 vs 22.01), and the same pattern appears for JSD (22.73 vs 23.79) and for GKD when temperature is removed (24.05 vs 24.67). Section 5.2 itself states that the pure POCL framework "underperforms compared to baselines without POCL," and Table 7 shows that hard-to-easy ordering also improves over the baseline in most cells. This directly contradicts the abstract's claim that "sorted training samples" are effective and the mechanism stated in Section 3.2. The paper must either reframe the contribution as an adaptive-temperature/ratio training schedule or provide a controlled experiment showing that difficulty ordering alone yields gains (for example, staged easy-to-hard without temperature versus staged random-order without temperature).
  2. [Section 5.1, Tables 2 and 5] The conclusion that "all variants of KLD with POCL outperform standard KLD without POCL, demonstrating the benefit of curriculum-based sample ordering" is invalid because those POCL variants include the adaptive temperature and ratio schedules. This comparison cannot separate the ranking method from the rest of the POCL pipeline, so it does not establish a benefit of curriculum-based sample ordering per se. The ranking-method comparison within POCL is informative, but the ordering-benefit claim is conflated with the adaptive-parameter schedules.
  3. [Tables 1 and 4] The paper reports means over five random seeds but provides no standard deviations, confidence intervals, or significance tests. Many of the reported gains are small (for example, OPT RKL +0.48 average, OPT JSD +0.77, OPT TVD +0.61), and several per-dataset cells decrease (for example, OPT JSD on SelfInst 12.57 vs 12.97 and on S-NI 23.30 vs 23.51; OPT RKL on S-NI 23.57 vs 23.92). Without variance estimates or a paired significance test, the "consistently improves" claim is not supported at the reported granularity. Please add per-seed results or error bars and perform paired significance tests, or explicitly downgrade the claim to "often improves in these experiments."
minor comments (6)
  1. [Section 4.1] The base models are described as "GPT-2 (120M) and OPT (350B)," but the actual student models are GPT-2 0.1B and OPT 0.3B; "350B" should be "350M." The same typo appears in Table 1's caption where "GPT2-120M" is used.
  2. [Algorithm 1 and Appendix A.2.2] Algorithm 1 says "while not converged for p epochs" but p is not defined in the input list, and "converged" is not operationalized. Appendix A.2.2 later says the framework is trained for 8 epochs per stage and for 40% of baseline epochs; please make the training-length accounting explicit and consistent between the algorithm, Section 4.1, and the appendix.
  3. [Equation (2)] The displayed formula for FRscore is typeset incorrectly ("nX i") and should be a sum over the ranked lists i; please fix the notation so that ri is clearly the rank of the sample in list i.
  4. [Tables 1 and 4, notes] The table notes attribute GKD to reference [1], but the GKD method is described in the text as reference [4]; the citation is inconsistent. Please correct the reference.
  5. [Figure 4 caption] In the caption for Figure 4, "STF ratio" should be "SFT ratio."
  6. [Section 2.3] The text says "Our theoretical analysis suggests" that CL addresses catastrophic forgetting and training-inference mismatch, but no formal theoretical analysis appears in the paper; the section contains only intuitive arguments. Either remove the phrase "theoretical analysis" or provide a precise statement with supporting argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: POCL's components are empirical training inputs, not restatements of the evaluation, and there are no load-bearing self-citations.

full rationale

The paper's claimed derivation chain is not circular. POCL's two components—the difficulty measurer (Eq. 2, fusing student Rouge-L and cross-entropy ranks) and the Baby Step scheduler (Eqs. 3–4, staged introduction plus temperature and alpha schedules)—are training-time inputs, not restatements of the evaluation outcome. The student is optimized with cross-entropy and KD losses, not by directly maximizing the Rouge-L metric, and the sample ordering is a one-time curriculum signal computed from the initial student model. No parameter is fitted to the reported test scores in a way that would rename the fit as a prediction, and hyperparameters such as n=4, tau0=1, tau_n=2, alpha0=0.3, and alpha_n=0 are fixed or selected on a validation split. There are no load-bearing self-citations: none of the references are to the authors' own prior work, and the cited Baby Step [33], reciprocal rank fusion [32], and curriculum-temperature [41] results are external. The informal 'theoretical analysis' in Sec. 2.3 is a derivation gap, not a self-referential reduction. The most serious weakness is an attribution confound, disclosed in Sec. 5.2: the pure curriculum scheduler without temperature/ratio ('+POCL (w/o temp. & ratio)') scores 23.21 vs. 23.49 for plain KLD on DollyEval and is also worse on S-NI and UnNI, so 'sorted training samples' is not established as the causal driver. That is a correctness/interpretation concern, not circularity, so the circularity score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method relies on several hand-set hyperparameters and on unproved assumptions about what makes samples 'easy' and why easy-to-hard ordering should help. The key assumption about the curriculum's benefit is directly tested and fails in the paper's own ablation, though the full recipe (with temperature scheduling) still works.

free parameters (7)
  • n (number of difficulty subsets) = 4
    Hand-chosen; authors state it is empirically good and not tuned per dataset.
  • k (reciprocal rank fusion constant) = 60
    Taken from RRF reference [32], dampens outlier ranks.
  • tau0, tau_n (temperature endpoints) = 1, 2
    Empirically fixed endpoints of the temperature schedule in Eq. 3.
  • alpha0, alpha_n (SFT ratio endpoints, off-policy) = 0.3, 0
    Empirically fixed SFT ratio schedule for off-policy KD in Eq. 4.
  • alpha (on-policy) = 0
    Set constant for on-policy KD, as stated in Sec 3.3.
  • p (epochs per stage) = 2 (implied)
    Algorithm 1 says 'while not converged for p epochs'; total 8 epochs over 4 stages implies p=2, but p is not explicitly reported.
  • learning rate and batch size = chosen from {5e-4, 1e-4, 5e-5} x {8, 16}
    Selected based on validation Rouge-L, as stated in Sec A.2.2.
assumptions (4)
  • domain assumption ROUGE-L and cross-entropy loss are valid difficulty measures for curriculum ordering.
    No justification that these metrics correlate with learning benefit or sample difficulty.
  • ad hoc to paper Progressive difficulty exposure stabilizes KD and mitigates distribution shift.
    Presented as 'theoretical analysis' in Sec 2.3 but no derivation is given; contradicted by Table 3 ablation.
  • domain assumption Higher distillation temperature late in training improves distillation.
    Known from prior work [41]; assumed here and supported by the paper's ablations.
  • domain assumption The initial student model's output provides a stable difficulty signal.
    Difficulty is computed once before training (Algorithm 1), so the signal depends on the initial student model's quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework." pith.science (2026). https://pith.science/paper/T55LJSKL

@misc{pith2026250605695,
  author       = {Pith},
  title        = {Pith review of: Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T55LJSKL}},
  note         = {Machine review of arXiv:2506.05695}
}
read the original abstract

Knowledge Distillation (KD) compresses large language models (LLMs) by transferring the teacher model's capabilities to a smaller student model, reducing inference cost and memory usage while maintaining performance. However, existing KD methods for LLMs often fail to prevent significant shifts in the student model's distribution during training, leading to issues such as catastrophic forgetting, mode collapse, and training-inference mismatch. To address these challenges, we propose a novel, plug-in curriculum learning framework inspired by the strength training principle of "progressive overload" (POCL), which can be seamlessly integrated into existing white-box KD approaches with minimal computational overhead. The framework comprises two core components: (1) a difficulty measurer that ranks and partitions training samples from easy to hard, and (2) a training scheduler that incrementally introduces these subsets into the distillation process at fixed intervals while applying loss functions with progressively rising temperatures. By starting with the easiest samples and progressively increasing the difficulty, the approach enhances both the stability and efficiency of learning. Extensive experiments in instruction-following settings demonstrate that POCL consistently improves the performance of distilled student models across various white-box KD methods and model families. Our findings highlight the effectiveness of sorted training samples in KD for LLMs. More generally, our work demonstrates how to structure training data within the KD process to enhance the stability and performance of distilled LLMs.

Figures

Figures reproduced from arXiv: 2506.05695 by the authors.

Figure 1
Figure 1. Effectiveness of the POCL framework across KD methods on DollyEval. This figure compares off-policy (KLD [1], RKL, JSD, TVD [2], SKL and SRKL [3]) and on-policy (GKD [4]) KD methods with and without POCL, evaluated using ROUGE-L scores. The setup uses GPT-2-1.5B → GPT-2-0.1B. POCL improves all base white-box KD methods. *Corresponding author. †The code for our method is publicly available at https://github.com/liuli… view at source ↗
Figure 2
Figure 2. Progressive Overload-Based Curriculum Learning (POCL) Framework for KD. Drawing on principles from strength training, POCL models the teacher-student KD process as a coach-athlete training regimen. The training set’s size and difficulty correspond to training volume and intensity, respectively. The student model begins with small, simple data subsets and progressively advances through increasingly complex and larger… view at source ↗
Figure 3
Figure 3. ROUGE-L scores for KD vs. KD+POCL on dolly validation set [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation study of key POCL components—training sample ranking, distillation temperature, and STF ratio—on the performance of the distilled student model across different KD methods. In the figures: (a) ’E2H’ (’easy-to-hard’) and ’H2E’ (’hard-to-easy’) indicate the orde…
Figure 5
Figure 5. Figure 5: Prompt template used for training and evaluation in instruction-following experiments, adapted from [24] Below, we provide descriptions of the datasets used for training and evaluation: • databricks-dolly-15K [35]: An open-source dataset containing 15,000 human-written…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages

  1. [1]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

  2. [41]

    Curriculum tem- perature for knowledge distillation

    Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum tem- perature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504–1512, 2023

    work page 2023

  3. [2]

    F-divergence minimization for sequence-level knowledge distillation

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. F-divergence minimization for sequence-level knowledge distillation. arXiv preprint arXiv:2307.15190, 2023

    arXiv 2023

  4. [3]

    Distillm: Towards streamlined distillation for large language models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898, 2024

    arXiv 2024

  5. [4]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024

    2024

  6. [5]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  7. [6]

    The costly dilemma: generalization, evaluation and cost-optimal deployment of large language models

    Abi Aryan, Aakash Kumar Nain, Andrew McMahon, Lucas Augusto Meyer, and Harpreet Singh Sahota. The costly dilemma: generalization, evaluation and cost-optimal deployment of large language models. arXiv preprint arXiv:2308.08061, 2023

    arXiv 2023

  8. [7]

    Pre-trained language models for text generation: A survey

    Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1–39, 2024

    work page 2024

Show all 45 references
  1. [8]

    Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum

    Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang, Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, vo...

    2024

  2. [9]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/ blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ , September 2024. Accessed: 2025-04- 21 (current date from system)

    2024

  3. [10]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  4. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  5. [12]

    A survey on knowledge distillation of large language models

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024

    2024 arXiv

  6. [13]

    Survey on knowledge distillation for large language models: methods, evaluation, and application

    Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology, 2024

    2024

  7. [14]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016

    2016

  8. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 9

    2024 arXiv

  9. [16]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , 2024. Published: 21 Jun 2024

    2024

  10. [17]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  11. [18]

    Learning to retrieve in-context examples for large language models

    Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164, 2023

    2023 arXiv

  12. [19]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2...

    2023 arXiv

  13. [20]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  14. [21]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  15. [22]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  16. [23]

    The ai index 2025 annual report

    Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, Tobi Walsh, Armin Hamrah...

    2025

  17. [24]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

    2023 arXiv

  18. [25]

    Autoregressive knowledge distillation through imitation learning

    Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. arXiv preprint arXiv:2009.07253, 2020

    2009 arXiv

  19. [26]

    Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling

    Wenda Xu, Rujun Han, Zifeng Wang, Long T Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. arXiv preprint arXiv:2410.11325, 2024

    arXiv

  20. [27]

    A survey on curriculum learning

    Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

    2021

  21. [28]

    Science and practice of strength training

    Vladimir M Zatsiorsky, William J Kraemer, and Andrew C Fry. Science and practice of strength training. Human kinetics, 2020

    2020

  22. [29]

    Dual-space knowledge distillation for large language models

    Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. Dual-space knowledge distillation for large language models. arXiv preprint arXiv:2406.17328, 2024

    2024 arXiv

  23. [30]

    Revisiting catastrophic forgetting in large language model tuning

    Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. arXiv preprint arXiv:2406.04836, 2024

    2024 arXiv

  24. [31]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004

    2004

  25. [32]

    Reciprocal rank fusion outperforms condorcet and individual rank learning methods

    Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759, 2009

    2009

  26. [33]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  27. [34]

    A comparison of most-to-least and least-to-most prompting on the acquisition of solitary play skills

    Myrna E Libby, Julie S Weiss, Stacie Bancroft, and William H Ahearn. A comparison of most-to-least and least-to-most prompting on the acquisition of solitary play skills. Behavior analysis in practice, 1:37–43, 2008

    2008

  28. [35]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  29. [36]

    Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Ha- jishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022. 10

    2022 arXiv

  30. [37]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023...

    2023

  31. [38]

    Benchmarking generalization via in-context instructions on 1,600+ language tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arX...

    2022 arXiv

  32. [39]

    Unnatural instructions: Tuning language models with (almost) no human labor

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022

    2022 arXiv

  33. [40]

    Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification

    Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 247–263. Springer, 2020

    2020

  34. [42]

    Distillm-2: A contrastive approach boosts the distillation of llms

    Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. Distillm-2: A contrastive approach boosts the distillation of llms. arXiv preprint arXiv:2503.07067, 2025

    2025 arXiv

  35. [43]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  36. [44]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  37. [45]

    Qwen3: The latest large language model series from alibaba cloud, 2025

    Qwen Team. Qwen3: The latest large language model series from alibaba cloud, 2025. Available at https: //github.com/QwenLM/Qwen3. A Technical Appendices A.1 Preliminary Formulation of White-box KD We present the background and preliminary formulation of white-box KD for auto-r...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.