Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that a two-stage distillation recipe—multi-agent black-box data augmentation followed by top-10-logit white-box fusion—makes smaller Qwen2.5 models beat their original instruct checkpoints across AlpacaEval 2.0…

desk verdict A practical industrial KD recipe with plausible gains and a useful top-10 logit trick, but the superiority-over-checkpoint claim needs robustness and decontamination checks before it can carry the paper. read the letter →

arxiv 2504.15027 v1 pith:ZH6LT6B2 submitted 2025-04-21 cs.CL

Chengyu Wang , Junbing Yan , Yuanhao Yue , Jun Huang This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationinstructionfollowinglargelanguagemodelsmodelcompressionmulti-agentdataaugmentationlogitfusionlightweightLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims a practical recipe for turning a family of large language models into smaller ones that follow instructions better than the checkpoints they were cut from. It argues that combining two kinds of knowledge distillation—multi-agent data augmentation with black-box teachers, then efficient white-box fusion using only the top ten token logits—produces students that outperform their original sizes on AlpacaEval 2.0, MT-Bench, and IFEval. The reported gains are largest for the smallest students, which matters because tiny models are the cheapest to deploy. If the recipe holds, distillation becomes a reliable upgrade for lightweight LLM deployment rather than a trade-off that accepts lower quality.

What carries the argument

The load-bearing object is the top-10 logit fusion used in the white-box stage. The authors observe that the sum of the top-10 token probabilities in the teacher's distribution is almost 1, so they keep only the top ten logits, align student and teacher vocabulary indices, and minimize a divergence such as $\mathrm{KL}$ over those ten entries after a temperature-scaled softmax. This reduces memory, storage, and reading time for teacher logits while preserving nearly all of the teacher's signal. The other half of the pipeline is the multi-agent controller: expansion, rewriting, selection, and verification agents that turn public instruction data into a larger, cleaner, CoT-rich set for black-box distillation. Together the two mechanisms let students inherit both task-level knowledge through data and token-level knowledge through logits.

What would settle it

Run an embedding- or n-gram-based overlap check between the augmented training set, including the in-house portions, and the prompts of AlpacaEval 2.0, MT-Bench, and IFEval; a high overlap rate would show the gain comes from memorization. Independently, repeat the evaluation of one distilled model, say the 1.5B version, five or more times with different seeds and sample orders; if the gap over the original checkpoint falls inside the run-to-run spread, the claimed improvement is not stable.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a distilled lightweight model can be stronger at instruction following than the original checkpoint it was derived from. Across 0.5B, 1.5B, 3B, and 7B sizes, the DistilQwen2.5 variants beat their Qwen2.5-Instruct counterparts on every reported metric, with the largest relative jumps at the smallest sizes (e.g., AlpacaEval 2.0 LC rises from 2.46 to 4.89 for the 0.5B model and from 6.69 to 13.69 for the 1.5B model). The two stages contribute separately: black-box KD supplies a higher-quality, CoT-reasoned training set, and white-box fusion adds further gains by matching top-10 token distributions. The authors also report that this white-box stage runs 3–5 times faster than the vanilla approach, and that a 3B student can nearly match a 7B teacher on SQL completion at 1.4× speed.

Load-bearing premise

The largest gain rests on the assumption that the three evaluation benchmarks are clean and that the teacher-generated training data does not overlap them; if contamination or run-to-run variance explains the gap, the superiority claim is overstated.

Editorial extensions

If this is right

  • Distillation can be used as a standard upgrade step: same-size students beat the original instruct checkpoints on all three reported benchmarks, so deployment teams do not have to sacrifice instruction-following quality when shrinking models.
  • The smallest students gain the most, which makes sub-1B models a viable target for edge and low-latency deployment rather than a last resort.
  • Top-10 logit fusion is efficient enough for industrial-scale white-box KD, since teacher logits can be precomputed offline with small storage and vocabulary mismatch handled by token alignment.
  • In a real SQL-completion task, a 3B student nearly matched a 7B teacher's adoption rate while cutting latency 1.4×, suggesting the pipeline transfers beyond generic instruction following.
  • White-box fusion adds measurable gains on top of black-box KD, so the two stages are complementary rather than redundant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If top-10 concentration of token probability mass is common across large language models, top-K sparsification could also accelerate other logit-hungry applications such as ranking, reranking, and selective prediction, not just distillation.
  • The paper's observation that gains diminish when the teacher grows from 14B to 72B suggests a testable rule for budgeting: distilling from a medium-sized teacher may be the cost-effective choice for most student sizes, which would change how practitioners allocate teacher API calls.
  • The agent-based data augmentation is task-category-preserving, so the same pipeline could be reused as a data-engine service for domains beyond chat, such as summarization or tool-use instruction sets, by swapping the seed instructions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper describes the training and evaluation of DistilQwen2.5, a family of lightweight instruction-following models distilled from Qwen2.5. The pipeline has two stages: (i) black-box distillation via multi-agent data augmentation, where proprietary LLMs (Qwen-max, GPT-4/GPT-4o) act as agents to expand, rewrite, select, and verify instruction-response pairs; and (ii) efficient white-box distillation, where the teacher's logits are precomputed offline with top-10 truncation and the student minimizes a divergence over this truncated distribution after standard fine-tuning. The authors evaluate the 0.5B, 1.5B, 3B, and 7B models on AlpacaEval 2.0 (LC), MT-Bench, and IFEval, reporting that all distilled models outperform the original checkpoints, with the largest gains for the smallest models. Additional experiments analyze task-level MT-Bench scores, teacher-size scaling, and dataset-size scaling, and two industrial use cases (SQL completion and a cloud KD platform) are described.

Significance. The paper proposes a practical distillation pipeline combining black-box multi-agent data augmentation with efficient white-box top-10 logit fusion, and reports consistent improvements over the Qwen2.5 instruct checkpoints across four model sizes on three benchmarks. Strengths of the work include the release of the models and processed dataset, a clear ablation isolating the white-box fusion step (asterisk vs non-asterisk rows in Table 1), a measured 3-5x speedup for teacher logit generation, and a consistent trend of larger gains for smaller students. If the evaluation concerns are resolved, the industrial recipe is a useful contribution. However, the current evaluation lacks contamination control, variance estimation, and comparison with existing KD objectives, which limits the strength of the central claim.

major comments (4)
  1. [Section 4.3 / Table 1] The central claim of the paper (Abstract, Section 4.3) is that the distilled models are 'significantly stronger' than their original checkpoints, and the evidence is Table 1. However, the training corpus described in Section 4.1 includes OpenHermes 2.5 and Cleaned Alpaca, public instruction-tuning datasets whose prompts are likely to overlap with the evaluation benchmarks AlpacaEval 2.0 and MT-Bench. The paper reports no deduplication of training data against evaluation prompts, no contamination analysis, and no held-out evaluation set. Since the evaluators include GPT-4 (AlpacaEval 2.0 and MT-Bench use GPT-4 as judge) and one of the teacher models is GPT-4, a portion of the reported gain could reflect style imitation or format overfitting rather than general instruction-following improvement. The IFEval strict-prompt gain for the 1.5B model (40.11 to 73.02, while instruct-loose moves only 55.40 to 60.63) is a pattern consistent with benchmark-specific format overfitting. Please add a systematic overlap analysis (e.g., longest common substring or n-gram matching between training prompts and benchmark prompts) and, if possible, evaluate on a held-out set of instructions that are provably absent from the training data.
  2. [Table 1] Table 1 reports a single run per configuration. The differences between DistilQwen2.5 and the original checkpoints are often small on MT-Bench (e.g., 0.5B: 5.49 vs 5.71, a 0.22-point difference), and MT-Bench is known to be sensitive to decoding parameters and seed. Without repeated runs or bootstrap confidence intervals, the statistical significance of the improvements is unclear. Please report results over multiple seeds (or provide bootstrap intervals) for at least the central configurations (e.g., 1.5B and 7B), with the decoding hyperparameters fixed.
  3. [Section 2 / Section 4.3] Section 2 surveys existing white-box KD methods (MiniLLM, reverse-KL, FuseLLM), but the experimental section does not compare the proposed top-10 model fusion against any of these methods on the same backbone. Consequently, the contribution of the 'efficient model fusion' component over standard KD objectives is not demonstrated. Add a same-backbone comparison on at least one student size (e.g., 1.5B) between the proposed method and a standard KD baseline (e.g., MiniLLM's objective or a forward-KL baseline) under identical data and compute.
  4. [Section 3.2] Section 3.2: The assertion that 'the sum of the probabilities of the top-10 tokens is almost equal to 1' is a key justification for truncating teacher logits to K=10, but no empirical evidence is shown. Provide a plot of cumulative probability mass vs K for representative teacher outputs, and report the downstream performance of the student when trained with full-logit KD versus top-10 KD (the text mentions exploratory experiments but does not present them). This is needed to validate that the truncation is lossless for the distillation objective.
minor comments (6)
  1. [Section 4.4] Section 4.4 is labeled 'Detailed Analysis' but is empty in the manuscript; please either remove the heading or supply the intended analysis.
  2. [Figure 4] Figure 4 appears to list 'DistilQwen2.5-1.5B-Instruct' twice, with scores 13.69 and 20.93; the 20.93 entry is likely the 3B model (Table 1 reports 20.91 for the 3B).
  3. [Table 2] Table 2: For the 1.5B model, the Writing score is unchanged (8.38) and STEM decreases (9.65 to 9.28); the text claims general improvement across tasks, so clarify or correct these entries.
  4. [References] The MT-Bench citation should be to the original MT-Bench paper (Zheng et al., 2023), not to MT-Bench-101 (Bai et al., 2024), which is a different benchmark.
  5. [Equation (3)] In Eq. (3), T denotes both the temperature and the teacher model; please use a distinct symbol such as tau for temperature to avoid ambiguity.
  6. [Limitations] The Limitations section would benefit from a sentence acknowledging that overlap between training data and evaluation benchmarks could affect the reported results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed improvements rest on external benchmark evaluations, and the distillation pipeline's components are not fitted to those benchmarks.

full rationale

The paper's central claim is that DistilQwen2.5 models improve instruction-following over Qwen2.5 checkpoints, supported by AlpacaEval 2.0, MT-Bench, and IFEval results in Table 1. Nothing in the method fits constants to these benchmarks. The black-box KD stage augments public and in-house instruction-response data using teacher agents, and the white-box fusion stage minimizes divergence to teacher top-10 logits; the top-10 truncation is justified by the empirical observation that the sum of top-10 probabilities is nearly 1, not by a benchmark-derived fit. The evaluation metrics are standard external benchmarks, and the training objectives do not include those benchmark scores. The self-citations to Yue et al. (2024a, 2024b) for preprocessing and task balance are prior pipeline details rather than load-bearing justifications of the measured gains, and no uniqueness theorem or definitional equivalence is used to force the reported outcome. Risks such as possible training/evaluation overlap and single-run scoring are experimental-validity concerns, not circularity. The derivation is therefore self-contained against external benchmarks.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard KD premises and a few engineering choices, with K=10 and an undisclosed temperature as the main free parameters. No invented physical or mathematical entities appear. The load-bearing unproven assumptions are the sufficiency of top-10 logits and the validity and contamination-freedom of the evaluation benchmarks.

free parameters (2)
  • K (number of top logits kept) = 10
    Chosen because the sum of top-10 token probabilities is observed to be near 1; this controls the approximation in white-box KD and the claimed efficiency gain.
  • Temperature T in softmax for divergence = not specified
    Used in the probability computation for teacher and student logits in Section 3.2; no value is disclosed, so the white-box distillation recipe is not fully reproducible.
assumptions (5)
  • domain assumption Supervised training on teacher-generated instruction-response pairs transfers instruction-following knowledge to the student.
    Black-box KD in Section 3.1 assumes imitation of teacher outputs is sufficient for the student to acquire the skill; this is a standard assumption of the field, not proved here.
  • domain assumption Top-10 token probabilities contain essentially all teacher knowledge because their sum is close to 1.
    The efficiency claim in Section 3.2 depends on this empirical observation, presented without quantitative support or error analysis.
  • domain assumption Benchmark scores on AlpacaEval 2.0, MT-Bench, and IFEval reflect general instruction-following capability and are not contaminated by the training data.
    The central comparison in Table 1 uses these benchmarks as ground truth; no contamination or stability analysis is provided.
  • domain assumption Token alignment between teacher and student vocabularies preserves the semantic content of logits.
    Invoked in Section 3.2 via FuseLLM's alignment operation; if alignment introduces noise, the white-box KD target is distorted.
  • domain assumption The proprietary teacher models used for selection, rewriting, and verification produce factually correct, unbiased outputs.
    The reliability of the whole augmentation pipeline in Section 3.1 rests on the teachers' quality; the limitations section admits biases may propagate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models." pith.science (2026). https://pith.science/paper/ZH6LT6B2

@misc{pith2026250415027,
  author       = {Pith},
  title        = {Pith review of: DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZH6LT6B2}},
  note         = {Machine review of arXiv:2504.15027}
}
read the original abstract

Enhancing computational efficiency and reducing deployment costs for large language models (LLMs) have become critical challenges in various resource-constrained scenarios. In this work, we present DistilQwen2.5, a family of distilled, lightweight LLMs derived from the public Qwen2.5 models. These distilled models exhibit enhanced instruction-following capabilities compared to the original models based on a series of distillation techniques that incorporate knowledge from much larger LLMs. In our industrial practice, we first leverage powerful proprietary LLMs with varying capacities as multi-agent teachers to select, rewrite, and refine instruction-response pairs that are more suitable for student LLMs to learn. After standard fine-tuning, we further leverage a computationally efficient model fusion approach that enables student models to progressively integrate fine-grained hidden knowledge from their teachers. Experimental evaluations demonstrate that the distilled models possess significantly stronger capabilities than their original checkpoints. Additionally, we present use cases to illustrate the applications of our framework in real-world scenarios. To facilitate practical use, we have released all the DistilQwen2.5 models to the open-source community.

Figures

Figures reproduced from arXiv: 2504.15027 by the authors.

Figure 1
Figure 1. Brief comparison between original Qwen2.5 and DistilQwen2.5 models in terms of AlpacaEval 2.0 (length-controlled) and IFEval scores. Knowledge distillation (KD) is a promising ap￾proach to addressing these challenges by transfer￾ring knowledge from a larger model (the teacher) to a smaller model (the student) (Xu et al., 2024). Previous works have primarily focused on specific KD techniques to develop more robust st… view at source ↗
Figure 2
Figure 2. Functionalities for LLMs/agents used in data [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Comparison between various small models (<10B) based on AlpacaEval 2.0 (length-controlled). (a) Student size: 0.5B (b) Student size: 1.5B [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Comparison between black-box KD and white [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Performance of white-box KD with varying [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Illustration of continual KD pipelines on the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EasyDistill packages established LLM knowledge-distillation techniques into a single modular toolkit with released distilled models, datasets, and Alibaba Cloud integration.

Reference graph

Works this paper leans on

25 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.401 Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues . In Proceedings of the 62nd Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2024.acl-long.401 2024

  2. [2]

    Hashimoto

    Yann Dubois, Bal \' a zs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. https://doi.org/10.48550/ARXIV.2404.04475 Length-controlled alpacaeval: A simple way to debias automatic evaluators . CoRR, abs/2404.04475

    arXiv doi:10.48550/arxiv.2404.04475 2024

  3. [3]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. https://openreview.net/forum?id=5h0qf7IBZZ Minillm: Knowledge distillation of large language models . In The Twelfth International Conference on Learning Representations. OpenReview.net

    2024

  4. [4]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. http://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . CoRR, abs/1503.02531

    arXiv 2015

  5. [5]

    Boyu Hou, Chengyu Wang, Xiaoqing Chen, Minghui Qiu, Liang Feng, and Jun Huang. 2023. https://doi.org/10.1109/ICASSP49357.2023.10095721 Prompt-distiller: Few-shot knowledge distillation for prompt-based language learners with dual contrastive learning . In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 1--5. IEEE

    arXiv 2023

  6. [6]

    Cheng - Yu Hsieh, Chun - Liang Li, Chih - Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen - Yu Lee, and Tomas Pfister. 2023. https://doi.org/10.18653/v1/2023.findings-acl.507 Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes . In Findings of the Association for Computa...

    doi:10.18653/v1/2023.findings-acl.507 2023

  7. [7]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.372 Tinybert: Distilling BERT for natural language understanding . In Findings of the Association for Computational Linguistics: EMNLP 2020 , volume EMNLP 2020 of Findings of ACL , pages 4163--4174. Associat...

    doi:10.18653/v1/2020.findings-emnlp.372 2020

  8. [8]

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. 2024. https://doi.org/10.18653/v1/2024.findings-acl.958 Selective reflection-tuning: Student-selected data recycling for LLM instruction-tuning . In Findings of the Association for Computational Linguistics, ACL 2024 , pages 16189--16211. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-acl.958 2024

Show all 25 references
  1. [9]

    Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. 2024. https://openreview.net/forum?id=1vrS1zwekw MUFFIN: curating multi-faceted instructions for improving instruction following . In The Twelfth International Conference on Learning Repr...

    2024

  2. [10]

    Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. 2021. https://doi.org/10.18653/v1/2021.acl-long.236 Meta-kd: A meta knowledge distillation framework for language model compression across domains . In Proceedings of the 59th Annual Meeting of th...

    2021 doi

  3. [11]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. http://arxiv.org/abs/1910.01108 Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter . CoRR, abs/1910.01108

    2019 arXiv

  4. [12]

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. https://doi.org/10.18653/v1/2020.acl-main.195 Mobilebert: a compact task-agnostic BERT for resource-limited devices . In Proceedings of the 58th Annual Meeting of the Association for Computat...

    2020 doi

  5. [13]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. https://openreview.net/forum?id=jiDsk12qcz Knowledge fusion of large language models . In The Twelfth International Conference on Learning Representations. OpenReview.net

    2024

  6. [14]

    Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. https://doi.org/10.1007/978-3-030-60450-9\_8 A large-scale chinese short-text conversation dataset . In Natural Language Processing and Chinese Computing - 9th CCF International Confe...

    2020 doi

  7. [15]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reasoning...

    2022

  8. [16]

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. https://doi.org/10.18653/v1/2023.acl-long.605 f-divergence minimization for sequence-level knowledge distillation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 10817--108...

    2023 doi

  9. [17]

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. 2025. https://aclanthology.org/2025.coling-main.383/ Rethinking kullback-leibler divergence in knowledge distillation for large language models . In Proceedings of the 31st International Conference o...

    2025

  10. [18]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. https://doi.org/10.48550/ARXIV.2402.13116 A survey on knowledge distillation of large language models . CoRR, abs/2402.13116

    2024 doi arXiv

  11. [19]

    Yuanhao Yue, Chengyu Wang, Jun Huang, and Peng Wang. 2024 a . https://doi.org/10.48550/ARXIV.2412.04871 Building a family of data augmentation models for low-cost LLM fine-tuning on the cloud . CoRR, abs/2412.04871

    2024 doi arXiv

  12. [20]

    Yuanhao Yue, Chengyu Wang, Jun Huang, and Peng Wang. 2024 b . https://aclanthology.org/2024.findings-emnlp.350 Distilling instruction-following abilities of large language models with task-aware curriculum planning . In Findings of the Association for Computational Linguistics...

    2024

  13. [21]

    Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024. https://aclanthology.org/2024.emnlp-main.1010 Dual-space knowledge distillation for large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages...

    2024

  14. [22]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian - Yun Nie, and Ji - Ro...

    2023 doi arXiv

  15. [23]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://doi.org/10.48550/ARXIV.2311.07911 Instruction-following evaluation for large language models . CoRR, abs/2311.07911

    2023 doi arXiv

  16. [24]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.