Pith. sign in

REVIEW 3 major objections 4 minor 58 references

Self-Evolution Knowledge Distillation for LLM-based Machine Translation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that token-adaptive distillation—mixing teacher and ground-truth probabilities into the student's own distribution for hard tokens—adds about 1.4 SacreBLEU on WMT22 over Forward KD.

desk verdict Empirically promising but the mechanism story is contradicted by the paper's own loss equations. read the letter →

arxiv 2412.15303 v1 pith:6KXVUWWZ submitted 2024-12-19 cs.CL

Yuncheng Song , Liang Ding , Changtong Zan , Shujian Huang This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationmachinetranslationlargelanguagemodelstoken-leveldifficultyself-evolutionlearningSacreBLEUWMT22distributionsmoothing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard knowledge distillation for translation models treats every token the same, even though tokens differ in how hard they are to learn. It proposes Self-Evolution KD, which first uses the KL divergence between a blended target distribution (teacher plus ground truth) and the student's distribution to label each token hard or easy, then for hard tokens trains the student against a smoothed proxy distribution that already contains part of the student's own output. Across En↔De and En↔Cs on WMT22 test sets, this raises SacreBLEU by an average of about 1.4 points over Forward KD, roughly closing the gap to the teacher. The authors interpret the gain as better transfer of teacher knowledge, and show the dynamic token selection beats both treating all tokens as easy and treating all as hard.

What carries the argument

The central mechanism is token-level 'learning difficulty' $d_i$, computed as the KL divergence between the mixed target distribution $\tilde{y}_i = (1-\lambda)y_i + \lambda p_i$ and the student distribution $q_i$, together with the proxy-distribution smoothing $\hat{q}_i = \beta q_i + (1-\beta)\tilde{y}_i$ applied only when $d_i > \Gamma$. This converts the teacher's soft labels and the ground truth into a token-specific prior that is folded into the student's own distribution, so hard tokens are trained against a distribution partway between student and target while easy tokens are left alone.

What would settle it

A reader could test whether the gains vanish when hard-token selection is replaced by random selection of the same number of tokens, or whether the improvement disappears when $\Gamma$ is annealed from 0.4 to 0 as training progresses; if random or annealed variants match Self-Evolution KD, the claimed dynamic-difficulty mechanism is not the cause. Concretely, run the same WMT22 En→De setup with a per-token 'hard' label assigned by a fixed pretrained frequency list instead of the student's KL divergence; if BLEU is unchanged, the self-question stage is not doing the work.

Watch

Extended reading notes

Core claim

Self-Evolution KD claims that adaptive token-level distillation outperforms uniform distillation for LLM-based machine translation. The method mixes the teacher distribution and the one-hot ground-truth distribution into a target distribution $\tilde{y}_i$, measures each token's learning difficulty as $d_i = \operatorname{KL}(\tilde{y}_i \| q_i)$, and, when $d_i$ exceeds a threshold $\Gamma$, replaces the student distribution $q_i$ in the KL loss with a softened proxy $\hat{q}_i = \beta q_i + (1-\beta)\tilde{y}_i$, so the student learns the target from a distribution already close to it. Easy tokens keep the ordinary loss $\operatorname{KL}(\tilde{y}_i \| q_i)$. With $\Gamma = 0.4$ and $\beta = 0.5$, the method reports an average $+1.44$ SacreBLEU and $+0.28$ COMET over Forward KD on WMT22 for Llama-13B→Llama-7B, and $+1.7$ SacreBLEU when the teacher is Llama-30B; it also reports higher similarity between student output and teacher output, which the authors use to support the claim that the gain comes from better knowledge transfer.

Load-bearing premise

The load-bearing premise is that the KL divergence between the mixed target distribution (teacher plus one-hot ground truth) and the student distribution correctly identifies the tokens that will benefit from proxy-distribution smoothing, using a fixed threshold $\Gamma = 0.4$ that never changes during training.

Editorial extensions

If this is right

  • Average SacreBLEU gains of about 1.44 points over Forward KD across En→De, De→En, En→Cs, and Cs→En on the WMT22 test sets, with the largest gain (2.33 BLEU) on Cs→En.
  • Self-Evolution KD approximately matches or exceeds the Llama-13B teacher on BLEU, and outperforms Reverse KD, NoEvo KD, and SKEW KD, which treat tokens uniformly.
  • A dynamic threshold-based token selection ($\Gamma$) beats a fixed top-K selection, supporting the claim that the number of hard tokens should shrink as training proceeds.
  • Raising the teacher from 13B to 30B parameters increases the gain over Forward KD to about 1.7 SacreBLEU, suggesting the method scales with the teacher-student capacity gap.
  • Student generations become more similar to teacher generations (average +2.8 SacreBLEU against teacher text), which the authors read as evidence of better knowledge transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond translation, the same student-smoothed target with a difficulty gate could apply to other autoregressive sequence tasks with imbalanced token difficulty, such as summarization or code generation; the paper does not test those settings.
  • Because $\Gamma$ and $\beta$ are fixed, a testable extension is to anneal $\Gamma$ over training or predict a per-token $\beta$ with a small network, an idea the paper's limitation section explicitly flags as more elegant.
  • The threshold on raw KL divergence may conflate lexical rarity with semantic difficulty; a frequency-aware or position-aware difficulty measure could sharpen which tokens are treated as hard.
  • The gains are measured on a small, high-quality parallel dataset; testing the same mechanism under data-scarce conditions, where distillation matters most, would clarify whether the prior-knowledge mixing is the active ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Self-Evolution KD, a white-box knowledge-distillation method for instruction-tuned LLM machine translation. The method computes a token-level learning difficulty as the KL divergence between the student distribution and a mixed target (ground-truth one-hot plus teacher soft distribution), classifies tokens as easy or hard with a fixed threshold Γ, and for hard tokens minimizes KL(ỹ || β q + (1−β)ỹ) instead of the easy-token objective KL(ỹ || q). Experiments on WMT22 En↔De and En↔Cs with a Llama-13B teacher and a Llama-7B student report an average +1.44 SacreBLEU over Forward KD, plus results with a 30B teacher, ablations of Γ, β, and token-selection strategies, and a teacher-similarity analysis.

Significance. If the empirical gains hold, the paper addresses a relevant practical question: how to make white-box KD token-aware for LLM-based translation. The strongest evidence is the controlled comparison within the no-SFT mixed-target family: Self-Evolution KD outperforms NoEvo KD by about 1.30 BLEU and SKEW KD by about 0.54 BLEU on average in Table 1, supporting the value of dynamic token selection. The use of WMT22 as an external test set, the validation-based selection of Γ and β, and the larger-teacher experiment in Table 2 are also useful. However, the central mechanism claim—that the method injects prior knowledge to prioritize hard tokens—is contradicted by the mathematics of Eq. (7), and the headline comparison to Forward KD is not controlled because it also changes the target distribution and removes the SFT loss. The manuscript needs substantial revision before its central claims can be accepted.

major comments (3)
  1. [§3.2, Eq. (7)] The hard-token loss does not implement the claimed prioritization of hard tokens. Since KL(ỹ || ·) is convex in its second argument, L^h_i = KL(ỹ_i || β q_i + (1−β)ỹ_i) ≤ β KL(ỹ_i || q_i) + (1−β) KL(ỹ_i || ỹ_i) = β d_i. With the default β=0.5 and Γ=0.4, a token just above the threshold has loss at most 0.205, while a token just below the threshold has loss 0.39. The method therefore systematically down-weights the very tokens it labels hard, rather than giving them a stronger learning signal. This contradicts the abstract's and §5.3.1's claim that the gains come from 'better knowledge transfer from teachers'; the teacher-similarity improvement in Figure 4 is also what one would expect from down-weighting tokens where teacher and ground truth disagree. Please provide effective per-token gradient-norm or loss analyses showing that hard tokens actually receive a stronger signal, or revise the mechanism claim and reinterpret the method as an implicit regularizer.
  2. [§5.1, Table 1; §4.3] The dagger markers in Table 1 assert a statistically significant difference (p<0.05), but the paper never describes the significance test, the resampling procedure, the number of runs, or the confidence intervals. Since the headline claim is a set of BLEU deltas between 0.85 and 2.33 points, the authors should either report the paired test used (e.g., bootstrap or approximate randomization over sentences) and any multi-seed variance, or remove the significance claim. Without this information, the reader cannot distinguish real gains from run-to-run or test-set noise, especially for COMET differences as small as 0.11.
  3. [§3.1–3.2 and §5.1] The reported +1.44 average gain over Forward KD is not a controlled comparison. Forward KD in Eq. (3) uses the teacher-only target p and includes the SFT loss, whereas Self-Evolution KD in Eq. (9) uses the mixed target ỹ=(1−λ)y+λp and drops the SFT loss. The NoEvo and SKEW ablations share the no-SFT mixed-target loss family and do isolate the token-selection effect, but the paper's central comparison to Forward KD cannot be attributed to the adaptive token mechanism. Please add an ablation that applies the hard/easy token selection on top of the standard Forward KD loss, or separately report the effects of replacing p by ỹ and of removing the SFT term.
minor comments (4)
  1. [§4.2] The validation set is described as 'WMT21 En→De and Cs→En test data', but the experiments evaluate four translation directions; clarify which validation data were used for De→En and En→Cs, since this affects checkpoint and hyperparameter selection.
  2. [§4.3] The text says all models use beam search but sets the beam size to 1; this is greedy decoding, and the terminology should be corrected.
  3. [Fig. 2(a), 2(c)] State explicitly how many values of Γ and β were searched and whether these figures use the same WMT21 validation split as the checkpoint selection; the current text leaves the search range and axis-label details ambiguous.
  4. [§5.3.2, Eq. (10), Fig. 5] The comparison in Figure 5 varies λ for the SKEW KD (teacher) variant but shows only a single point for Self-Evolution KD; specify whether that point is the Table 1 average and add the corresponding λ sweep if the claim is that no λ value closes the gap.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported WMT22 gains are external-benchmark measurements, and the loss equations do not encode the result.

full rationale

The paper's derivation chain is empirical rather than definitional. The token-difficulty score d_i = KL(ỹ_i || q_i) (Eq. 5) and the hard-token loss L^h_kl = KL(ỹ_i || β q_i + (1−β)ỹ_i) (Eqs. 6–7) define the training objective, but the paper's central claim—an average +1.44 SacreBLEU improvement on WMT22—is measured on a held-out benchmark, not computed from those equations. The key hyperparameters Γ and β are selected on the WMT21 validation set (Figs. 2(a) and 2(c)) and then applied to the WMT22 test sets, so the test numbers are not fitted inputs renamed as predictions. The paper's self-citations to prior self-evolution work (e.g., Peng et al., 2023b; Zhong et al., 2023b) are used for motivation and for the convention of a fixed threshold, but the controlled comparisons against Forward KD, Reverse KD, NoEvo, and SKEW baselines provide independent evidence for the improvement claim; the argument does not reduce to the self-citations. The Limitation section's concession that Γ is an empirical preset value is a hyperparameter-tuning concern, not circularity. The skeptic's convexity observation (that L^h ≤ β KL(ỹ||q), so hard tokens receive a scaled-down loss) is a potential mismatch between the stated mechanism and the implemented objective, but it does not make the WMT22 result equivalent to the loss by construction, so it is a correctness or interpretation concern rather than a circular step. No step in the paper's chain reduces a claimed prediction to its own input. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method depends on three hyperparameters (λ, β, Γ), one of which (Γ) is explicitly acknowledged as empirical. No new entities are introduced; the 'prior knowledge' is a linear mixture of existing distributions.

free parameters (3)
  • lambda (mixing ratio in target) = 0.5
    Eq. 4 combines teacher distribution and one-hot ground truth at equal weight. Not ablated.
  • beta (proxy mixing ratio) = 0.5
    Eq. 6 blends student and target distributions; tuned on WMT21 validation (Fig. 2c).
  • Gamma (hard-token threshold) = 0.4
    Stage 1 classifies tokens as hard if KL divergence exceeds Γ; tuned on WMT21 validation (Fig. 2a) and fixed throughout training.
assumptions (3)
  • domain assumption The mixed target ỹ=(1-λ)y+λp is an appropriate teaching signal
    Eq. 4 defines the target; if this mixing is not appropriate, the difficulty measure and losses are distorted.
  • domain assumption KL divergence d_i=KL(ỹ_i||q_i) measures per-token learning difficulty
    Stage 1 uses Eq. 5 to classify tokens; presumes student distributions are calibrated enough for the divergence to be meaningful.
  • ad hoc to paper A single fixed Γ is valid for all tokens and training steps
    The paper sets Γ=0.4 and, in its Limitation, says a dynamic threshold would be more elegant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Evolution Knowledge Distillation for LLM-based Machine Translation." pith.science (2026). https://pith.science/paper/6KXVUWWZ

@misc{pith2026241215303,
  author       = {Pith},
  title        = {Pith review of: Self-Evolution Knowledge Distillation for LLM-based Machine Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KXVUWWZ}},
  note         = {Machine review of arXiv:2412.15303}
}
read the original abstract

Knowledge distillation (KD) has shown great promise in transferring knowledge from larger teacher models to smaller student models. However, existing KD strategies for large language models often minimize output distributions between student and teacher models indiscriminately for each token. This overlooks the imbalanced nature of tokens and their varying transfer difficulties. In response, we propose a distillation strategy called Self-Evolution KD. The core of this approach involves dynamically integrating teacher distribution and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process. It adjusts the ratio of prior knowledge based on token learning difficulty, fully leveraging the teacher model's potential. Experimental results show our method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets. Further analysis indicates that the improvement comes from better knowledge transfer from teachers, confirming our hypothesis.

Figures

Figures reproduced from arXiv: 2412.15303 by the authors.

Figure 1
Figure 1. Overall framework of our Self-Evolution KD. It mainly contains two stages: ① self-question: calculating the learning difficulty by the KL divergence between the student distribution and target distribution, and dividing tokens into different categories. comparison means comparing the learning difficulty with the preset threshold Γ; ② self-evolution: building proxy distribution for different tokens by smoothing the t… view at source ↗
Figure 2
Figure 2. 2(a) and 2(b): Effect of Γ and percent (K) for selecting hard-to-learn tokens. 2(c): Effect of β to determine the mixsture proportion of prior knowledge. We report their average SacreBLEU points on the above￾mentioned validation dataset in 2(a) and 2(c). As for 2(b), the average SacreBLEU points on WMT22 test sets are reported since we compare different distillation strategies. thus we retain it as our default setti… view at source ↗
Figure 5
Figure 5. Effect of the loss weight of the SKEW KD (Teacher). We only report the Self-Evolution KD for reference. only considering the teacher knowledge as prior knowledge would significantly reduce the KL di￾vergence loss and mislead the student model to emphasize the SFT loss, thus potentially curtailing the benefits derivable from teacher knowledge. Fol￾lowing Ko et al. (2024), we redefine the distillation objective as (SK… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of teacher’s knowledge trans￾fer across different distillation strategies. fectiveness of knowledge transfer across various distillation strategies in this part. We regard the generation text of teacher model on the WMT22 En↔De, En↔Cs test sets as the “refer…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations

    2024

  3. [3]

    Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ond r ej Bojar, Rajen Chatterjee, et al. 2021. Findings of the 2021 conference on machine translation (wmt21). In Proceedings of the sixth conference on machine translation

    work page 2021

  4. [4]

    Hongzhan Chen, Xiaojun Quan, Hehong Chen, Ming Yan, and Ji Zhang. 2024. Knowledge distillation for closed-source language models. arXiv preprint arXiv:2401.07013

    arXiv 2024

  5. [5]

    Kehai Chen, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2020. Content word aware neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    work page 2020

  6. [6]

    Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806

    arXiv 2017

  7. [7]

    Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, pages 10867--10878. PMLR

    2023

  8. [8]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations

    2024

Show all 58 references
  1. [9]

    Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and Andr \'e FT Martins. 2023. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics

    2023

  2. [10]

    Junliang Guo, Zhirui Zhang, Linli Xu, Boxing Chen, and Enhong Chen. 2021. Adaptive adapters: An efficient way to incorporate bert into neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing

    2021

  3. [11]

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210

    2023 arXiv

  4. [12]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  5. [13]

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint a...

    2023 arXiv

  6. [14]

    Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023 a . Parrot: Translating during chat using large language models tuned with human translation and feedback. In Findings of the Association for Computational Linguist...

    2023

  7. [15]

    Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023 b . Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 1(10)

    2023 arXiv

  8. [16]

    Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. arXiv preprint arXiv:2304.03245

    2023 arXiv

  9. [17]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT. Minneapolis, Minnesota

    2019

  10. [18]

    Gyeongman Kim, Doohyuk Jang, and Eunho Yang. 2024. Promptkd: Distilling student-friendly knowledge for generative language models via prompt tuning. arXiv preprint arXiv:2402.12842

    2024 arXiv

  11. [19]

    Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947

    2016 arXiv

  12. [20]

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898

    2024 arXiv

  13. [21]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, et al. 2022. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT)

    2022

  14. [22]

    Jan Koco \'n , Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szyd o, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. Chatgpt: Jack of all trades, master of none. Information Fusion

    2023

  15. [23]

    Mike Lewis, Yinhan Liu, Naman Goyal, et al. 2020. BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    2020

  16. [24]

    Xuebo Liu, Longyue Wang, Derek F Wong, Liang Ding, Lidia S Chao, Shuming Shi, and Zhaopeng Tu. 2021. On the complementarity between pre-training and back-translation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021

    2021

  17. [25]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics

    2020

  18. [26]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. Error analysis prompting enables human-like translation evaluation in large language models. In Findings of the Association for Computational Linguistics ACL 2024

    2024

  19. [27]

    Shilong Pan, Zhiliang Tian, Liang Ding, Haoqi Zheng, Zhen Huang, Zhihua Wen, and Dongsheng Li. 2024. POMP : Probability-driven meta-graph prompter for LLM s in low-resource unsupervised neural machine translation. In Proceedings of the 62nd Annual Meeting of the Association fo...

    2024

  20. [28]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023 a . Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  21. [29]

    Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, and Dacheng Tao. 2023 b . Token-level self-evolution training for sequence-to-sequence learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2...

    2023

  22. [30]

    Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023 c . Towards making the most of chatgpt for machine translation. In Findings of EMNLP

    2023

  23. [31]

    Steven T Piantadosi. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review

    2014

  24. [32]

    Matt Post. 2018. A call for clarity in reporting bleu scores. WMT 2018

    2018

  25. [33]

    Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2020. Glancing transformer for non-autoregressive neural machine translation. arXiv preprint arXiv:2008.07905

    2020 arXiv

  26. [34]

    Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. 2022. Better teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206.06067

    2022 arXiv

  27. [35]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research

    2020

  28. [36]

    Jun Rao, Xuebo Liu, Zepeng Lin, Liang Ding, Jing Li, Dacheng Tao, and Min Zhang. 2024. Exploring and enhancing the transfer of distribution in knowledge distillation for autoregressive language models. arXiv preprint arXiv:2409.12512

    2024 arXiv

  29. [37]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2020

  30. [38]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  31. [39]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint

    2023

  32. [40]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  33. [41]

    Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. arXiv preprint arXiv:2105.12967

    2021 arXiv

  34. [42]

    Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. arXiv preprint arXiv:2304.02210

    2023 arXiv

  35. [43]

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657

    2024 arXiv

  36. [44]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674

    2023 arXiv

  37. [45]

    Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022 a . Bridging cross-lingual gaps during leveraging the multilingual sequence-to-sequence pretraining for text generation and understanding. arXiv preprint arXiv:2204.07834

    2022 arXiv

  38. [46]

    Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022 b . On the complementarity between pre-training and random-initialization for resource-rich machine translation. In Proceedings of the 29th International Conference on Computational Linguistics

    2022

  39. [47]

    Changtong Zan, Liang Ding, Li Shen, Yibing Zhen, Weifeng Liu, and Dacheng Tao. 2024. Building accurate translation-tailored llms with language aware instruction tuning. arXiv preprint arXiv:2403.14399

    2024 arXiv

  40. [48]

    Songming Zhang, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jian Liu, and Jie Zhou. 2022. Conditional bilingual mutual information based adaptive training for neural machine translation. arXiv preprint arXiv:2203.02951

    2022 arXiv

  41. [49]

    Haoqi Zheng, Qihuang Zhong, Liang Ding, Zhiliang Tian, Xin Niu, Dongsheng Li, and Dacheng Tao. 2023. Self-evolution learning for mixup: Enhance data augmentation on few-shot text classification tasks. arXiv preprint arXiv:2305.13547

    2023 arXiv

  42. [50]

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023 a . Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint

    2023

  43. [51]

    Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023 b . Self-evolution learning for discriminative language model pretraining. arXiv preprint arXiv:2305.15275

    2023 arXiv

  44. [52]

    Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. 2024. Revisiting knowledge distillation for autoregressive language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    2024

  45. [53]

    Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, et al. 2022. Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on superglue. arXiv preprint arXiv:2212.01853

    2022 arXiv

  46. [54]

    Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. 2020. Incorporating bert into neural machine translation. In International Conference on Learning Representations

    2020

  47. [55]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023 a . Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675

    2023 arXiv

  48. [56]

    Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023 b . Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948

    2023 arXiv

  49. [57]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.