Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MultiLevelOT claims that optimal transport at both token and sequence levels lets a student language model absorb a teacher's knowledge even when their tokenizers and vocabularies do not match, and reports consistent gains over prior…

desk verdict Useful cross-tokenizer distillation loss with consistent small gains, but the token-level math is underspecified for unequal vocabularies—fix that before accepting. read the letter →

arxiv 2412.14528 v2 pith:K5I7V237 submitted 2024-12-19 cs.CL

Xiao Cui , Mo Zhu , Yulei Qin , Liang Xie , Wengang Zhou , Houqiang Li This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationcross-tokenizeroptimaltransportSinkhorndistancelargelanguagemodelslogitalignmentvocabularymismatchsequence-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation usually requires teacher and student to share a tokenizer, because divergence losses compare vocabulary dimensions one by one. This paper claims that constraint is unnecessary: a loss built on optimal transport can align the two models' output distributions even when their vocabularies and tokenizations differ. The proposed MultiLevelOT computes optimal transport distances at two levels, across vocabulary dimensions within tokens and across tokens within a sequence, and combines an absolute-difference cost, a logarithmic cost, and a Sinkhorn distance cost. On extractive QA, generative QA, and summarization, it reports consistent gains over prior cross-tokenizer methods such as ULD across several student models. If the claim holds, practitioners can distill across model families and architectures without retraining aligners or standardizing tokenizers.

What carries the argument

The machinery is a two-level optimal transport objective. At the token level, the paper reconstructs ULD's token-wise OT as an equivalent sorted elementwise loss and generalizes it by ranking all tokens in the sequence jointly, truncating to the top-k logit dimensions, and using two cost matrices: absolute difference $C^{tok}_{ij}(t)=|t_i(t)-s_j(t)|$ and logarithmic $C^{tok}_{ij}(t)=-t_i(t)\log s_j(t)$. This yields the holistic absolute difference loss $L_{HAD}$ and sequential logarithmic loss $L_{SL}$, after which a single optimal transport plan, diagonal in form, applies to all tokens. At the sequence level, the cost between token rows is fed into a Sinkhorn distance, $P_{\lambda} = \arg\min_P \sum_{ij} P_{ij}C_{ij} - \lambda h(P)$, approximated by iterative Sinkhorn normalization, giving $L_{SD}$; the total loss weights these terms with cross-entropy: $L = \sum_t L_{CE}(y(t),s(t)) + \alpha(L_{HAD}+\beta L_{SL}+\gamma L_{SD})$. The design claim is that diverse cost matrices capture both local logit geometry and global sequence structure while avoiding dimensional or token-by-token correspondence.

What would settle it

Replace the sequence-level ranking and permutation in Eqs. (10)-(11) with a random permutation of the student's vocabulary dimensions and re-run the three tasks; if scores remain close to the reported ones, the alignment Q is not doing the claimed work, and if they collapse, it is. A second check: on a held-out set, look up the actual vocabulary tokens behind the top-k ranked dimensions of the teacher and student after matching and measure how often they are the same word or a translation or paraphrase.

Watch

Extended reading notes

Core claim

The paper's central claim is that cross-tokenizer knowledge distillation can be made universal by replacing dimension-wise or token-wise divergence with multi-level optimal transport on logit distributions. After temperature-scaling both models' logits, MultiLevelOT ranks each model's vocabulary dimensions by their summed logits over the whole sequence, applies a permutation to the student so its dimensions are matched to the teacher's, truncates both to the top k dimensions, and then applies two token-level losses, a holistic absolute difference loss and a sequential logarithmic loss, plus a sequence-level Sinkhorn distance loss that measures the transport cost between the teacher's and student's token rows. The paper claims this consistently outperforms ULD and other baselines across all datasets and student models, cuts the teacher-student performance gap on QED by over 71% relative to ULD, and generalizes across model families, architectures, and scales without extra modules.

Load-bearing premise

The token-level losses assume that after ranking and truncating the vocabulary dimensions, the i-th surviving dimension of the teacher and the i-th surviving dimension of the student are close enough in meaning that elementwise comparison is informative, yet the paper gives no procedure for computing the matching permutation when the vocabularies differ in size.

Editorial extensions

If this is right

  • A single distillation recipe works for teachers and students from different model families, such as LLaMA, Mistral, and Qwen, and for student scales from 160M to 1B, with no extra trainable modules.
  • The method improves on ULD on every dataset and student model reported, including the unlabeled setting where only teacher-generated text is available.
  • Sequence-level transport removes the need for rigid token-by-token alignment, so words split differently by different tokenizers can still be matched.
  • Because the total loss is a weighted sum of cross-entropy with three OT-based terms, it can be dropped into an existing supervised fine-tuning loop with three scalar weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension the paper only gestures at is multi-teacher distillation: because the loss needs no vocabulary correspondence, several teachers with different tokenizers could supervise one student by averaging their OT losses, provided the sequence-level costs are comparable.
  • The sequence-level Sinkhorn cost is built from pairwise token distances, so its memory and time grow with sequence length squared; for long-document or long-context tasks, chunking or a more approximate OT plan would likely be needed.
  • The semantic content of the learned permutation is testable: if the top-k ranked teacher and student dimensions after matching frequently correspond to the same word or morpheme, the elementwise losses are transferring lexical knowledge; if they do not, the reported gains may come from distributional shaping rather than lexical alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MultiLevelOT, a cross-tokenizer knowledge distillation loss for LLMs with different tokenizers. The method combines token-level losses (a holistic absolute difference loss and a sequential logarithmic loss, both applied after sequence-level ranking and top-k truncation) with a sequence-level Sinkhorn distance loss. Experiments on extractive QA (QED), generative QA (FairytaleQA), and summarization (DIALOGSum) use LLaMA2-7B as the teacher and OPT, Pythia, Bloomz, and mT0 students; the reported tables show consistent gains over SFT, SeqKD, MinED, and ULD, together with ablations and hyperparameter sensitivity studies.

Significance. If the method is correctly specified, it is a practically useful contribution: a logit-only, module-free CTKD loss that claims to work across vocabulary sizes, model families, and architectures, with released code. The paper's strengths are the breadth of experiments (multiple tasks, students, and teachers), the ablations that isolate each loss component, and the explicit hyperparameter analysis for the Sinkhorn iterations N and truncation size k. However, the formal specification of the token-level alignment is incomplete in the cross-tokenizer setting, and the main empirical claims are supported only by single-run point estimates. Both issues need to be addressed before the central claims are fully supported.

major comments (3)
  1. [Methods, §3] The token-level OT problem is only well-posed when the teacher and student vocabularies have the same size. Eq. (8) requires a doubly stochastic matrix P of size m x n, forcing m=n, and Eq. (10)-(11) define Q as a permutation matrix, which also requires m=n. Yet every reported experiment uses unequal vocabularies (e.g., LLaMA2-7B has about 32k tokens while OPT/Pythia have about 50k and Bloomz-560M has about 250k). The paper does not state how Q or the top-k truncation in Eq. (12) is computed when m != n, nor whether unmatched dimensions are discarded before or after truncation. Because L_HAD and L_SL in Eqs. (14) and (16) depend on this step, the central loss is underspecified in exactly the cross-tokenizer setting the paper targets and cannot be reproduced from the manuscript alone. Please provide an explicit algorithmic construction for m != n and demonstrate that the identity-P claim in Eqs. (13)-(14) holds under it.
  2. [Experiments, Tables 1-2] The central claim that MultiLevelOT 'consistently outperforms all baseline methods across all datasets and student models' rests on single-run point estimates. No standard deviation, number of random seeds, or significance test is reported, and some margins are small; for example, Table 1, FairytaleQA, OPT-350M: Ours 46.96 vs MinED 46.11. Please report variance across at least three seeds or otherwise justify that the reported differences are not within run-to-run noise.
  3. [Methods, Eqs. (7)-(8)] The same symbol P is used for the token-level transport plan (size m x n before truncation, k x k after) and the sequence-level plan (size T x T), and Eq. (8) states one set of constraints that cannot apply to both plans simultaneously. Please introduce separate notation, e.g., P_tok and P_seq, and state their respective marginal constraints; as written, the objective in Eq. (7) is ambiguous and cannot be verified.
minor comments (6)
  1. [Abstract and §3] The phrase 'eliminating the need for dimensional or token-by-token correspondence' overstates the method: Eqs. (14) and (16) are elementwise after ranking and truncation, so a rank-based dimensional correspondence is still established. Please soften or qualify this claim.
  2. [Experiments, 'Comparison with SOTA'] The 'over 71%' gap reduction in the QED task is consistent with the average row of Table 1 (teacher 61.68, ULD 59.30, Ours 60.99), but per-student reductions vary; please state explicitly that the figure refers to the average over the three students.
  3. [Table 5] The table caption contains a typo: 'FairtaleQA' should be 'FairytaleQA', and 'Rouge-LSUM' should be 'Rouge-LSum' for consistency with the rest of the paper.
  4. [Eq. (10)] The notation 'tSR = t[argsort(...)]' is ambiguous; please clarify that argsort returns the indices that sort the summed logits in descending order and that tSR is the resulting reordered matrix, and define sSR after applying Q in the same way.
  5. [Eq. (15)] The notation s^k_i(t) collides with the truncation index k; consider using a tilde or a superscript '(k)' to denote truncated logits, and define the range of the truncation explicitly.
  6. [Baselines] The exclusion of DSKD is motivated by its added learnable parameters, but since DSKD is a cross-tokenizer baseline, please report its performance as an additional reference or provide a quantitative comparison that isolates the effect of the extra modules.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central outperformance claim is benchmarked against external baselines, and the token-level OT losses reduce to ranked L1/CE by design rather than by circular reasoning.

full rationale

The paper's central claim—that MultiLevelOT outperforms ULD, SeqKD, MinED, and SFT on QED, FairytaleQA, and DIALOGSum—is supported by Tables 1-6 against independent baselines on held-out tasks; no fitted parameter is renamed as a prediction. The derivation from Eq. (7) to Eqs. (14)-(16) is a self-contained construction: after the ranking permutation Q in Eqs. (10)-(11) and the top-k truncation in Eq. (12), the OT problem in Eq. (13) is solved by the identity plan because both vectors are sorted, giving the ranked L1 loss (14), and Eqs. (15)-(16) similarly reduce to ranked cross-entropy. This reduction is a deliberate modeling choice, not a circular dependency. The only self-citations are to SinKD for Sinkhorn distance and for the mode-averaging critique of KL divergence; Sinkhorn distance is standard (Cuturi 2013) and SinKD is independently published, so these citations do not carry the derivation by themselves. The main non-circular weakness is that Eqs. (10)-(11) define Q as a square permutation matrix while all reported teacher-student pairs have different vocabulary sizes (m≠n), leaving the token-level losses underspecified in the exact cross-tokenizer setting; this is a correctness/reproducibility concern, not circularity, and therefore does not raise the circularity score.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The method relies on standard OT facts and on a domain-specific assumption that ranked logit dimensions are semantically comparable across tokenizers. The main free parameters are loss weights, temperatures, Sinkhorn iterations, and the top-k threshold, all set empirically and ablated only on QED.

free parameters (9)
  • alpha = 0.15
    Weight for holistic absolute difference loss; taken from ULD, not re-tuned.
  • beta = 0.1
    Weight for sequential logarithmic loss; taken from ULD.
  • gamma = 0.1
    Weight for Sinkhorn distance loss; empirically set.
  • tau_SL = 1
    Temperature for sequential logarithmic loss; empirically set.
  • tau_SD = 2
    Temperature for Sinkhorn distance loss; empirically set.
  • lambda = 0.1
    Entropy regularization weight for Sinkhorn; empirically set.
  • N = 20
    Number of Sinkhorn iterations; empirically set and ablated on QED.
  • k = 50
    Top-k truncation threshold; empirically set and ablated on QED.
  • learning_rate = 1e-6
    Following ULD.
assumptions (5)
  • standard math The 1D Wasserstein distance between two empirical distributions equals the L1 distance between their sorted values.
    Used to justify the ranked L1 loss in Eq. (5) and Eq. (14).
  • standard math Sinkhorn iterations converge to the entropy-regularized optimal transport plan (Cuturi 2013).
    Used in Eq. (17)-(20) for the sequence-level loss.
  • domain assumption Logits from models with different tokenizers can be meaningfully compared after sequence-level ranking and truncation.
    Central to the token-level losses; not proven, and depends on the permutation Q.
  • domain assumption The permutation Q in Eq. (11) exists and can be found for unequal vocabulary sizes.
    The paper describes Q as a permutation matrix but m and n can differ; no algorithm given.
  • domain assumption Teacher-generated text is a valid substitute for ground truth in unlabeled distillation.
    Used to create pseudo-labels in the unlabeled setting, following ULD.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models." pith.science (2026). https://pith.science/paper/K5I7V237

@misc{pith2026241214528,
  author       = {Pith},
  title        = {Pith review of: Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5I7V237}},
  note         = {Machine review of arXiv:2412.14528}
}
read the original abstract

Knowledge distillation (KD) has become a prevalent technique for compressing large language models (LLMs). Existing KD methods are constrained by the need for identical tokenizers (i.e., vocabularies) between teacher and student models, limiting their versatility in handling LLMs of different architecture families. In this paper, we introduce the Multi-Level Optimal Transport (MultiLevelOT), a novel approach that advances the optimal transport for universal cross-tokenizer knowledge distillation. Our method aligns the logit distributions of the teacher and the student at both token and sequence levels using diverse cost matrices, eliminating the need for dimensional or token-by-token correspondence. At the token level, MultiLevelOT integrates both global and local information by jointly optimizing all tokens within a sequence to enhance robustness. At the sequence level, we efficiently capture complex distribution structures of logits via the Sinkhorn distance, which approximates the Wasserstein distance for divergence measures. Extensive experiments on tasks such as extractive QA, generative QA, and summarization demonstrate that the MultiLevelOT outperforms state-of-the-art cross-tokenizer KD methods under various settings. Our approach is robust to different student and teacher models across model families, architectures, and parameter sizes. Codes and models are available at https://github.com/2018cx/Multi-Level-OT.

Figures

Figures reproduced from arXiv: 2412.14528 by the authors.

Figure 1
Figure 1. An illustration of vocabulary mismatch resulting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our pipeline. MultiLevelOT computes sequence-aware token-level and sequence-level optimal transport [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance at different student scales (Pythia [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion

    cs.CL 2025-01 reject novelty 5.0 of 10

    InfiFusion fuses multiple large language models into one pivot model using enhanced universal logit distillation, and reports that the fused model outperforms all source models on 11 benchmarks with a fraction of the ...

Reference graph

Works this paper leans on

55 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    R.; Geist, M.; and Bachem, O

    Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Garea, S. R.; Geist, M.; and Bachem, O. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR

    work page 2024

  4. [4]

    Arjovsky, M.; Chintala, S.; and Bottou, L. 2017. Wasserstein generative adversarial networks. In ICML, 214--223

    work page 2017

  5. [5]

    Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  6. [6]

    Bhardwaj, R.; Vaidya, T.; and Poria, S. 2021. KNOT: Knowledge distillation using optimal transport for solving NLP tasks. arXiv preprint arXiv:2110.02432

    work page Pith review arXiv 2021

  7. [7]

    G.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M

    Biderman, S.; Schoelkopf, H.; Anthony, Q. G.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In ICML, 2397--2430

    work page 2023

  8. [8]

    Boizard, N.; El-Haddad, K.; Hudelot, C.; and Colombo, P. 2024. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs. arXiv preprint arXiv:2402.12030

    arXiv 2024

Show all 55 references
  1. [9]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. In NeurIPS, 1877--1901

    2020

  2. [10]

    Chen, P.; Zhao, R.; He, T.; Wei, K.; and Yang, Q. 2022. Unsupervised domain adaptation of bearing fault diagnosis based on Join Sliced Wasserstein Distance. ISA transactions, 129: 504--519

    2022

  3. [11]

    Chen, Y.; Liu, Y.; Chen, L.; and Zhang, Y. 2021. DialogSum: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06762

    2021 arXiv

  4. [12]

    Cui, X.; Qin, Y.; Gao, Y.; Zhang, E.; Xu, Z.; Wu, T.; Li, K.; Sun, X.; Zhou, W.; and Li, H. 2024 a . SinKD: Sinkhorn Distance Minimization for Knowledge Distillation. TNNLS

    2024

  5. [13]

    Cui, X.; Qin, Y.; Gao, Y.; Zhang, E.; Xu, Z.; Wu, T.; Li, K.; Sun, X.; Zhou, W.; and Li, H. 2024 b . Sinkhorn Distance Minimization for Knowledge Distillation. In COLING, 14846--14858

    2024

  6. [14]

    Cuturi, M. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 26

    2013

  7. [15]

    R.; Pang, Y.; and Chen, H

    Du, B.; Xie, W.; Li, Y.; Yang, Q.; Zhang, W.; Negenborn, R. R.; Pang, Y.; and Chen, H. 2023. Safe Adaptive Policy Transfer Reinforcement Learning for Distributed Multiagent Control. TNNLS

    2023

  8. [16]

    Fang, G.; Bao, Y.; Song, J.; Wang, X.; Xie, D.; Shen, C.; and Song, M. 2021. Mosaicking to distill: Knowledge distillation from out-of-domain data. In NeurIPS, 11920--11932

    2021

  9. [17]

    Giarelis, N.; Mastrokostas, C.; and Karacapilidis, N. 2023. Abstractive vs. extractive summarization: An experimental review. Applied Sciences, 13(13): 7620

    2023

  10. [18]

    Gu, J.; Qian, X.; Zhang, Q.; Zhang, H.; and Wu, F. 2023 a . Unsupervised domain adaptation for Covid-19 classification based on balanced slice Wasserstein distance. COMPUT BIOL MED, 107207

    2023

  11. [19]

    Gu, Y.; Dong, L.; Wei, F.; and Huang, M. 2023 b . Knowledge Distillation of Large Language Models. arXiv preprint arXiv:2306.08543

    2023 arXiv

  12. [20]

    Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; and Courville, A. C. 2017. Improved training of wasserstein gans. NeurIPS, 30

    2017

  13. [21]

    He, S.; Jiang, Y.; Zhang, H.; Shao, J.; and Ji, X. 2022. Wasserstein unsupervised reinforcement learning. In AAAI, volume 36, 6884--6892

    2022

  14. [22]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  15. [23]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    2023 arXiv

  16. [24]

    Kim, Y.; and Rush, A. M. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947

    2016 arXiv

  17. [25]

    Ko, J.; Kim, S.; Chen, T.; and Yun, S.-Y. 2024. DistiLLM: Towards Streamlined Distillation for Large Language Models. arXiv preprint arXiv:2402.03898

    2024 arXiv

  18. [26]

    B.; and Collins, M

    Lamm, M.; Palomaki, J.; Alberti, C.; Andor, D.; Choi, E.; Soares, L. B.; and Collins, M. 2021. Qed: A framework and dataset for explanations in question answering. TACL, 9: 790--806

    2021

  19. [27]

    Lan, Y.; Xu, X.; Fang, Q.; and Hao, J. 2023. Sample efficient deep reinforcement learning with online state abstraction and causal transformer model prediction. TNNLS

    2023

  20. [28]

    J.; and Piccardi, M

    Li, S.; Unanue, I. J.; and Piccardi, M. 2023. Improving Machine Translation and Summarization with the Sinkhorn Divergence. In PAKDD, 149--161. Springer

    2023

  21. [29]

    Liu, C.; Tao, C.; Feng, J.; and Zhao, D. 2022. Multi-granularity structural knowledge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1001--1011

    2022

  22. [30]

    Liu, Y.; Zhu, L.; Wang, X.; Yamada, M.; and Yang, Y. 2023. Bilaterally normalized scale-consistent sinkhorn distance for few-shot image classification. TNNLS

    2023

  23. [31]

    Lu, S.; Ye, H.-J.; and Zhan, D.-C. 2022. Faculty Distillation with Optimal Transport. arXiv preprint arXiv:2204.11526

    2022 arXiv

  24. [32]

    Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date

    2024

  25. [33]

    S.; Shen, S.; Yong, Z

    Muennighoff, N.; Wang, T.; Sutawika, L.; Roberts, A.; Biderman, S.; Le Scao, T.; Bari, M. S.; Shen, S.; Yong, Z. X.; Schoelkopf, H.; et al. 2023. Crosslingual Generalization through Multitask Finetuning. In ACL, 15991--16111

    2023

  26. [34]

    T.; and Luu, A

    Nguyen, T. T.; and Luu, A. T. 2022. Improving neural cross-lingual abstractive summarization via employing optimal transport distance for knowledge distillation. In AAAI, volume 36, 11103--11111

    2022

  27. [35]

    Park, G.; Kim, G.; and Yang, E. 2021. Distilling Linguistic Context for Language Model Compression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 364--378

    2021

  28. [36]

    Peyr \'e , G.; Cuturi, M.; et al. 2019. Computational optimal transport: With applications to data science. FTML, 11(5-6): 355--607

    2019

  29. [37]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  30. [38]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Y.; Wiseman, S.; and Gimpel, K

    Tu, L.; Pang, R. Y.; Wiseman, S.; and Gimpel, K. 2020. ENGINE: Energy-based inference networks for non-autoregressive machine translation. arXiv preprint arXiv:2005.00850

    2020 arXiv

  32. [40]

    Villani, C.; and Villani, C. 2009. The wasserstein distances. Optimal Transport: Old and New, 93--111

    2009

  33. [41]

    Wan, F.; Huang, X.; Cai, D.; Quan, X.; Bi, W.; and Shi, S. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491

    2024 arXiv

  34. [42]

    Wei, Y.; Li, X.; Lin, L.; Zhu, D.; and Li, Q. 2022. Causal discovery on discrete data via weighted normalized Wasserstein distance. TNNLS

    2022

  35. [43]

    Weilin, C.; Jie, Q.; Ruichu, C.; and Zhifeng, H. 2023. On the Role of Entropy-Based Loss for Learning Causal Structure With Continuous Optimization. TNNLS

    2023

  36. [44]

    Wen, Y.; Li, Z.; Du, W.; and Mou, L. 2023. f-Divergence Minimization for Sequence-Level Knowledge Distillation. In ACL, 10817--10834

    2023

  37. [45]

    Wu, T.; Tao, C.; Wang, J.; Zhao, Z.; and Wong, N. 2024. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. arXiv preprint arXiv:2404.02657

    2024 arXiv

  38. [46]

    Xu, J.; Li, C.; Huang, F.; Li, Z.; Xie, X.; and Philip, S. Y. 2023. Sinkhorn distance minimization for adaptive semi-supervised social network alignment. TNNLS

    2023

  39. [47]

    J.-J.; Bradford, N.; Sun, B.; et al

    Xu, Y.; Wang, D.; Yu, M.; Ritchie, D.; Yao, B.; Wu, T.; Zhang, Z.; Li, T. J.-J.; Bradford, N.; Sun, B.; et al. 2022. Fantastic Questions and Where to Find Them: FairytaleQA--An Authentic Dataset for Narrative Comprehension. arXiv preprint arXiv:2203.13947

    2022 arXiv

  40. [48]

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  41. [49]

    M.; Li, Z.; Mallya, A.; Hoiem, D.; Jha, N

    Yin, H.; Molchanov, P.; Alvarez, J. M.; Li, Z.; Mallya, A.; Hoiem, D.; Jha, N. K.; and Kautz, J. 2020. Dreaming to distill: Data-free knowledge transfer via deepinversion. TPAMI, 8715--8724

    2020

  42. [50]

    Zhang, J.; Liu, T.; and Tao, D. 2021. An optimal transport analysis on generalization in deep learning. TNNLS, 34(6): 2842--2853

    2021

  43. [51]

    Zhang, J.; Muhamed, A.; Anantharaman, A.; Wang, G.; Chen, C.; Zhong, K.; Cui, Q.; Xu, Y.; Zeng, B.; Chilimbi, T.; et al. 2023. ReAugKD: Retrieval-augmented knowledge distillation for pre-trained language models. In Proceedings of the Annual Meeting of the Association for Compu...

    2023

  44. [52]

    Zhang, Q.; Leng, S.; Ma, X.; Liu, Q.; Wang, X.; Liang, B.; Liu, Y.; and Yang, J. 2024 a . CVaR-Constrained Policy Optimization for Safe Reinforcement Learning. TNNLS

    2024

  45. [53]

    V.; et al

    Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  46. [54]

    Zhang, S.; Zhang, X.; Sun, Z.; Chen, Y.; and Xu, J. 2024 b . Dual-Space Knowledge Distillation for Large Language Models. arXiv preprint arXiv:2406.17328

    2024 arXiv

  47. [55]

    Zhou, W.; Xu, C.; and McAuley, J. 2022. BERT Learns to Teach: Knowledge Distillation with Meta Learning. In ACL, 7037--7049

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.