Pith. sign in

REVIEW 4 major objections 4 minor 48 references

InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fusing three specialized LLMs into one pivot model beats all three on 11 benchmarks, using 160 H800 GPU hours.

desk verdict A plausible incremental fusion method that deserves review, but the central loss equation is underspecified and needs fixing. read the letter →

arxiv 2501.02795 v3 pith:3ZWVZPFI submitted 2025-01-06 cs.CL cs.CV

Zhaoyi Yan , Yiming Zhang , Baoyi He , Yuhao Fu , Qi Zhou , Zhijie Sang , Chunlin Ji , Shengyu Zhang
show 2 more authors
Fei Wu Hongxia Yang
This is my paper · ORCID
classification cs.CLcs.CV
keywords LLMfusionknowledgedistillationUniversalLogitTop-Klogitsstandardizationmulti-teachercross-vocabularyalignmentmodelmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a single pivot language model can absorb the complementary skills of several domain-specialized models without retraining from scratch or requiring the models to share vocabularies. The recipe is to distill only the top-$K$ logits of each source model into the pivot, z-score normalizing those logits first, and to measure the teacher-student gap with a 1-Wasserstein distance that works across different vocabularies. On 11 benchmarks covering math, code, reasoning, instruction following, and text reasoning, the fused model scores 79.92 on average, above the pivot Phi-4 (75.08) and all three source models, while training in about 160 H800 GPU hours. The practical stakes are large: if this holds, model improvement can come from fusing existing specialized models cheaply rather than pretraining ever-larger models.

What carries the argument

The load-bearing object is an enhanced Universal Logit Distillation loss. Universal Logit Distillation compares two models' next-token probability distributions that live on different vocabularies by sorting each distribution's probabilities and taking the 1-Wasserstein distance between the sorted lists; InfiFusion modifies this by keeping only the top-$K$ logits and standardizing them, subtracting the mean and dividing by the standard deviation, before taking the elementwise $\ell^1$ difference. This makes the distillation loss insensitive to logit scale and to the long tail of low-probability tokens, and it lets one pivot model be trained against several sources at once. The framework also contributes two fusion pipelines: pairwise fusion, which trains one pivot copy per source and then merges the copies with task arithmetic, TIES, or SCE; and unified fusion, which sums the per-source losses into a single training run.

What would settle it

One decisive check: take a fixed source model and a fixed pivot, permute the source's vocabulary labels consistently, and recompute the Equation (5) loss; a loss that is a true cross-vocabulary distance should be unchanged under relabeling, whereas an elementwise top-K comparison that does not sort or map tokens will change. Running the full InfiFusionu pipeline with source models that share almost no tokenizer vocabulary with the pivot would settle the same question at the benchmark level.

Watch

Extended reading notes

Core claim

The paper's central claim is that the main obstacle to fusing models with different tokenizers and training histories, namely that their next-token distributions live on different vocabularies, can be sidestepped by comparing only the most probable logits after standardizing them. InfiFusion takes a pivot model and distills each source model's knowledge into it by minimizing, at each time step, the elementwise $\ell^1$ difference between the pivot's and the source's top-$K$ logits, after each has been z-score normalized, plus a supervised fine-tuning loss on the same data. In unified fusion, the losses from all sources are summed and one training run produces the final model; in pairwise fusion, each source is distilled separately and the resulting models are merged by parameter arithmetic. The same 11-benchmark average is reached by both strategies, with unified fusion using about one third of the GPU hours. The paper concludes that the combination of Universal Logit Distillation, top-$K$ selection, and logits standardization is what makes cross-model fusion work.

Load-bearing premise

The load-bearing premise is that the elementwise difference between top-K logits from models with different vocabularies is a meaningful distance, yet the paper never states how the K tokens from each vocabulary are matched before subtraction.

Editorial extensions

If this is right

  • If InfiFusion's result holds, a small pivot model can surpass each of its equal-sized or larger specialists on aggregate benchmarks, so model capability can be pooled without merging parameters directly.
  • Unified fusion reaches nearly the same average score as the more expensive pairwise-then-merge route while using about $1/S$ of the GPU hours, with $S=3$, suggesting simultaneous multi-teacher distillation is the cheaper route to the same capability.
  • Because the distance is computed on standardized logits, source models do not need to share vocabularies or architectures with the pivot, which widens the set of models that can be fused.
  • The ablations tie performance to top-$K$ selection, best at $K=10$, and to logits standardization, indicating that noise from low-probability logits and scale differences are the main things that hurt naive logit distillation.
  • The same training recipe benefits from more fusion data, with 144k samples beating 50k, so further gains are expected from scaling the distillation dataset rather than from more compute per sample.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the reported benchmarks, the same top-$K$ standardized-logit loss should apply to fusing more than three sources or to distilling from a much larger teacher into a small pivot, since the per-source cost is additive and the authors' own ablations show gains from additional sources and data.
  • A direct test of the method's mechanism would be to fuse source models deliberately chosen to have almost no tokenizer overlap with the pivot; the gains would either persist if the standardized sorted-logit distance is doing the work or shrink if hidden token alignment is responsible.
  • Because logits are z-score standardized, the loss is invariant to each model's output scale, which suggests InfiFusion could also serve as a calibration-agnostic distiller, transferring ranking knowledge between models with very different confidence distributions, an effect the paper does not measure.
  • The 160 GPU-hour figure excludes the 1.5 TB of pre-extracted teacher hidden states; at much larger data scales, storage and I/O rather than GPU time may become the true cost of fusion, so the efficiency advantage should be quoted with that preprocessing included.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces InfiFusion, a framework for fusing multiple domain-specialized LLMs into a single pivot model using a distillation objective that combines Top-K logit selection, logit standardization, and the Universal Logit Distillation loss. Two fusion strategies are proposed: pairwise fusion, where each source is distilled separately and then the resulting models are merged, and unified fusion, where all sources are distilled simultaneously. The authors report that InfiFusion outperforms the pivot model Phi-4 and several baselines across 11 benchmarks, while requiring only about 160 H800 GPU hours for the unified variant. The paper also includes ablations on the Top-K value, logits standardization, number of source models, and dataset scale, plus an appendix that attempts to relate the two fusion strategies.

Significance. If the method is implementable as described and the reported gains are robust, the work would offer a computationally efficient approach to combining heterogeneous LLMs without requiring identical architectures or vocabularies. The idea of extending Universal Logit Distillation with Top-K selection and standardization is plausible and could be a useful contribution to model fusion. However, the central training objective is underspecified in a way that prevents reproduction, the theoretical analysis in the appendix is not a proof, and the empirical validation lacks statistical rigor and suffers from model selection on the evaluation benchmarks. These issues substantially weaken the significance of the claims as currently presented.

major comments (4)
  1. [Section 3.3.1, Eq. (5)] The pairwise fusion loss in Eq. (5) is defined as an L1 distance between normalized top-K logits of the pivot and source models, but the source and pivot models have different vocabularies. The paper never states how the K logits from each model are matched: are they sorted by value, aligned by token ID, or mapped through a shared space? Without this specification, the elementwise subtraction in Eq. (5) is not a well-defined distance, and the training objective cannot be implemented or reproduced. If the intended operation is sorting by value, that should be stated explicitly, since it changes the loss from a token-level alignment to a comparison of order statistics; if some other alignment is used, it must be described. This is a load-bearing issue because Eq. (5) underlies both InfiFusion_p and InfiFusion_u.
  2. [Appendix A, Eq. (20)] The paper claims in Section 3.3.2 to provide 'a detailed theoretical analysis' of the relationship between unified and pairwise fusion, but Appendix A does not prove that relationship. Equation (20) simply assumes that a global loss L exists such that the zero-gradient condition at the individual optima implies the zero-gradient condition at the averaged parameters. This is an unproved assumption, not a derivation. The appendix should either present a rigorous proof under explicit conditions or clearly label this as a conjecture. As written, the theoretical support for pairwise fusion is absent.
  3. [Section 4.1 and Table 5 / Table 7] The choice of K=10 in Eq. (5) is selected by ablating on the same 11 benchmarks used for the final evaluation (Table 5), and the pivot and source models are selected based on Table 7, which reports performance on the same 11 benchmarks. This constitutes selection on the evaluation metric, which can inflate reported performance. The authors should use a separate validation set for model selection and hyperparameter tuning, or at minimum report the results across multiple random seeds and show that the selected configuration generalizes. Without this, the reported average scores are difficult to interpret.
  4. [Table 2 and Section 4.2] All benchmark numbers in Table 2 are single runs without error bars, standard deviations, or multiple seeds. The reported differences are small: InfiFusion_u achieves 79.92, InfiFusion_p&TA 79.96, FuseChat 78.70, and SFT 77.96. It is unclear whether these gaps are statistically meaningful given the lack of repeated trials. The authors should report variance across at least three seeds and, if possible, significance tests for the main comparisons.
minor comments (4)
  1. [Section 3.1.3, Eq. (3)] Equation (3) defines the 1-Wasserstein distance with a sum over V elements, but if Vo and Vs differ, the sorted sequences have different lengths. The text explains padding with zeros, but this should be explicitly reflected in the equation or its surrounding notation for clarity.
  2. [Section 3.3.1, Eq. (6)] The sentence following Eq. (6) has a grammatical issue: 'Let Mo,s be the optimized pivot model, its parameters are θo,s are obtained via' should be revised to a single coherent statement.
  3. [Appendix A, first sentence] The appendix opens with 'In this session, we analyze the relationship...' — 'session' should be 'section'.
  4. [Abstract] The abstract contains a grammatical error: 'significantly reduces computational costs' should be 'significantly reducing computational costs' for parallel structure with the preceding phrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported benchmark gains are external empirical results, not forced by construction; the notable weakness is an underspecified loss definition, which is a reproducibility concern rather than a circularity.

full rationale

The paper's central claim is an empirical one: InfiFusion_u obtains 79.92 average score on 11 external benchmarks by training with a ULD-style loss plus Top-K selection and logits standardization. None of the reported scores are defined in terms of the training loss, and no equation in the paper reduces the claimed outcome to its own inputs. Equation (5) is a training objective, not a derived prediction; even if the cross-vocabulary alignment of Top-K logits is underspecified, that is a correctness/reproducibility issue, not a circular dependency. The ULD loss is attributed to external prior work (Cui et al., 2024; Boizard et al., 2025), and logits standardization is attributed to Sun et al., 2024; there is no load-bearing self-citation chain. The choice K=10 and the selection of pivot/source models were made using the same benchmark suite, which is a mild selection-on-evaluation concern, but the final numbers are measured on held-out benchmarks and are not fitted parameters renamed as predictions. No uniqueness theorem, ansatz smuggled via self-citation, or renaming of a known result carries the argument. Therefore the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is empirical, so the main burden is on the loss definition and the model-selection procedure. The ill-defined Equation (5) is the heaviest item.

free parameters (3)
  • K (Top-K size) = 10
    Chosen by ablation on the 11-benchmark average (Table 5); the reported results depend on this choice, and values of 5, 15, 20, 25 give lower scores.
  • lambda (loss balancing weight) = 0.5
    Set by default without sensitivity analysis; the fusion results depend on the balance between distillation and SFT losses.
  • Dataset mixture sizes = 80K general / 50K math / 50K code
    Hand-picked sample counts from Infinity-Instruct, ScaleQuest-Math, and opc-sft-stage2; no justification is given for these proportions.
assumptions (4)
  • standard math The 1-Wasserstein distance with uniform cost is a valid alignment for cross-tokenizer distributions (Eq. 3).
    Used to define ULD loss in Eq. (4), following Boizard et al. and Cui et al.
  • ad hoc to paper Top-K logits from different vocabularies can be meaningfully compared after z-score standardization (Eq. 5).
    The paper does not define a token correspondence; this is the load-bearing assumption that the central loss works.
  • ad hoc to paper A global loss L exists for pairwise fusion such that its optimum equals the average of the per-source local optima (Appendix A, Eq. 20).
    Stated as an assumption with no proof; used to argue that unified fusion is a global optimizer.
  • domain assumption Model merging methods (Task Arithmetic, TIES, SCE) preserve the distilled knowledge when combining per-source copies (Eq. 7).
    Borrowed from prior merging literature; the paper uses default hyperparameters and reports performance drops for TIES/SCE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion." pith.science (2026). https://pith.science/paper/3ZWVZPFI

@misc{pith2026250102795,
  author       = {Pith},
  title        = {Pith review of: InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ZWVZPFI}},
  note         = {Machine review of arXiv:2501.02795}
}
abstract

We introduce InfiFusion, an efficient training pipeline designed to integrate multiple domain-specialized Large Language Models (LLMs) into a single pivot model, effectively harnessing the strengths of each source model. Traditional fusion methods either merge model parameters directly or rely on knowledge distillation with rigid assumptions, limiting their flexibility and efficiency. InfiFusion overcomes these limitations by enhancing Universal Logit Distillation (ULD) with Top-K selection and Logits Standardization. We propose two fusion strategies: Pairwise Fusion (InfiFusion$_p$), where each source model knowledge is distilled individually into the pivot model followed by merging and Unified Fusion (InfiFusion$_u$), where knowledge from all source models is distilled simultaneously into the pivot model. InfiFusion outperforms the state-of-the-art models, such as Qwen-2.5-14B-Instruct and Phi-4, across 11 widely applied benchmarks covering reasoning, coding, mathematics, and instruction-following tasks. Notably, InfiFusion achieves this superior performance while significantly reduces computational costs, completing full training with only 160 H800 GPU hours compared to the millions typically required for traditional LLM training.

Figures

Figures reproduced from arXiv: 2501.02795 by the authors.

Figure 1
Figure 1. Performance of InfiFusion on the pivot models [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the InfiFusion framework, incorporating Top-K selection and logits standardization. Two [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 9 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  2. [2]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations

    2024

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [5]

    Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. 2025. https://arxiv.org/abs/2402.12030 Towards cross-tokenizer distillation: the universal logit distillation loss for llms . Preprint, arXiv:2402.12030

    arXiv 2025

  5. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [7]

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889--7901

    work page 2023

  7. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [9]

    Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. 2024. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. arXiv preprint arXiv:2412.14528

    arXiv 2024

Show all 48 references
  1. [10]

    Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. 2024. Unleashing reasoning capability of llms via scalable question synthesis from scratch. arXiv preprint arXiv:2410.18693

    2024 arXiv

  2. [11]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161

    2019 arXiv

  3. [12]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations

    2024

  4. [13]

    Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. 2020. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11020--11029

    2020

  5. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  6. [15]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  7. [16]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  8. [17]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. 2024. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905

    2024 arXiv

  9. [18]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  10. [19]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations

  11. [20]

    Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317--1327

    2016

  12. [21]

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898

    2024 arXiv

  13. [22]

    Young-Suk Lee, Md Sultan, Yousef El-Kurdi, Tahira Naseem, Asim Munawar, Radu Florian, Salim Roukos, and Ram \'o n Fernandez Astudillo. 2023. Ensemble-instruct: Instruction tuning data generation with a heterogeneous mixture of lms. In Findings of the Association for Computatio...

    2023

  14. [23]

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. arXiv preprint arXiv:2402.00530

    2024 arXiv

  15. [24]

    Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. 2024. https://arxiv.org/abs/2411.16085 Cautious optimizers: Improving training with one line of code . Preprint, arXiv:2411.16085

    2024

  16. [25]

    Michael Matena and Colin Raffel. 2022. https://arxiv.org/abs/2111.09832 Merging models with fisher-weighted averaging . Preprint, arXiv:2111.09832

    2022 arXiv

  17. [26]

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. In The Thirty-eighth Annual ...

    2024

  18. [27]

    Cuong Pham, Tuan Hoang, and Thanh-Toan Do. 2023. Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6435--6443

    2023

  19. [28]

    Qwen, An Yang, and Baosong Yang etal. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2025 arXiv

  20. [29]

    V Sanh. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

    2019 arXiv

  21. [30]

    Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. 2024. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408.15881

    2024 arXiv

  22. [31]

    Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. 2024. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731--15740

    2024

  23. [32]

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. https://doi.org/10.18653/v1/D19-1441 Patient knowledge distillation for BERT model compression . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confe...

    2019 doi

  24. [33]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  25. [34]

    Inar Timiryasov and Jean-Loup Tastet. 2023. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. arXiv preprint arXiv:2308.02019

    2023 arXiv

  26. [35]

    Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1908.08962 Well-read students learn better: On the importance of pre-training compact models . Preprint, arXiv:1908.08962

    2019 arXiv

  27. [36]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024 a . https://openreview.net/pdf?id=jiDsk12qcz Knowledge fusion of large language models . In The Twelfth International Conference on Learning Representations

    2024

  28. [37]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024 b . Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491

    2024 arXiv

  29. [38]

    Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. 2024 c . Fusechat: Knowledge fusion of chat models. arXiv preprint arXiv:2402.16107

    2024 arXiv

  30. [39]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, and Simon Kornblith. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing i...

    2022

  31. [40]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

    2024

  32. [41]

    Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprint arXiv:1911.07176

    2019 arXiv

  33. [42]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099

    2023 arXiv

  34. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  35. [44]

    Mingyang Zhang, Jing Liu, Ganggui Ding, Xinyi Yu, Linlin Ou, and Bohan Zhuang. 2024 a . https://arxiv.org/abs/2412.15283 Channel merging: Preserving specialization for merged experts . Preprint, arXiv:2412.15283

    2024 arXiv

  36. [45]

    Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024 b . https://arxiv.org/abs/2406.17328 Dual-space knowledge distillation for large language models . Preprint, arXiv:2406.17328

    2024 arXiv

  37. [46]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  38. [47]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931

    2024 arXiv

  39. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.