Pith. sign in

REVIEW 4 major objections 8 minor 55 references

MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

T0 review · 4 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing unweighted averaging with Dare or Ties merging, plus perplexity-based routing and separated attention layers, improves homogeneous Mixture-of-Experts merging, and that heterogeneous experts with different…

desk verdict Solid homogeneous MoE merging results, but the heterogeneous router claim is undercut by the paper's own routing analysis; needs controls before the headline claim holds. read the letter →

arxiv 2502.00997 v3 pith:SZSNJS5K submitted 2025-02-03 cs.CL cs.AI

Yuhang Zhou , Giannis Karamanolakis , Victor Soto , Anna Rumshisky , Mayank Kulkarni , Furong Huang , Wei Ai , Jianhua Lu This is my paper · ORCID
classification cs.CLcs.AI
keywords Mixture-of-ExpertsmergingmodeltaskvectorsDareTiesperplexityroutingheterogeneousexpertparameterinterference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard recipe for merging domain-specialized language models into a Mixture-of-Experts model, keep feed-forward layers separate and unweighted-average everything else, leaves performance on the table because averaged layers suffer parameter interference. Its central claim is that replacing averaging with Dare or Ties task-vector merging, routing sequences by perplexity, and keeping attention layers separate yields better merged MoEs, especially when post-merge fine-tuning is limited. For experts with different architectures, the paper claims that a projector-and-router design can merge them into a single MoE that routes each input sequence to the most competent expert. This matters because it lowers the cost of building multi-domain MoEs and widens merging to models from different teams and architectures.

What carries the argument

The workhorse is the task vector, the difference between an expert's parameters and the base model's parameters. Dare and Ties are used as merging operators on non-FFN layers: Dare randomly drops a fraction of task-vector entries and rescales, while Ties zeroes small-magnitude entries and resolves sign conflicts before summing, reducing interference. For routing without fine-tuning, the paper uses a perplexity-based heuristic: the reciprocal of each expert's perplexity on the input sequence is turned into softmax weights over the top-K experts. For heterogeneous experts, the machinery is three-part: a shared embedding and head initialized by averaging expert embeddings, per-expert randomly initialized Proj-in and Proj-out layers that translate between the shared hidden dimension and each expert's dimension, and a sequence-level router built on the average token embedding.

What would settle it

Re-run the heterogeneous merge with the shared embedding and head replaced by a learned alignment module fit on paired domain data, and check whether the math expert becomes the top-routed expert on GSM8K and MATH. If accuracy and math routing share do not improve, the claim that the router dynamically selects the most competent expert is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the choice of layer-merging operator is load-bearing for MoE merging. In the homogeneous setting, applying Dare or Ties merging to all non-FFN layers, instead of unweighted averaging, consistently improves average performance over the Branch-Train-Mix baseline across math, code, and knowledge benchmarks, and the improvement is largest in the early stages of fine-tuning. When no fine-tuning is affordable, a sequence-level router that selects experts by lowest perplexity, combined with unmerged attention layers, beats random routing and dense Dare/Ties merging. In the heterogeneous setting, the paper shows that experts with different depths, hidden dimensions, and tokenizers can be combined by inserting randomly initialized projector layers into a shared embedding space and training a sequence-level router; the resulting MoE outperforms each dense expert in its own domain and beats a three-expert homogeneous MoE baseline.

Load-bearing premise

The heterogeneous merging result stands on the assumption that a shared embedding and head formed by simply averaging expert parameters, together with randomly initialized projectors, can align the hidden spaces of models with different architectures and tokenizers well enough for the trained router to identify the most competent expert per input.

Editorial extensions

If this is right

  • Using Dare or Ties merging on non-FFN layers raises average MoE performance by 6.94% and 9.72% relative to BTX averaging, and routes more tokens to the domain expert on math benchmarks.
  • With no fine-tuning, perplexity routing plus separated attention layers outperforms random routing by 16.8% and dense Dare merging by 13.6% on average.
  • Heterogeneous merging with projectors and sequence-level routing achieves 43.02% and 27.78% relative improvement over the best dense experts for MoEs with Math Olmo and Math TinyLlama, respectively.
  • The heterogeneous method extends MoE merging to models with different layer counts, hidden dimensions, and tokenizers, which previous BTX-style merging could not handle.
  • Both homogeneous and heterogeneous methods reduce reliance on expensive post-merge MoE fine-tuning, cutting training cost from 448 to 100 (B parameters times B tokens) in the no-fine-tuning setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the paper's own routing analysis suggests the heterogeneous router is partly an output ensemble: the math expert receives the second-highest routing share on math benchmarks because the averaged shared embedding sits closer to the other experts. A load-balancing loss or a learned embedding alignment is a natural test of whether true expert specialization is achievable.
  • Beyond the paper, the perplexity router is a testable approximation to an oracle router; comparing its domain-expert recall against an oracle that always picks the true domain expert on held-out domain data would quantify how much of the gain comes from routing accuracy versus the merged weights themselves.
  • Beyond the paper, the projector design should transfer to multimodal MoEs with vision, audio, or graph experts, as the paper notes, but the same embedding-alignment risk would apply whenever tokenizers or input spaces differ.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper studies merging multiple domain-specialized LLMs into a single Mixture-of-Experts model. For homogeneous experts (same architecture branched from a common base), it proposes replacing the unweighted averaging of non-FFN layers used by BTX with Dare or Ties task-vector merging, adding a sequence-level router based on the perplexity of the inference prompt, and keeping attention layers separate to avoid task-vector count mismatch. For heterogeneous experts (different architectures and tokenizers, e.g., TinyLlama and OLMo), it proposes projecting inputs and outputs through randomly initialized MLP projectors, a shared averaged embedding and head, and a fine-tuned sequence-level router. Experiments on six benchmarks with 1B-scale models report higher average scores than BTX and improved no-fine-tuning performance, and the authors claim the first framework for merging heterogeneous models into an MoE.

Significance. The homogeneous contribution is incremental but useful: applying established dense-merging techniques (Dare/Ties) to MoE non-FFN layers is natural, and the paper documents consistent gains over BTX with transparent training-cost tables. The no-fine-tuning routing heuristics are practical and falsifiable. The heterogeneous claim is the main novelty and is currently not supported by the evidence: the paper's own routing analysis shows the math expert is not selected first on math inputs, and no control separates the effect of the trained router from output ensembling or extra parameters. The manuscript's detailed routing figures and explicit limitation discussion are commendable, but the central dynamic-routing claim requires additional controls before the paper can be accepted.

major comments (4)
  1. [Section 5.2, Figure 6] On GSM8K and MATH, the math expert (Math-Olmo or Math-TinyLlama) receives the second-highest routing probability, not the highest. This directly contradicts the abstract and Section 3.2 claim that the router 'learns to route token sequences dynamically to the appropriate expert.' To support the routing claim, the paper should report oracle-routing (force math expert as top-1), uniform-top-2 (no router), and output-ensemble (average expert outputs with fixed equal weights) controls on the same benchmarks. It should also include a parameter-matched baseline, since the heterogeneous MoE uses roughly 4B parameters versus 2.8B for the 3-expert MoE baseline; without these, the Table 4 gains cannot be attributed to learned routing.
  2. [Section 3.2, Table 4] The alignment mechanism—randomly initialized Proj-in/Proj-out layers and an averaged shared embedding/head—is asserted rather than validated. The paper's own Section 5.2 attributes the math routing failure to embedding/tokenizer mismatch, which is an admission that the central assumption of the method is only partially met. Please add quantitative alignment diagnostics, such as token-level embedding distances or CCA between expert hidden spaces before and after projection, routing accuracy against an oracle label, and multiple random seeds for the projector initializations, to show that the router is using meaningful cross-expert alignment rather than picking experts by an artifact of the averaged embedding.
  3. [Section 5.1.2, Table 3] The no-fine-tuning comparison lacks the full 2x2 design. There is a 'Merge attention + random routing' row and a 'Separate attention + PPL routing' row, but no 'Separate attention + random routing' row. Since separating attention increases parameter count and changes the model, the relative contribution of the PPL router cannot be isolated. Add this control, and ideally also 'Merge attention + PPL,' to support the claim that both techniques each contribute to the 8.08 average.
  4. [Appendix A] The hyperparameters p (retain ratio) and lambda (scaling) are reported as set to 80% and 1/3 'according to our preliminary exploration.' If this exploration used the same evaluation benchmarks, the reported gains over BTX are partially the result of test-set selection. Specify the validation split and show sensitivity of Table 1 and Table 3 results to p and lambda, or use a fixed default from the dense-merging literature.
minor comments (8)
  1. [Section 4] The word 'settting' in the opening sentence of Section 4 is a typo and should be 'setting.'
  2. [Appendix A] The word 'hiddn' in the description of the Olmo-1B model is a typo and should be 'hidden.'
  3. [Figure 8 caption] The caption contains the typo 'HuamnEval' and should read 'HumanEval.'
  4. [Figure 9 caption] The caption contains the typo 'T ask Vector Routing' and should read 'Task Vector Routing.'
  5. [Section 3.1.2] The notation for the retain ratio p is unclear: p is described as a percentage (80%) but the rescaling formula uses 0.01p; clarify whether p is a fraction or a percentage and make the formula consistent.
  6. [Section 7] The phrase 'finisher merging' should read 'Fisher merging.'
  7. [Section 3.2] The padding-zero averaging step is confusing because all experts in the experiments have the same hidden dimension; state explicitly whether this step applies only when hidden dimensions differ and how padding zeros affect the average.
  8. [Table 1] The header 'Avg.Dense' appears to have a formatting issue and should be split into 'Avg.' and 'Dense' for readability.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild circularity: Dare/Ties gains are partly produced by tuning p and λ on the evaluation benchmarks; the rest of the pipeline is externally evaluated.

  1. fitted input called prediction [Appendix A (Implementation Details); results in Tables 1 and 3]
    "For Dare-merging and Ties merging (both dense and MoE), we set the scaling term λ to 1 3 and the retain ratio p of the model parameters of two methods are set to 80% to gain the optimal performance, according to our preliminary exploration."

    The two hyperparameters (retain ratio p and scaling term λ) are explicitly selected 'to gain the optimal performance' on the same six benchmarks whose numbers are then reported as evidence that Dare/Ties merging outperforms BTX and dense baselines. Because no separate validation split is described, the reported superiority is the result of optimizing these hyperparameters on the evaluation metric itself, so the comparative claim is partly a fitted outcome rather than an independent prediction. This is a mild circularity: the other contributions (PPL routing, attention separation, heterogeneous projectors + router) retain external, label-free content, but the headline Dare/Ties gains are inflated by test-set selection.

full rationale

The paper is largely self-contained: the homogeneous and heterogeneous merging pipelines are evaluated on external benchmarks (GSM8K, MATH, MBPP, HumanEval, NQ, TriviaQA), and the routing heuristics use input perplexity or gradients rather than benchmark labels, so the routing-performance link is not definitional. The heterogeneous router's failure to route math inputs to the math expert (Figure 6, Section 5.2) is a limitation and a confound, not a circular reduction: the router is trained, and the math gains could partly be an ensemble or capacity effect, but the claim does not reduce to its inputs by construction. The single mild circularity is the selection of the Dare/Ties hyperparameters p and λ (and the 'optimal performance' language in Appendix A) on the same benchmarks used to report the headline improvements; this inflates the Dare/Ties-vs-BTX comparison but does not undermine the independent PPL routing, attention-separation, or heterogeneous-framework contributions. Self-citations appear only in future-work or limitation passages and are not load-bearing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The paper's empirical gains depend on several unproven assumptions: that dense merging recipes transfer to MoE non-FFN layers, that prompt perplexity is a reliable routing signal, that separating attention helps, and that averaged embedding/head layers with random projectors align heterogeneous models. The two hyperparameters p and lambda were tuned on the evaluation benchmarks, which is a form of fitting to the test data.

free parameters (2)
  • Dare/Ties retain ratio p = 0.8 (80%)
    Set to 80% to gain optimal performance according to preliminary exploration (Appendix A). This is tuned on the evaluation benchmarks, so the reported gains may be optimistic.
  • Dare/Ties scaling lambda = 1/3
    Also set for optimal performance via preliminary exploration (Appendix A). Affects the magnitude of merged task vectors.
assumptions (4)
  • domain assumption Task vector interference, as characterized in dense merging, transfers to the non-FFN layers of MoE merging.
    The paper applies Dare and Ties to attention, embedding, and norm layers based on the dense merging literature (Sections 2.1 and 3.1.2), without proving that the same interference patterns hold in this setting.
  • ad hoc to paper Perplexity of the inference prompt on an expert is a good proxy for that expert's ability to continue the sequence.
    This is the paper's routing heuristic (Section 3.1.3). It is plausible but not derived, and the paper's own Table 2 shows it misroutes MBPP (code) prompts to the math expert.
  • ad hoc to paper Separating attention layers removes inconsistency caused by different numbers of task vectors in attention vs FFN layers.
    Stated as a hypothesis in Section 3.1.3; the ablation (Table 3) does not consistently support it, since task-vector routing with separated attention is worse than with merged attention.
  • domain assumption Averaging embedding and head layers, with zero-padding for smaller models, provides a valid initialization for the shared embedding and head in heterogeneous merging.
    Assumed in Section 3.2 without analysis of whether zero-padding distorts the representation space.
invented entities (2)
  • Proj-in and Proj-out projector layers
    purpose: Map hidden states between the shared maximum dimension and each heterogeneous expert's dimension.
    Randomly initialized MLPs trained as part of the merged MoE; no external falsifiable prediction.
  • Shared embedding layer Me and shared head Mh
    purpose: Provide a common token representation and output space across experts with different vocabularies or embedding sizes.
    Initialized from an average of expert embeddings and heads with zero-padding; internal to the model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs." pith.science (2026). https://pith.science/paper/SZSNJS5K

@misc{pith2026250200997,
  author       = {Pith},
  title        = {Pith review of: MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZSNJS5K}},
  note         = {Machine review of arXiv:2502.00997}
}
read the original abstract

The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging.

Figures

Figures reproduced from arXiv: 2502.00997 by the authors.

Figure 1
Figure 1. Overview of the proposed MoE framework for homogeneous model merging. We replace aver￾aging with Dare or Ties merging to reduce parameter interference. Additionally, we introduce novel routing heuristics to enhance performance without fine-tuning. 3 Methodology We define our research problem as follows: Given l dense expert models with parameters [θ1, θ2, . . . , θl ], each pretrained on different do￾mains, we aim t… view at source ↗
Figure 2
Figure 2. Different types of parameter interference and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed MoE framework for heterogeneous experts. Each color represents one heterogeneous expert. n1, · · · , n4 refers to the number of layers in each expert. embedding and head layer parameters are initial￾ized from an averaging of the embedding and head layers of each expert. For experts with a hidden dimension less than dm, we add padding zeros for their embedding and head layers before averaging… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: 0 5 10 15 20 25 Layer number 0.0 0.1 0.2 0.3 0.4 0.5 Cosine similarity Attention FFN [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Routing probability of experts on GSM8K and MATH for different merging methods. ity of each method of two math datasets (MATH and GSM8K) in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Routing probability of experts on GSM8K and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Routing probability of experts on MBPP, Hu [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Routing probability of tow routing heuristics [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 8
Figure 8. Figure 8: Routing probability of experts on MBPP, HuamnEval, Natural Questions and TriviaQA for the MoE w/ Olmo and MoE w/ TinyLlama. weights with the cosine similarity (Sim) as below: α = SoftMax(top-K(Sim(ginf , τ1), . . . , Sim(ginf , τl))) D Supplementary Results In this sec…
Figure 10
Figure 10. Figure 10: Performance with varied fine-tuning token [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 3 canonical work pages

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  5. [5]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092

    arXiv 2023

  6. [6]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066

    arXiv 2024

  7. [7]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. 2024. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932--1945

    work page 2024

  8. [8]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

Show all 55 references
  1. [9]

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257

    2024 arXiv

  2. [10]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838

    2024 arXiv

  3. [11]

    Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2023. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177

    2023 arXiv

  4. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  7. [15]

    Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63--S63

    1977

  8. [16]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  9. [17]

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849

    2022 arXiv

  10. [18]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  11. [19]

    Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. 2024. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406.12034

    2024 arXiv

  12. [20]

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055

    2022 arXiv

  13. [21]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  14. [22]

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306

    2022 arXiv

  15. [23]

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. 2024 a . Uni-moe: Scaling unified multimodal llms with mixture of experts. arXiv preprint arXiv:2405.11273

    2024 arXiv

  16. [24]

    Zongxia Li, Ishani Mondal, Huy Nghiem, Yijun Liang, and Jordan Lee Boyd-Graber. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.548 PEDANTS : Cheap but effective and interpretable answer equivalence . In Findings of the Association for Computational Linguistics: EMNLP...

    2024 doi

  17. [25]

    Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. 2025. http://arxiv.org/abs/2501.02189 Benchmark evaluations, applications, and challenges of large vision language models: A survey

    2025 arXiv

  18. [26]

    Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, et al. 2024 a . Large language models and causal inference in collaboration: A comprehensive survey. arXiv preprint arXiv:2403.09606

    2024 arXiv

  19. [27]

    Xiaoyu Liu, Jiaxin Yuan, Yuhang Zhou, Jingling Li, Furong Huang, and Wei Ai. 2024 b . Csrec: Rethinking sequential recommendation from a causal perspective. arXiv preprint arXiv:2409.05872

    2024 arXiv

  20. [28]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747

    2023 arXiv

  21. [29]

    Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703--17716

    2022

  22. [30]

    OpenAI. 2023. http://arxiv.org/abs/2303.08774 Gpt-4 technical report

    2023 arXiv

  23. [31]

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786

    2023 arXiv

  24. [32]

    Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, and Frederic Sala. 2024. Pretrained hybrids with mad skills. arXiv preprint arXiv:2406.00894

    2024

  25. [33]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  26. [34]

    Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. 2024. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint arXiv:2402.14207

    2024 arXiv

  27. [35]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    2017 arXiv

  28. [36]

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. 2024. http://arxiv.org/abs/2403.07816 Branch-train-mix: Mixing expert llms into a mixture-of-experts llm

    2024 arXiv

  29. [37]

    Together Computer . 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: an open dataset for training large language models

    2023

  30. [38]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  31. [39]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491

    2024 arXiv

  32. [40]

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and Cao Xiao. 2024 a . http://arxiv.org/abs/2405.15973 Enhancing visual-language modality alignment in large vision language models via sel...

    2024 arXiv

  33. [41]

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. 2024 b . Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529

    2024 arXiv

  34. [42]

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasin...

    2022

  35. [43]

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. 2024. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415

    2024 arXiv

  36. [44]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

    2024

  37. [45]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

    2024

  38. [46]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  39. [47]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385

    2024 arXiv

  40. [48]

    Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2022. http://arxiv.org/abs/2210.05144 Mixture of attention heads: Selecting attention heads per token

    2022 arXiv

  41. [49]

    Yuhang Zhou and Wei Ai. 2024. Teaching-assistant-in-the-loop: Improving knowledge distillation from imperfect teacher models in low-budget scenarios. arXiv preprint arXiv:2406.05322

    2024 arXiv

  42. [50]

    Yuhang Zhou, Suraj Maharjan, and Beiye Liu. 2023. Scalable prompt generation for semi-supervised learning with language models. arXiv preprint arXiv:2302.09236

    2023 arXiv

  43. [51]

    Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. 2024 a . http://arxiv.org/abs/2311.08648 Explore spurious correlations at the concept level in language models for text classification

    2024 arXiv

  44. [52]

    Yuhang Zhou, Jing Zhu, Paiheng Xu, Xiaoyu Liu, Xiyao Wang, Danai Koutra, Wei Ai, and Furong Huang. 2024 b . Multi-stage balanced distillation: Addressing long-tail challenges in sequence-level knowledge distillation. arXiv preprint arXiv:2406.13114

    2024 arXiv

  45. [53]

    Jing Zhu, Yuhang Zhou, Shengyi Qian, Zhongmou He, Tong Zhao, Neil Shah, and Danai Koutra. 2024. Multimodal graph benchmark. arXiv preprint arXiv:2406.16321

    2024 arXiv

  46. [54]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.