Pith. sign in

REVIEW 5 major objections 6 minor 65 references

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grouping languages by short fine-tuning deviations, then expanding the most divergent layers into mixture-of-experts modules with one expert per language group, cuts multilingual perplexity by 11.4% on average over continual pretraining.

desk verdict Solid empirical MoE-for-multilingual paper; capacity scaling is real, but the parameter-deviation grouping mechanism is not yet nailed down. read the letter →

arxiv 2506.12388 v1 pith:24YO6YD6 submitted 2025-06-14 cs.CL cs.AI

Chong Li , Yingzhuo Deng , Jiajun Zhang , Chengqing Zong This is my paper
classification cs.CLcs.AI
keywords multilinguallargelanguagemodelscurseofmultilingualitymixture-of-expertsnegativetransfersimilarityparameterdeviationcross-lingualcontinualpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the curse of multilinguality, degraded language modeling when many languages compete in one model, can be reduced by adding capacity exactly where languages clash rather than by growing the whole model. The authors first fine-tune a base multilingual model for ten steps on each language's monolingual corpus and use the resulting layer-wise parameter deviations as a model-specific fingerprint of that language. Similar languages are grouped by these fingerprints, the layers with the largest deviations are converted into mixture-of-experts layers with one expert per group, and a router is trained to send each language's tokens to its group's expert. In experiments over 18 to 128 languages they report an average perplexity improvement of 11.4% over continued pretraining and a 9.6% gain over an ensemble-of-experts baseline while using 3.6x fewer parameters. If the method works as claimed, it points to a cheap route for scaling multilingual capacity and for adapting new languages without forgetting old ones.

What carries the argument

The load-bearing object is the parameter-deviation fingerprint $\Delta\theta^x = \theta_0 - \theta^x$, the change in the model's parameters after ten steps of fine-tuning on the monolingual corpus of language $x$. It does two jobs at once: its layer-wise norm $\|\Delta\theta^x_l\|$ identifies which layers carry language-specific knowledge (the layers with the largest deviations are the ones expanded), and its cosine similarity between languages, aggregated by the intra-group minimum $\min_{x,y \in G_k} \frac{\Delta\theta^x \cdot \Delta\theta^y}{\|\Delta\theta^x\|\|\Delta\theta^y\|}$, drives the balanced greedy clustering of similar languages. The expanded MoE layers then contain one expert per language group, with a router trained by a language-group classification loss $\mathcal{L}_{\mathrm{RC}}(\theta) = -\sum_x \sum_{i=1}^M \log P_i(l|x;\theta)$ so that tokens from a language are routed to their group's expert without any language tag at inference.

What would settle it

Train the same base model and token budget with four variants: the deviation-derived groups, equally balanced random groups, groups from typological vectors instead of fingerprints, and layers chosen by smallest rather than largest deviation. If the deviation-grouped model does not beat the random-grouped one, or if the smallest-deviation layer choice does not lag the largest-deviation choice, the fingerprint's causal role is falsified; a more direct check is to measure the correlation between fingerprint cosine similarity and per-language gradient conflict during the first epochs of continued pretraining.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that per-language parameter-deviation vectors $\Delta\theta^x$ obtained from only ten steps of monolingual fine-tuning are a stable, model-intrinsic measure of language similarity, and that clustering languages by these vectors while expanding only the top-$\epsilon$ layers into group-specialized mixture-of-experts layers reduces negative transfer between dissimilar languages. The model first uses cosine similarity between these fingerprints to partition languages into balanced groups via a greedy maximin algorithm; then the layers whose deviation norms are largest are replaced by MoE layers whose experts are initialized from the original dense layer, and each expert is later trained on the tokens of one language group under a combined language-modeling and router classification loss. The reported consequences are an 11.4% average perplexity improvement over continual pretraining, a 9.6% improvement over X-ELM with 3.6x fewer parameters, better results than the Branch-Train-Mix baseline at 128 languages, and reduced catastrophic forgetting when new languages are added by copying the expert of the most similar group and fine-tuning only that expert and the router.

Load-bearing premise

The whole grouping rests on the premise that ten steps of monolingual fine-tuning leave a stable parameter-deviation fingerprint that genuinely reflects which languages conflict with each other and which layers need more capacity; if those fingerprints do not track the real negative-transfer structure, the expert groups are arbitrary and the reported gains would not replicate.

Editorial extensions

If this is right

  • Capacity is added only where it is needed: the layers with the highest per-language deviation receive dedicated expert parameters, while the remaining layers stay shared across all languages.
  • The router learns language identity indirectly, so inference needs no explicit language identification and remains a single forward pass over the model.
  • Clustering quality matters most for low-resource languages: replacing deviation-based groups with random groups costs 1.4 perplexity points on average in the 18-language setting, with Urdu hit hardest.
  • New languages can be added by copying the expert of the most similar group and fine-tuning only that expert and the router, which keeps forgetting of previously learned languages lower than dense continual adaptation.
  • The recipe transfers across base models and corpora, with the largest gains on unseen and extremely low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the fingerprints cost ten fine-tuning steps per language, the method's practical cost grows linearly with the number of languages; transferring fingerprints computed on a smaller model to a larger one, which the paper's own cross-model clustering experiment hints at, is a natural way to test whether the grouping signal is largely model-agnostic.
  • The router classification loss makes expert specialization legible, yet one could probe whether a purely unsupervised routing objective such as an entropy penalty without language labels achieves the same specialization, which would extend the method to code-switched or unlabeled text.
  • The paper fixes the language groups once before training; an implicit open question is whether regrouping during continued pretraining would track the drift of language representations and improve the gains further.
  • A direct test of the paper's premise would be to measure whether fingerprint cosine similarity actually predicts gradient conflict between languages during the first epochs of continued pretraining, which would turn the method into a principled negative-transfer detector.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DMoE, a multilingual LM extension method that first fine-tunes a base model on monolingual data for ten steps to obtain per-language parameter-deviation vectors, uses cosine similarity of these vectors to cluster languages into groups, extends the layers with the largest deviation norms into mixture-of-experts layers with one expert per language group, and trains the router with an auxiliary language-group classification loss. Experiments on BLOOM560M, BLOOM1.7B, Gemma2B, and Qwen2.5 models, with 18 and 128 languages, report perplexity gains over continual pretraining and over X-ELM, and in-context learning improvements on five multilingual benchmarks. The central claim is that deviation-based, negative-transfer-aware grouping and layer-wise capacity allocation improve multilingual language modeling while using fewer parameters than X-ELM.

Significance. If the empirical claims hold, the paper makes a useful contribution to multilingual LLM capacity allocation: it offers a model-specific, cheap way to build language similarity from parameter deviations, and it shows that converting selected dense layers into language-group-specific MoE layers can improve perplexity and slow catastrophic forgetting. The breadth is a genuine strength: four base models, 18 and 128 language settings, held-out language splits, and ablations against random clusters, LANG2VEC clusters, and a no-classification-loss variant. The authors also state that code and model weights will be released, which aids reproducibility. The main caveat is that the central quantitative claims are not yet supported with variance estimates, and at least one headline number in the abstract/introduction is inconsistent with the reported tables, so the size and statistical reliability of the gains need tightening before the claims can be taken at face value.

major comments (5)
  1. [§4.2, Table 2] The reported '+11.4% over + Pre-train' does not match the averages in Table 2 for the model discussed in the surrounding text. For BLOOM560M the average goes from 21.6 to 19.5, a 9.7% improvement; the 11.4% figure appears only in the Gemma2B row (14.0 to 12.4). The sentence 'DMoE obtains the highest average improvement on perplexity (+11.4% over + Pre-train)' is therefore misleading, and the comparisons '+0.8% and +2.2%' do not correspond to the X-ELM and Branch-Train-Mix rows in the same table. Please report per-model percentages and state explicitly which row or average the headline refers to.
  2. [Table 2, Table 4] The main perplexity tables report a single run per configuration and no seed variance or confidence intervals. This matters because the core mechanism claim is that deviation-based grouping is better than other groupings: the DMoE (6 Groups) row is 19.5 vs. 19.8 for LANG2VEC clusters, a difference of 0.3 PPL, while the random-cluster baseline is 20.9. Without an estimate of run-to-run variability, it is not established that the model-specific grouping outperforms LANG2VEC, which is load-bearing for the 'negative-transfer-aware grouping' claim. The ± values in Table 4 are standard deviations across languages, not across independent training runs, so they do not address this. Please add at least multi-seed variance for the main comparisons, or, if that is infeasible, temper the claim and justify the cost.
  3. [§3.1 vs. Appendix C.1] The method as formalized is not the method as evaluated. Equation (1) and the surrounding text define Δθ as the concatenation of parameter deviations from all N layers, but Appendix C.1 states that 'we adopt the last 3 layers of the parameter derivation to calculate the language similarity by default.' The language grouping used in all reported experiments therefore uses only the last 3 layers, not the full vector in Eq. (1). Please either revise the formalization to specify the layer subset, or justify why the last-3-layer approximation is equivalent for the clustering objective; otherwise the paper is internally inconsistent about its own core similarity measure.
  4. [Appendix B] The stability evidence for the load-bearing parameter-deviation fingerprints is limited to within-run comparisons: Figure 8 shows that the deviation at step 10 is similar to the deviation at step 40 for a single optimization trajectory, and Figure 9 shows similar matrices across steps. There is no evidence across random seeds, data subsets, or the low-resource languages where 10-step updates are most likely to be optimizer-noise-dominated. Since the entire method depends on these fingerprints being stable and meaningful, please add a multi-seed stability analysis, or at least report clustering stability across seeds and show that the resulting language groups are not driven by initialization noise.
  5. [§3.2, Figure 5] The claim that layers with the largest parameter deviations are the right layers to expand rests on Figure 5, which compares top-ϵ expansion with a random-augmentation baseline for one model (BLOOM560M). The figure has no error bars or statistical testing, and the text reports only a single setting (ϵ=0.4) for the rest of the paper. Given that the layer-selection rule is a central design choice, please provide additional evidence for the superiority of deviation-based layer selection, ideally across more than one base model and with variance information.
minor comments (6)
  1. [Throughout] There are several typos and word-choice issues: 'statisticize' should be 'statistically analyze' or similar, 'derivation' is used where 'deviation' is meant, 'benifits' should be 'benefits', and phrases like 'the one at the 10th step' are awkward. A careful proofread would improve readability.
  2. [Table 2] The label 'w.o/ Class. Loss' in the text appears as 'w/o Class. Loss' in the table; please make the notation consistent. Also, the table header for resource categories is not clearly separated from the language columns, which makes the table harder to parse.
  3. [Figure 3 and Figure 9] The similarity heatmaps are difficult to read in grayscale print; adding numerical values or using a more perceptually uniform colormap would help.
  4. [§3.2] The sentence 'Thus the top-ϵ of dense layers and extended to the mixture-of-experts layers' is grammatically incomplete; it should be 'the top-ϵ dense layers are extended...'.
  5. [§4.3] The statement that 'the eighty languages with Latin script improved by 2.9 PPL' is not tied to a table or appendix column; please indicate where this calculation can be verified from the 128-language results.
  6. [Appendix A] The hyper-parameter α is reported as 'empirically set to 1.28', but there is no sensitivity analysis for α, even though the router classification loss is a claimed contribution; a short ablation would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed perplexity gains are held-out empirical results, and no load-bearing step reduces to fitted inputs or self-citations.

full rationale

The paper's derivation chain is empirical rather than definitional. It computes per-language parameter deviations by ten steps of monolingual fine-tuning, clusters languages by cosine similarity (Eq. 1 and Algorithm 1), converts the top-epsilon deviation layers into MoE layers, trains with a router classification loss, and measures perplexity on held-out validation splits of CulturaX and MADLAD-400. The headline improvements (11.4% over + Pre-train, 9.6% over X-ELM with fewer parameters) are measured results, not quantities forced by the construction. No equation in the paper makes the evaluation metric equal to the grouping objective or to the parameter-deviation fit. Hyperparameters K, epsilon, and alpha are selected on validation, but they are not presented as predictions; that is standard empirical tuning and does not constitute a fitted-input-called-prediction step. The claim that high-deviation layers are language-specific is explicitly stated as an assumption in Section 3.2 ('We assume that...'), so it is a support or correctness risk rather than a circular reduction. Self-citations (e.g., Wang and Zhang 2022; Li et al. 2024) appear in related work and background and are not load-bearing: the method's stability evidence is the in-paper Appendix B analysis, and the central evaluation is against external baselines and held-out data. No uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed under new coordinates. Under the required standard of exhibiting a specific reduction (Eq. X = Eq. Y by construction, or a fit renamed as prediction), no circular step can be identified. The Appendix C.1 note that 'we adopt the last 3 layers of the parameter derivation to calculate the language similarity by default' is an internal consistency discrepancy with Eq. (1), not a circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or architectural entities beyond standard MoE components. It relies on several domain assumptions about parameter deviation as a language similarity signal and about which layers benefit from expansion, plus hyperparameters selected on validation data.

free parameters (5)
  • Number of language groups K = 2, 3, 6, 9 (18 languages); 16 (128 languages)
    Chosen by comparing validation perplexity; more groups generally helped.
  • Layer expansion fraction epsilon = 0.4
    Set from ablation in Figure 5 balancing perplexity and parameter count.
  • Router classification loss weight alpha = 1.28
    Empirically set, Appendix A.
  • Monolingual fine-tuning steps for deviation = 10
    Chosen based on cosine-similarity stability analysis in Appendix B.
  • Number of layers used for similarity = last 3 layers
    Default in Appendix C.1 despite Section 3.1 using all layers.
assumptions (5)
  • domain assumption Parameter deviation after short monolingual fine-tuning captures language-specific knowledge and similarity relations.
    Used to cluster languages and select layers; Section 3.1 and Appendix B.
  • domain assumption Layers with larger parameter deviation are the ones that need extra capacity.
    Stated as an assumption in Section 3.2; supported only by ablations, not derived.
  • domain assumption Cosine similarity over concatenated parameter deviations is an appropriate metric for language grouping.
    Equation 1; no principled justification beyond empirical comparison with LANG2VEC and random.
  • standard math Greedy balanced clustering approximates the NP-hard maximin objective well enough.
    Algorithm 1; no approximation guarantee claimed.
  • domain assumption Language grouping transfers across model sizes within a model family.
    BLOOM560M uses BLOOM1.7B's grouping in the 128-language experiment (Appendix C.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model." pith.science (2026). https://pith.science/paper/24YO6YD6

@misc{pith2026250612388,
  author       = {Pith},
  title        = {Pith review of: Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24YO6YD6}},
  note         = {Machine review of arXiv:2506.12388}
}
read the original abstract

The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior performance. It mainly comes from limited capacity and negative transfer between dissimilar languages. To address this issue, we propose a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages. Specifically, the model is first tuned on monolingual corpus to determine the parameter deviation in each layer and quantify the similarity between languages. Layers with more deviations are extended to mixture-of-experts layers to reduce competition between languages, where one expert module serves one group of similar languages. Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters. Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned.

Figures

Figures reproduced from arXiv: 2506.12388 by the authors.

Figure 1
Figure 1. (a) We first statisticize layer-wise parameter [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our method to group and scale up the multilingual LLM. (a) Given pre-training languages, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. The improvement of DMoE comparing to the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: The average perplexity of DMoE across 18 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: The average perplexity of BLOOM560M across language families after training on the 128 languages. In-context learning results on five multilingual datasets are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: illustrates the cosine similarity of the pa￾rameter deviation during fine-tuning. It can be found that the deviation is relatively small after 10 tuning steps, and the cosine similarity of the one at the 10th step between the parameter deviation at the 40th step is hig…
Figure 9
Figure 9. Figure 9: The language similarity matrices calculated by the parameter derivation at different fine-tuning steps using [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The distribution of parameter deviation ∥∆θ x∥ across layers of BLOOM560M for 18 languages [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: The cosine similarity between 18 languages [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: The cosine similarity between 18 languages [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: The cosine similarity between 18 languages [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 15
Figure 15. Figure 15: The router distribution of top-1 expert for texts in different languages on models trained with randomly [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: The router distribution of top-1 expert for texts in different languages on models trained with router [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 22 canonical work pages

  1. [1]

    Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. https://doi.org/10.18653/v1/N19-1388 Massively multilingual neural machine translation . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3874--3884, Minneapolis, M...

    doi:10.18653/v1/n19-1388 2019

  2. [2]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, and et al

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, and et al. 2023. https://arxiv.org/abs/2305.10403 Palm 2 technical report . Preprint, arXiv:2305.10403

    arXiv 2023

  3. [3]

    Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.100 BUFFET : Benchmarking large language models for few-shot cross-lingual transfer . In Proceedings of the 2024 Conference of the North American Chapter of the Associat...

    doi:10.18653/v1/2024.naacl-long.100 2024

  4. [4]

    Smith, and Luke Zettlemoyer

    Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. https://aclanthology.org/2024.emnlp-main.604 Breaking the curse of multilinguality with cross-lingual expert language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10822--1083...

    work page 2024

  5. [5]

    Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen

    Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. https://aclanthology.org/2024.emnlp-main.236 When is multilinguality a curse? language modeling for 250 high- and low-resource languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074--4096, Miami, Florida, USA. Association for Comp...

    work page 2024

  6. [6]

    Ronan Collobert, Samy Bengio, and Yoshua Bengio. 2001. https://proceedings.neurips.cc/paper_files/paper/2001/file/36ac8e558ac7690b6f44e2cb5ef93322-Paper.pdf A parallel mixture of svms for very large scale problems . In Advances in Neural Information Processing Systems, volume 14. MIT Press

    work page 2001

  7. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    doi:10.18653/v1/2020.acl-main.747 2020

  8. [8]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475--2485, Brussels, Belgium. Association...

    doi:10.18653/v1/d18-1269 2018

Show all 65 references
  1. [9]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. https://arxiv.org/abs/2401.06066 Deepseekmoe: Towards ultimate...

    2024 arXiv

  2. [10]

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. https://doi.org/10.18653/v1/2022.acl-long.489 S table M o E : Stable routing strategy for mixture of experts . In Proceedings of the 60th Annual Meeting of the Association for Computationa...

    2022 doi

  3. [11]

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztaj...

    2024 arXiv

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  5. [13]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  6. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  7. [15]

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79--87

    1991

  8. [16]

    Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and Barry Haddow. 2025. https://arxiv.org/abs/2409.17892 Emma-500: Enhancing massively multilingual adaptation of large language model...

    2025

  9. [17]

    Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean

    Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. https://doi.org/10.1162/tacl_a_00065 G oogle ' s multilingual neural machine translat...

    2017 doi

  10. [18]

    Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181--214

    1994

  11. [19]

    Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. https://openreview.net/forum?id=Y45ZCxslFx MADLAD -400: A multilingual and document-level large audited dataset . In Thirty-seventh...

    2023

  12. [20]

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.28 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback . In Pro...

    2023 doi

  13. [21]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. https://openreview.net/forum?id=qrwe7XHTmYb \ GS \ hard: Scaling giant models with conditional computation and automatic sharding . In Int...

    2021

  14. [22]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translatio...

    2020 doi

  15. [23]

    Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.445 Improving in-context learning of multilingual generative language models with cross-lingual alignment . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  16. [24]

    Smith, and Luke Zettlemoyer

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022. https://arxiv.org/abs/2208.03306 Branch-train-merge: Embarrassingly parallel training of expert language models . Preprint, arXiv:2208.03306

    2022 arXiv

  17. [25]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O ' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, M...

    2022 doi

  18. [26]

    Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin

    Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. https://aclanthology.org/E17-2002 URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors . In Proceedings of the 15th Conference of ...

    2017

  19. [27]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  20. [28]

    Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. https://doi.org/10.18653/v1/D17-1268 Learning language representations for typology prediction . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2529--2535, Copenhage...

    2017 doi

  21. [29]

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed precision training. In International Conference on Learning Representations

    2018

  22. [30]

    Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. https://arxiv.org/abs/2412.0400...

    2024 arXiv

  23. [31]

    MistralAI. 2023. https://mistral.ai/news/mixtral-of-experts/ Mixtral of experts . Mistral AI news

    2023

  24. [32]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. https://aclanthology.org/2024.lrec-main.377 C ultura X : A cleaned, enormous, and multilingual dataset for large language models in 167 langu...

    2024

  25. [33]

    O stling and J \

    Robert \"O stling and J \"o rg Tiedemann. 2017. https://aclanthology.org/E17-2102 Continuous multilinguality with language vectors . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pages ...

    2017

  26. [34]

    Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. https://doi.org/10.18653/v1/2022.naacl-main.255 Lifting the curse of multilinguality by pre-training modular transformers . In Proceedings of the 2022 Conference of the North ...

    2022 doi

  27. [35]

    Edoardo Maria Ponti, Goran Glava s , Olga Majewska, Qianchu Liu, Ivan Vuli \'c , and Anna Korhonen. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.185 XCOPA : A multilingual dataset for causal commonsense reasoning . In Proceedings of the 2020 Conference on Empirical Method...

    2020 doi

  28. [36]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/blog/language-unsupervised/ Improving language understanding by generative pre-training . OpenAI blog

    2018

  29. [37]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  30. [38]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. https://doi.org/10.1145/3394486.3406703 Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters . In Proceedings of the 26th ACM SIGKDD International Conferenc...

    2020

  31. [39]

    Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E Weston. 2021. https://openreview.net/forum?id=lMgDDWb1ULW Hash layers for large sparse models . In Advances in Neural Information Processing Systems

    2021

  32. [40]

    BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, and Alexandra Sasha Luccioni et al. 2023. https://arxiv.org/abs/2211.05100 Bloom: A 176b-parameter open-access multilingual language model . arXiv pre...

    2023 arXiv

  33. [41]

    Babak Shahbaba and Radford Neal. 2009. Nonlinear models using dirichlet process mixtures. Journal of Machine Learning Research, 10(8)

    2009

  34. [42]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. https://openreview.net/forum?id=B1ckMDqlg Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . In International Conference on Learning ...

    2017

  35. [43]

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wen Li, Wen tau Yih, Jason E Weston, and Xian Li. 2024. https://openreview.net/forum?id=nqLAuMOF6n Branch-train-mix: Mixing expert LLM s into a mixture-of-experts LLM...

    2024

  36. [44]

    Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2020. https://doi.org/10.18653/v1/2020.coling-main.318 Distill and replay for continual language learning . In Proceedings of the 28th International Conference on Computational Linguistics, pages 3569--3579, Barcelo...

    2020 doi

  37. [45]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and Pouya Tafti et al. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technol...

    2024 arXiv

  38. [46]

    Lucas Theis and Matthias Bethge. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/2b6d65b9a9445c4271ab9076ead5605a-Paper.pdf Generative image modeling using spatial lstms . In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc

    2015

  39. [47]

    Alexey Tikhonov and Max Ryabinin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.310 I t ' s A ll in the H eads: U sing A ttention H eads as a B aseline for C ross- L ingual T ransfer in C ommonsense R easoning . In Findings of the Association for Computational Linguistic...

    2021 doi

  40. [48]

    Volker Tresp. 2000. https://proceedings.neurips.cc/paper_files/paper/2000/file/9fdb62f932adf55af2c0e09e55861964-Paper.pdf Mixtures of gaussian processes . In Advances in Neural Information Processing Systems, volume 13. MIT Press

    2000

  41. [49]

    Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016. https://doi.org/10.18653/v1/N16-1161 Polyglot neural language models: A case study in cross-lingual phonetic representation lea...

    2016 doi

  42. [50]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. ht...

    2024 doi

  43. [51]

    Qian Wang and Jiajun Zhang. 2022. https://doi.org/10.1609/AAAI.V36I10.21396 Parameter differentiation based multilingual neural machine translation . In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of A...

    2022 doi

  44. [52]

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L \"u , Rui Hu, et al. 2023 a . Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341

    2023 arXiv

  45. [53]

    Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023 b . https://arxiv.org/abs/2307.06018 Polylm: An open so...

    2023 arXiv

  46. [54]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://doi.org/10.18653/v1/2024.acl-long.820 Do llamas work in E nglish? on the latent language of multilingual transformers . In Proceedings of the 62nd Annual Meeting of the Association for Computati...

    2024 doi

  47. [55]

    Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023. https://doi.org/10.18653/v1/2023.findings-acl.48 Overcoming catastrophic forgetting in massively multilingual continual learning . In Fin...

    2023 doi

  48. [56]

    Junhong Wu, Yuchen Liu, and Chengqing Zong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.398 F - MALLOC : Feed-forward memory allocation for continual learning in neural machine translation . In Proceedings of the 2024 Conference of the North American Chapter of the Assoc...

    2024 doi

  49. [57]

    Shijie Wu and Mark Dredze. 2020. https://doi.org/10.18653/v1/2020.repl4nlp-1.16 Are all languages created equal in multilingual BERT ? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120--130, Online. Association for Computational Linguistics

    2020 doi

  50. [58]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  51. [59]

    Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. https://arxiv.org/abs/2305.18098 Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages . Preprint, arXiv:2305.18098

    2023 arXiv

  52. [60]

    Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. https://doi.org/10.18653/v1/D19-1382 PAWS - X : A cross-lingual adversarial dataset for paraphrase identification . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the ...

    2019 doi

  53. [61]

    Yang Zhao, Junnan Zhu, Lu Xiang, Jiajun Zhang, Yu Zhou, Feifei Zhai, and Chengqing Zong. 2022. Life-long learning for multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:2212.02800

    2022 arXiv

  54. [62]

    Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. 2025. https://doi.org/10.1609/AAAI.V39I24.34805 Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing . In ...

    2025 doi

  55. [63]

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf Mixture-of-experts with expert...

    2022

  56. [64]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  57. [65]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.