Pith. sign in

REVIEW 5 major objections 5 minor 40 references

Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LayerMoE claims that allocating new MoE experts per layer in inverse proportion to cross-lingual hidden-state similarity lets an LLM expand to new languages with 60% fewer added parameters while better preserving old ones.

desk verdict A plausible parameter-efficient multilingual expansion method with an untested core heuristic and thin empirical margins; worth refereeing but in need of major revisions. read the letter →

arxiv 2505.22582 v1 pith:MUSCOAMF submitted 2025-05-28 cs.CL

Xue Zhang , Yunlong Liang , Fandong Meng , Songming Zhang , Yufeng Chen , Jinan Xu , Jie Zhou This is my paper
classification cs.CL
keywords layer-wiseexpertallocationmixture-of-expertsmultilingualLLMexpansioncatastrophicforgettingcross-lingualhiddenstatesimilarityMoE-LPRlifelonglanguageparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that when an LLM is expanded with new languages by adding Mixture-of-Experts (MoE) layers, the number of new experts should vary by layer, and the layer-wise signal for how many to add is how similar the hidden states of different languages are after the attention module. It proposes LayerMoE, which measures pairwise cosine similarity of hidden-state activations at each layer and allocates experts in inverse proportion to that similarity, giving more experts where languages look most different and fewer where representations already overlap. It also inserts a small classifier before the router on the most similar layers to send old-language tokens to the original expert, mitigating forgetting. On Qwen1.5-1.8B and Llama-3.2-3B, the paper reports that this beats the uniform-allocation MoE-LPR baseline with 60% fewer added experts in a single expansion and 33.3% fewer in lifelong expansion. If right, this makes continual multilingual expansion cheaper and safer for existing models.

What carries the argument

The load-bearing mechanism is the per-layer indicated similarity $S^i$, computed as the average pairwise cosine similarity of hidden states after the attention module between sampled token sets of the new and old language groups (and within the new group). The allocation formula $N^i = \lceil ( (S^i)^{-1} / \sum_j (S^j)^{-1} ) \cdot \delta \rceil$ distributes a total expert budget $\delta$ inversely to similarity, so high-similarity layers get few new experts and low-similarity layers get many. On the top-$K$ layers by new-old similarity, a linear classifier $W_c \in \mathbb{R}^{h \times 2}$ in front of the router decides whether a token belongs to an old language and, if so, routes it to expert 0.

What would settle it

Compute, per layer and with the total expert budget held fixed, the marginal gain in new-language benchmark score from adding one expert, and correlate that gain with the layer's measured new-old hidden-state similarity; if higher-similarity layers show equal or larger marginal gains, the inverse-allocation premise is false. A cheaper check: compare LayerMoE's allocation against a random allocation with the same total budget and the same top-$K$ classifier placement, repeated over several seeds; if the random allocation matches it, the similarity signal contributes nothing.

Watch

Extended reading notes

Core claim

The paper's central claim is that representation similarity between languages at each layer is a usable predictor of how much fresh capacity that layer needs when a language is added. The authors compute, per layer, the average cosine similarity between randomly sampled hidden states of old and new languages, call this the indicated similarity, and set the number of new experts proportional to the inverse of that similarity under a fixed total budget. They also find that layers with higher new-old similarity are exactly where the router gets confused, so they train a binary classifier there that routes old-language tokens straight to the frozen old expert. The reported experiments show that this scheme outperforms the previous method on both old-language retention and new-language acquisition while using far fewer added parameters.

Load-bearing premise

The load-bearing premise is that cross-lingual similarity of hidden states after attention is a valid inverse signal for how many new experts a layer needs, with no other factor — such as the total expert budget, the choice of which layers receive the classifier, or the training data size — doing the real work.

Editorial extensions

If this is right

  • At a fixed total number of added experts, similarity-based per-layer allocation beats uniform allocation on both old-language preservation and new-language acquisition.
  • Adding the router-front classifier on high-similarity layers reduces old-language forgetting beyond what language-prior routing alone achieves.
  • The method transfers to a different base model (Llama-3.2-3B) and to a generative task (machine translation on FLORES), not just multiple-choice benchmarks.
  • In lifelong expansion, LayerMoE stays ahead with one-third fewer total experts, and the order in which language groups are added changes which languages gain most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The inverse-similarity rule reallocates a fixed parameter budget to layers where representations are most language-specific; an implication the paper leaves implicit is that the advantage over uniform allocation should shrink as the total budget grows very large, since capacity ceases to be the binding constraint.
  • The offline similarity computation could be reused as a prior for language pairs: the same vector of per-layer similarities might predict good allocations for other base models of the same family without retraining.
  • Since the classifier helps most on high-similarity layers, a testable extension is whether a single language-identification head shared across layers, or a head trained on new-new pairs as well, further improves routing in lifelong settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes LayerMoE, a method for expanding an existing LLM with new languages by adding layer-wise MoE experts. The method first measures cosine similarity between hidden states of old and new languages at each layer, then allocates more experts to layers with lower similarity and fewer to layers with higher similarity (Eq. 10). On high-similarity layers, a classifier is inserted before the router to route old-language tokens to the frozen old expert. Experiments on Qwen1.5-1.8B and Llama-3.2-3B across single-expansion and lifelong-expansion settings report that LayerMoE outperforms MoE-LPR baselines while using fewer added parameters, and the authors include ablations showing degradation under random expert allocation and without the classifier.

Significance. If the results hold, LayerMoE is a practically valuable contribution to parameter-efficient multilingual expansion: it reduces the parameter cost of adding languages and improves old-language retention compared to a strong MoE baseline. The paper ships code and reports experiments on four benchmarks, two models, and two expansion settings, with an ablation study and an honest limitations section. However, the significance is tempered by the small size of the reported gains, the absence of variance estimates, and the fact that the central mechanism—that HSA similarity should drive expert allocation in the specific inverse direction—is not directly tested.

major comments (5)
  1. [§3.2, Eq. (10); §5.2, Table 3] The central directional claim—higher HSA similarity requires fewer new experts—is not tested. The ablation 'w/random' replaces the similarity-based allocation with a random schedule, but this only varies the allocation pattern while preserving non-uniformity; it does not control for the direction of the relation. A uniform baseline such as MoE-LPR(3*24) is also not a directional control. To establish that Eq. (10)'s inverse relation, rather than merely non-uniformity or the presence of a classifier, drives the gains, the paper should include a reversed allocation (N_i proportional to S_i), an allocation using a shuffled similarity vector, or an allocation transferred from a different language pair. Without such a counterfactual, the reported gains could equally be attributed to the tuned non-uniform schedule and classifier placement.
  2. [§4.1; Table 4] The number of classifier layers K (top-7 for single-expansion, top-5 for lifelong-expansion) is selected on the same evaluation benchmarks used for the headline results, with the text stating that these settings are taken 'according to the best performance.' Table 4 shows that the differences among top-6, top-7, and top-8 (avgs 40.23, 40.26, 40.17) are very small, and the paper reports no multiple seeds or error bars anywhere. The headline improvements compared to MoE-LPR (e.g., 0.19/0.44 points in New/Old-avg for G1) could therefore partly reflect selection noise. The authors should report seed variance or, better, select K on a validation set and evaluate on a held-out test set.
  3. [Abstract; Tables 1 and 2] The abstract's claim of '60% fewer experts' and '33.3% fewer experts' is inaccurate. The tables report parameter reductions (1.6B vs 4B = 60% parameter reduction; 3.2B vs 4.8B = 33.3% parameter reduction), but the corresponding expert counts are 72 vs 144 (50% fewer) and 144 vs 168 (14.3% fewer) under the paper's δ=72-per-expansion allocation. The abstract and the 'fewer experts' phrasing should be corrected to refer to parameter counts, or the expert-count percentages should be computed correctly. This matters because the title and headline claim are explicitly about being parameter-efficient.
  4. [§3.1, Eq. (8)] The description of the similarity computation is not reproducible as written. The paper says Q=100,000 tokens are sampled per language and that similarity is computed as the average over all Q^2 pairs. With Q=10^5, this is 10^10 cosine computations per layer per language pair, which is computationally infeasible for the 24-layer model and multiple language pairs described. Please clarify whether pairs are subsampled, whether the mean HSA vector is used, or whether some other approximation is applied.
  5. [§5.4, Table 5] The generalization study on Llama-3.2-3B covers only one setting (G0→G1) and compares only against MoE-LPR(3*24), with no ablation of the allocation or classifier components. This supports the claim that the overall pipeline transfers, but it does not provide evidence that the similarity direction or the classifier design generalizes. The text should be scoped accordingly rather than claiming broad cross-model effectiveness.
minor comments (5)
  1. [§3.3, Eq. (13)] In Eq. (13), cls(x) is defined as argmax(x·Wr), but the classifier network is Wc; it should be argmax(x·Wc).
  2. [§3.3 and §3.2] The text says the classifier placement uses S_new&old computed 'following Eq.(9)', but Eq.(9) defines S as the average of S_new&old and S_new&new. Please clarify whether the classifier uses only the new-old component or the averaged value.
  3. [§4.1, lifelong-expansion] For lifelong expansion, it is not explicitly stated whether L_old at each step includes all previously acquired languages (G0 plus earlier groups) or only the original G0. This affects Eq.(9) and should be clarified.
  4. [§4.1] The text refers to the 'lm-evaluation-hardness' framework, which appears to be a typo for 'lm-evaluation-harness.'
  5. [§4.2] The sentence 'Gao et al. proposes MOLA' is missing the citation marker; it should cite (Gao et al., 2024a) at the first mention of MOLA.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; disclosed budget choices and a non-load-bearing self-citation keep this at the low end.

full rationale

After walking the derivation chain, I find no step where a claimed 'prediction' is definitionally equal to its inputs. Eq. (10) maps measured similarities to per-layer expert counts by construction (inverse proportionality), but the paper presents this as the proposed allocation rule, not as an empirical finding derived from the data; the empirical content lies in the downstream benchmark comparison, the same-budget comparison to MoE-LPR(3*24), and the random-allocation ablation. Section 4.3 sets delta=72 to match MoE-LPR(3*24), so the headline '60% fewer parameters' relative to MoE-LPR(6*24) is an arithmetic consequence of the chosen budget rather than a discovered property; the Limitations section explicitly discloses this. The top-7/top-5 classifier placement is chosen by best performance on the same evaluation benchmarks, which is a selection and overfitting risk, not an equation-level circularity. The self-citation of Zhang et al. (2024b) for language-agnostic/specific layers is present but not load-bearing: the same claim is also supported by Tang et al. (2024), and the central results are independently benchmarked. The absence of a reversed-allocation control is a hypothesis-testing gap, not circularity. Overall, the central claim retains independent empirical content; the main caveats are disclosed by-construction budget choices and test-set selection.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on the similarity-to-expert-need hypothesis, the frozen-expert preservation assumption, and a tuned layer count. The total expert budget delta and K are the main fitted choices; no invented physical or architectural entities beyond the new classifier head are introduced.

free parameters (3)
  • delta (target total new experts per expansion) = 72 for Qwen single-expansion (1.6B params)
    Set by hand to match MoE-LPR (3*24) for a fair comparison, not chosen by optimization; the Limitations section admits the optimal total was not explored. This controls the claimed 60% and 33.3% parameter-reduction percentages.
  • K (number of high-similarity layers with routing classifier) = 7 for single-expansion, 5 for lifelong-expansion
    Selected according to best performance on the same evaluation benchmarks used in the main tables (Section 4.1 and Table 4), so it is fitted to the reported result rather than fixed a priori.
  • gamma and beta loss weights = 0.1
    beta follows MoE-LPR and gamma is set equal to beta. No sensitivity analysis is reported for the classifier loss weight.
assumptions (3)
  • domain assumption Per-layer hidden-state cosine similarity between languages is monotonically indicative of the number of new experts needed, with higher similarity requiring fewer experts.
    Stated as a hypothesis in Section 3.1 and operationalized in Eq. (10). Only indirect evidence is the random-allocation ablation; no mechanistic or counterfactual control is given.
  • domain assumption Routing old-language tokens to the original frozen expert (expert 0) preserves old-language capabilities, including after multiple expansion rounds.
    Inherited from MoE-LPR stage-2 and applied by the classifier in Eq. (13). The paper does not test whether expert 0 alone can retain all old languages in lifelong expansion.
  • domain assumption A binary classifier trained on 50K old-language and 100K new-language examples generalizes well enough to hard-route tokens at inference.
    The hard-routing decision in Eq. (13) assumes classifier accuracy on unseen tokens in both old and new languages; no classifier accuracy or error analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts." pith.science (2026). https://pith.science/paper/MUSCOAMF

@misc{pith2026250522582,
  author       = {Pith},
  title        = {Pith review of: Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MUSCOAMF}},
  note         = {Machine review of arXiv:2505.22582}
}
read the original abstract

Continually expanding new languages for existing large language models (LLMs) is a promising yet challenging approach to building powerful multilingual LLMs. The biggest challenge is to make the model continuously learn new languages while preserving the proficient ability of old languages. To achieve this, recent work utilizes the Mixture-of-Experts (MoE) architecture to expand new languages by adding new experts and avoid catastrophic forgetting of old languages by routing corresponding tokens to the original model backbone (old experts). Although intuitive, this kind of method is parameter-costly when expanding new languages and still inevitably impacts the performance of old languages. To address these limitations, we analyze the language characteristics of different layers in LLMs and propose a layer-wise expert allocation algorithm (LayerMoE) to determine the appropriate number of new experts for each layer. Specifically, we find different layers in LLMs exhibit different representation similarities between languages and then utilize the similarity as the indicator to allocate experts for each layer, i.e., the higher similarity, the fewer experts. Additionally, to further mitigate the forgetting of old languages, we add a classifier in front of the router network on the layers with higher similarity to guide the routing of old language tokens. Experimental results show that our method outperforms the previous state-of-the-art baseline with 60% fewer experts in the single-expansion setting and with 33.3% fewer experts in the lifelong-expansion setting, demonstrating the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2505.22582 by the authors.

Figure 1
Figure 1. The similarity of HSAs in different languages [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The procedure of our LayerMoE. In Step 1, we first calculate the indicated similarity for each layer. In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The expert allocation of our method and base [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The detailed similarity and expert allocation of our method for each layer of Qwen1.5-1.8B under the five [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: The detailed similarity and expert allocation of our method for each layer of Llama-3.2-3B under the G0 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 5 canonical work pages

  1. [1]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.614 Do all languages cost the same? tokenization in the era of commercial language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904--9923, ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and...

    arXiv 2024

  3. [3]

    Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L \"u bbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. 2024. https:...

    doi:10.18653/v1/2024.findings-naacl.247 2024

  4. [4]

    Catherine Arnett and Benjamin K. Bergen. 2024. https://arxiv.org/abs/2411.14198 Why do language models perform worse for morphologically complex languages? Preprint, arXiv:2411.14198

    work page Pith review arXiv 2024

  5. [5]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://doi.org/10.18653/v1/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of t...

    doi:10.18653/v1/2024.acl-long.44 2024

  6. [6]

    Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. https://arxiv.org/abs/2308.13198 Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons . Preprint, arXiv:2308.13198

    arXiv 2023

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    arXiv 2018

  8. [8]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. https://arxiv.org/abs/2312.09979 Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin . Preprint, arXiv:2312.09979

    arXiv 2024

Show all 40 references
  1. [9]

    Kawin Ethayarajh. 2019. https://doi.org/10.18653/v1/D19-1006 How contextual are contextualized word representations? C omparing the geometry of BERT , ELM o, and GPT -2 embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and t...

    2019 doi

  2. [10]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. https://arxiv.org/abs/2101.03961 Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity . Preprint, arXiv:2101.03961

    2022 arXiv

  3. [11]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. 2024 a . https://arxiv.org/abs/2402.08562 Higher layers need more lora experts . Preprint, arXiv:2402.08562

    2024 arXiv

  4. [12]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  8. [17]

    Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2025. https://arxiv.org/abs/2405.10936 A survey on large language models with multilingualism: Recent advances and...

    2025 arXiv

  9. [18]

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-demo.28 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback . In ...

    2023 doi

  10. [19]

    Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023 b . https://doi.org/10.18653/v1/2023.findings-emnlp.878 C hat GPT beyond E nglish: Towards a comprehensive evaluation of large language models in multilingual l...

    2023 doi

  11. [20]

    Tianhao Li, Shangjie Li, Binbin Xie, Deyi Xiong, and Baosong Yang. 2024. https://arxiv.org/abs/2407.00875 Moe-ct: A novel approach for large language models training with resistance to catastrophic forgetting . Preprint, arXiv:2407.00875

    2024 arXiv

  12. [21]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024 a . https://aclanthology.org/2024.lrec-main.377 C ultura X : A cleaned, enormous, and multilingual dataset for large language models in 167 la...

    2024

  13. [22]

    Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024 b . https://arxiv.org/abs/2312.00738 Seallms...

    2024 arXiv

  14. [23]

    Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. https://arxiv.org/abs/2305.15425 Language model tokenizers introduce unfairness between languages . Preprint, arXiv:2305.15425

    2023 arXiv

  15. [24]

    Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1141 A lpha L o RA : Assigning L o RA experts based on layer training quality . In Proceedings of the 2024 Conference on Empirical Metho...

    2024 doi

  16. [25]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  17. [26]

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.243 How good is your tokenizer? on the monolingual performance of multilingual language models . In Proceedings of the 59th Annual Meeting of the ...

    2021 doi

  18. [27]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  19. [28]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. https://doi.org/10.18653/v1/2024.acl-long.309 Language-specific neurons: The key to multilingual capabilities in large language models . In Proceedings of the 62nd...

    2024 doi

  20. [29]

    NLLB Team. 2022. No language left behind: Scaling human-centered machine translation

    2022

  21. [30]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen1.5/ Introducing qwen1.5

    2024

  22. [31]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. ht...

    2024 doi

  23. [32]

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanq...

    2023 arXiv

  24. [33]

    Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023 b . https://arxiv.org/abs/2307.06018 Polylm: An open so...

    2023 arXiv

  25. [34]

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. https://arxiv.org/abs/2401.02415 Llama pro: Progressive llama with block expansion . Preprint, arXiv:2401.02415

    2024 arXiv

  26. [35]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  27. [36]

    Shaolei Zhang, Kehao Zhang, Qingkai Fang, Shoutao Guo, Yan Zhou, Xiaodong Liu, and Yang Feng. 2024 a . https://arxiv.org/abs/2411.16300 Bayling 2: A multilingual large language model with efficient language alignment . Preprint, arXiv:2411.16300

    2024 arXiv

  28. [37]

    Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2024 b . https://arxiv.org/abs/2406.16416 Multilingual knowledge editing with language-agnostic factual neurons . Preprint, arXiv:2406.16416

    2024 arXiv

  29. [38]

    Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.603 Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages . In Pr...

    2024 doi

  30. [39]

    Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. 2024. https://arxiv.org/abs/2408.11396 Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing . Preprint, ar...

    2024 arXiv

  31. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.