Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Expert merging for MoE-VLMs should follow phase-conditioned roles, not global routing aggregation; RoleMerge demonstrates the gains.

desk verdict Phase-aware merging is a real idea with careful experiments, but the Tiny InfoVQA cells in Table 1 are implausible and need fixing before the headline numbers can be trusted. read the letter →

arxiv 2608.04454 v1 pith:OGK4TIGE submitted 2026-08-05 cs.CV cs.LG

Hongyu Zhang , Cheng Yan , Xiang Xia , Wuyang Zhang This is my paper
classification cs.CVcs.LG
keywords mixture-of-expertsvision-languagemodelsexpertmergingmodelcompressionroutingstatisticsphase-conditionedrolestraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-experts vision-language models keep every expert in memory even though each token activates only a few, so shrinking the expert pool is a real deployment gain. This paper argues that the usual ground for choosing which experts to merge—aggregating routing statistics over all tokens—obscures the phase structure of multimodal inference, where image-context, question, and answer tokens route differently and image tokens dominate the counts. RoleMerge instead gives each expert a phase-normalized Routing Role Profile and merges experts with compatible profiles while protecting answer-decoding traffic, all without retraining. Across three models and six benchmarks, it retains more of the full model's macro-average than alternative merging methods at matched expert-retention ratios, with relative gains up to 9.6 percent.

What carries the argument

The load-bearing object is the Routing Role Profile (RRP): for each expert, the conditional distribution over the three inference phases, computed from phase-normalized routing counts with a uniform phase prior. It converts raw routing traces into a description of what each expert does during image-context processing, question processing, and answer decoding. The expert–phase information loss—the KL-divergence-based drop in mutual information when two experts are merged—turns role differences into a merge score, and the answer-aware score adds a signed collision term that preserves answer-decoding expert distinctions.

What would settle it

Re-run the matched comparisons after deliberately shifting the phase boundary—for example, counting the first decoded token as part of the question phase—and check whether RoleMerge's advantage over global aggregation survives; if a small boundary shift erases or reverses the gain, the reported benefit depends on the specific partition rather than on phase-conditioned expert roles.

Watch

Extended reading notes

Core claim

The central claim is that merge compatibility in MoE-VLMs should be judged by phase-conditioned expert roles, not by globally aggregated routing statistics. RoleMerge encodes each expert's role as a Routing Role Profile (RRP), the conditional distribution over the three inference phases (image-context, question, answer) under a uniform phase prior, and measures the cost of merging two experts as the induced decrease in expert–phase mutual information, $d_{EP,\ell}(i,j) = P_\ell(i)D_{KL}(q_{\ell,i}\|m_{\ell,ij}) + P_\ell(j)D_{KL}(q_{\ell,j}\|m_{\ell,ij})$. A signed collision adjustment built from answer-decoding routing mass delays merges that would collapse distinct experts carrying concentrated answer-generation traffic. Once the grouping is fixed, expert parameters and the corresponding router entries are averaged with the same within-group weights, so the compression is achieved without retraining either the experts or the router.

Load-bearing premise

The method's correctness rests on the assumption that the deterministic token partition into image-context, question, and answer phases (Appendix B.1) correctly identifies the functional phases of MoE-VLM inference, and that routing traces from the TCS calibration set are representative of routing on the six evaluation benchmarks.

Editorial extensions

If this is right

  • At every evaluated expert-retention ratio, RoleMerge retains more of the full model's six-task macro-average than all comparison methods in all nine model–retention settings, with relative gains up to 9.6 percent.
  • Because RoleMerge merges only routed expert parameters and router output entries, the visual encoder, attention modules, shared parameters, and decoding interface stay fixed, so the compression is training-free.
  • The ablations show that both phase separation and the correspondence between each expert and its own profile are load-bearing: pooling phases or permuting profiles degrades answer-carrier recoverability and multimodal accuracy.
  • On the largest model, Qwen3-VL-30B-A3B-Instruct at retention .625, the method cuts routed-expert parameters by 37.5 percent and total bf16 storage by 35.0 percent while keeping 91.9 percent of the full model's macro-average.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same phase-conditioned profiles could be repurposed for expert pruning: experts whose routing concentrates in low-token phases such as answer decoding are prime pruning candidates, and phase-conditioned saliency could plausibly beat activation-based saliency.
  • Because the paper focuses on merge compatibility, combining expert–phase information loss with parameter-conflict-aware fusion (for example subspace- or curvature-based) is a natural next step that directly addresses the acknowledged limitation of parameter interference inside merged groups.
  • The phase partition is itself a modeling choice; a stronger stress test would vary the boundary between question and answer tokens and check whether the reported gains are stable, since the paper's own calibration audit shows source identity matters as much as source count.
  • The gains are measured on fixed QA-style benchmarks with greedy decoding; applying RoleMerge to open-ended or multilingual generation, where phase structure is less clean, would require re-examining the phase definitions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper argues that training-free expert merging in mixture-of-experts vision-language models should be based on phase-conditioned expert roles rather than globally aggregated routing statistics. The authors propose RoleMerge, which constructs a Routing Role Profile for each expert from phase-normalized routing counts over image-context, question, and answer phases, assesses merge compatibility via an expert–phase information loss, and augments the merge score with a signed collision adjustment to preserve answer-decoding distinctions. Expert and router parameters are then merged with shared within-group weights. Experiments on DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and Qwen3-VL-30B-A3B-Instruct across six benchmarks and three retention ratios report that RoleMerge retains more macro-average performance than several existing methods, with relative improvements up to 9.6%.

Significance. The proposed reformulation is well-motivated and the implementation is careful: matched baselines under a common protocol, paired bootstrap confidence intervals for two of the three models, component ablations, calibration-source/size/composition analyses, and a deterministic algorithm. If the reported results are correct, the paper provides a simple, training-free improvement to MoE-VLM compression and evidence that phase structure is a relevant signal for expert specialization. The main weakness is the lack of released code/data and the fact that one set of reported scores (InfoVQA on DeepSeek-VL2-Tiny) appears internally inconsistent and is not covered by uncertainty quantification; this must be resolved before the empirical claims can be fully credited.

major comments (3)
  1. [Table 1 (Sec. 5.2)] RoleMerge's reported InfoVQA scores on DeepSeek-VL2-Tiny exceed the uncompressed Full Model at ρ=.625 and ρ=.75 (50.7 and 59.0 vs. 49.3) and exactly match it at ρ=.50. Because merging is a lossy, training-free compression, this is implausible and suggests an evaluation inconsistency such as mismatched example lists, a score-recording error, or different decoding settings. The paper provides no paired uncertainty or re-check for Tiny, unlike the Small and Qwen3-VL comparisons, so the anomaly is unquantified. These values contribute to the reported Avg (the 9.7-point surplus at ρ=.75 adds about 1.6 points), and thus the retention percentages and margins over the best baseline depend on them. Please verify the InfoVQA rows, correct or explain them, and add uncertainty estimates for Tiny.
  2. [Appendix C.3] The baseline implementations are acknowledged as adapted ports; for REAM, the paper states that the calibration replay does not retain the complete visual sequence and attention state required by REAM's sequential mode. This means the REAM rows in Table 1 may reflect a weakened version of the method, potentially biasing the comparison. Because REAM is included in the main table, please assess the impact of the incomplete visual/attention state on REAM's grouping quality, provide a more faithful port if possible, and discuss how the comparison changes.
  3. [Appendix B.1 / Sec. 3.2] The method's justification rests on the deterministic partition of routed tokens into image-context, question, and answer phases, but the paper provides no sensitivity analysis to this phase-boundary definition (e.g., moving the system prompt or interleaved text to a different bucket) and no validation that the buckets correspond to functional inference phases. A control experiment varying the phase boundaries would help establish that the reported gains are due to phase-conditioned roles rather than an artifact of the specific token split.
minor comments (5)
  1. [Algorithm 1 / Sec. 4] The value of ε is never specified; please state the exact value used in the experiments.
  2. [Sec. 5.1 / Appendix C] The calibration-set sizes for DeepSeek-VL2-Small and Qwen3-VL are not stated in the main text (the appendix mentions 64 per source for Small and 128 per source for Tiny; the Qwen3-VL size is unclear). Please make the calibration budgets explicit for all models.
  3. [Sec. 4.3] ACR is introduced in the main text before it is defined; the definition appears only in Appendix A.3. Please provide a one-sentence definition when it is first mentioned.
  4. [Table 1] Consider using a cleaner table format, e.g., separating the Full Model row and using distinct symbols for Best/Second-best, to improve readability.
  5. [Conclusion] The statement that 'relative improvements of up to 9.6%' is based on one setting; consider reporting the range of relative gains across all model–retention combinations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: RoleMerge's information-loss equations are mathematical identities, and the central performance claims are assessed against external baselines on held-out benchmarks with fixed, sensitivity-audited hyperparameters.

full rationale

The derivation chain is self-contained. Eq. (4) is an exact mutual-information decomposition, I_P(S;B)-I_P(C;B) = sum_G sum_{e in G} P(e) D_KL(q_e || q_G), and Eq. (5) is its pairwise specialization; these are identities that define a compatibility measure, not conclusions assumed from the data. RRPs are constructed from calibration routing counts, and the merge score is used as a deterministic greedy grouping criterion. The paper does not fit any parameter to the six-task outcome: lambda=.03 is fixed and swept, the TCS calibration set is pre-specified and selected via calibration-only routing diagnostics, and sensitivity analyses (Tables 3, 10-14) show broad plateaus rather than a tuned optimum. The central claim is benchmarked against external methods (MC-SMoE, MergeMoE, HC-SMoE, REAM, Sub-MoE, NAMEx) under a matched protocol, with paired bootstrap uncertainty for the main comparisons. ACR is explicitly labeled a mechanism diagnostic rather than a universal performance predictor; although its per-group form is algebraically related to the collision statistic, the benchmark evidence does not reduce to ACR. The Discussion candidly lists limitations: RRPs 'should not be interpreted as universal semantic labels' and RoleMerge 'does not directly address parameter interference within merged experts'; these are scope statements, not circular supports. The Table 1 Tiny InfoVQA values above the Full Model (50.7/59.0 vs 49.3) are a possible evaluation-consistency risk, but that is a correctness concern, not a self-referential derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or speculative entities. The Routing Role Profile is a derived statistic from observed routing counts, not a postulated object. The main unstated costs are the domain assumptions about phase structure, calibration representativeness, and parameter-interference-free averaging.

free parameters (2)
  • collision weight lambda = 0.03 (default)
    Controls the strength of the answer-aware collision adjustment in Eq. (8). Chosen by hand; sensitivity sweep in Table 14 shows lambda=0.10 yields +0.64 Avg, so the default is not uniquely optimal.
  • epsilon (floor constant) = not specified
    Small positive constant used in Eq. (7) and Algorithm 1 to avoid division by zero and provide a positive floor in lambda normalization. Its value is never given, though results are likely insensitive.
assumptions (4)
  • domain assumption Three-phase token partition (image-context, question, answer) captures the functional structure of MoE-VLM inference
    Stated in Section 1 and used to construct all RRPs; if phase boundaries are wrong, the role profiles are misaligned with actual inference phases.
  • domain assumption Routing statistics on the TCS calibration set are representative of routing on downstream benchmarks
    The method pools routing traces only from TextVQA/ChartQA/ScienceQA; no distribution-shift analysis is provided, yet the merge decision for each layer depends entirely on these traces.
  • domain assumption Weighted averaging of expert parameters with frequency weights does not introduce harmful parameter interference
    Eq. (9) merges parameters directly; the Discussion explicitly states the method does not address parameter interference within merged experts.
  • standard math Standard mutual-information and KL-divergence identities
    Eq. (4) relies on I(S;B)-I(C;B)=I(S;B|C) for deterministic C=c(S); this is standard and not proved in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models." pith.science (2026). https://pith.science/paper/OGK4TIGE

@misc{pith2026260804454,
  author       = {Pith},
  title        = {Pith review of: Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGK4TIGE}},
  note         = {Machine review of arXiv:2608.04454}
}
read the original abstract

Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility. However, MoE-VLM inference is phase-structured: image-context tokens carry visual content, question tokens specify the query, and answer tokens produce the output, with different counts and routing distributions. Because image-context tokens are far more numerous, global aggregation can overemphasize image-context processing and obscure phase-conditioned expert roles, making experts serving different phases appear interchangeable and degrading model performance. We therefore argue that MoE-VLM expert merging should preserve phase-conditioned expert roles, judging compatibility by how experts serve different phases rather than globally aggregated routing statistics. Based on this view, we propose RoleMerge, a training-free method that constructs each expert's Routing Role Profile (RRP) from phase-normalized routing statistics, capturing its relative phase preference. Guided by expert-phase information loss, RoleMerge merges experts with compatible profiles and their corresponding router entries while preserving answer-decoding expert distinctions. Experiments on three models and multiple benchmarks show that RoleMerge preserves more of the full model's performance than alternative expert-merging methods at matched expert-retention ratios, with relative improvements of up to 9.6 percent in six-task macro-average performance. These results validate phase-conditioned expert roles as a more effective basis than global routing aggregation for MoE-VLM expert merging.

Figures

Figures reproduced from arXiv: 2608.04454 by the authors.

Figure 1
Figure 1. Phase-conditioned routing breadth, measured as exponentiated routing entropy normalized by expert count. Answer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. RoleMerge overview. Phase-separated routes form RRPs; answer-aware grouping combines role dissimilarity and the signed collision adjustment, then merges expert parameters and router entries with shared within-group weights. use phase-conditioned expert roles for merge compatibility. Training-free expert merging. By replacing groups of pretrained experts with fewer merged experts, training-free expert merging directl… view at source ↗
Figure 3
Figure 3. Fixed-budget calibration diagnostics on DeepSeek-VL2-Tiny at [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 25 canonical work pages

  1. [1]

    International Conference on Learning Representations , year =

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author =. International Conference on Learning Representations , year =

  2. [2]

    Journal of Machine Learning Research , volume =

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , author =. Journal of Machine Learning Research , volume =. 2022 , url =

    2022

  3. [3]

    Advances in Neural Information Processing Systems , volume =

    Scaling Vision with Sparse Mixture of Experts , author =. Advances in Neural Information Processing Systems , volume =. 2021 , url =

    work page 2021

  4. [4]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2026 , doi =

    work page 2026

  5. [5]

    Merge, Then Compress: Demystify Efficient

    Li, Pingzhi and Zhang, Zhenyu and Yadav, Prateek and Sung, Yi-Lin and Cheng, Yu and Bansal, Mohit and Chen, Tianlong , booktitle =. Merge, Then Compress: Demystify Efficient. 2024 , url =

    work page 2024

  6. [6]

    International Conference on Learning Representations , year =

    Mixture Compressor for Mixture-of-Experts LLMs Gains More , author =. International Conference on Learning Representations , year =

    work page

  7. [7]

    2026 , doi =

    Huang, Wei and Liao, Yue and Chen, Yukang and Liu, Jianhui and Tan, Haoru and Liu, Si and Zhang, Shiming and Yan, Shuicheng and Qi, Xiaojuan , journal =. 2026 , doi =

    work page 2026

  8. [8]

    and Teo, Rachel S.Y

    Nguyen, Dung Viet and Nguyen, Minh Hoang and Nguyen, Luc Q. and Teo, Rachel S.Y. and Nguyen, Tan M. and Tran, Linh Duy , booktitle =. 2025 , url =

    work page 2025

Show all 44 references
  1. [9]

    2511.04805 , archivePrefix =

    Zhao, Yushu and Wang, Zheng and Zhang, Minjia , year =. 2511.04805 , archivePrefix =

    arXiv

  2. [10]

    2026 , url =

    Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan , booktitle =. 2026 , url =

    2026

  3. [11]

    2026 , doi =

    Jha, Saurav and Hashemzadeh, Maryam and Pasand, Ali Saheb and Parviz, Ali and Lee, Min-Joong and Knyazev, Boris , journal =. 2026 , doi =

    2026

  4. [12]

    2025 , address =

    Chen, Yuanteng and Shao, Yuantian and Wang, Peisong and Cheng, Jian , booktitle =. 2025 , address =. doi:10.18653/v1/2025.acl-long.633 , url =

    2025 doi

  5. [13]

    doi:10.48550/arXiv.2407.21770 , url =

    Lin, Xi Victoria and Shrivastava, Akshat and Luo, Liang and Iyer, Srinivasan and Lewis, Mike and Ghosh, Gargi and Zettlemoyer, Luke and Aghajanyan, Armen , year =. doi:10.48550/arXiv.2407.21770 , url =. 2407.21770 , archivePrefix =

    doi arXiv

  6. [14]

    and Chen, Tianlong and Long, Qi , booktitle =

    Xin, Jiayi and Yun, Sukwon and Peng, Jie and Choi, Inyoung and Ballard, Jenna L. and Chen, Tianlong and Long, Qi , booktitle =. 2025 , volume =

    2025

  7. [15]

    doi:10.48550/arXiv.2412.10302 , url =

    Wu, Zhiyu and Chen, Xiaokang and Pan, Zizheng and Liu, Xingchao and Liu, Wen and Dai, Damai and others , year =. doi:10.48550/arXiv.2412.10302 , url =. 2412.10302 , archivePrefix =

    doi arXiv

  8. [16]

    doi:10.48550/arXiv.2511.21631 , url =

    Bai, Shuai and Cai, Yuxuan and Chen, Ruizhe and Chen, Keqin and Chen, Xionghui and Cheng, Zesen and others , year =. doi:10.48550/arXiv.2511.21631 , url =. 2511.21631 , archivePrefix =

    doi arXiv

  9. [17]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages =

    InfographicVQA , author =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages =. 2022 , url =

    2022

  10. [18]

    2024 , url =

    Yue, Xiang and Ni, Yuansheng and Zhang, Kai and Zheng, Tianyu and Liu, Ruoqi and Zhang, Ge and others , booktitle =. 2024 , url =

    2024

  11. [19]

    2024 , volume =

    Liu, Yuliang and Li, Zhang and Huang, Mingxin and others , journal =. 2024 , volume =. doi:10.1007/s11432-024-4235-6 , url =. 2305.07895 , archivePrefix =

    2024 arXiv

  12. [20]

    Think You Have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , year =. Think You Have Solved Question Answering? Try. doi:10.48550/arXiv.1803.05457 , url =. 1803.05457 , archivePrefix =

    doi arXiv

  13. [21]

    2020 , doi =

    Bisk, Yonatan and Zellers, Rowan and Le Bras, Ronan and Gao, Jianfeng and Choi, Yejin , journal =. 2020 , doi =

    2020

  14. [22]

    2020 , doi =

    Sakaguchi, Keisuke and Le Bras, Ronan and Bhagavatula, Chandra and Choi, Yejin , journal =. 2020 , doi =

    2020

  15. [23]

    Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus , booktitle =. Towards. 2019 , url =

    2019

  16. [24]

    2022 , address =

    Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul , booktitle =. 2022 , address =. doi:10.18653/v1/2022.findings-acl.177 , url =

    2022 doi

  17. [25]

    Advances in Neural Information Processing Systems , volume =

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  18. [26]

    Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, C. V. , booktitle =. 2021 , url =

    2021

  19. [27]

    Computer Vision -- ECCV 2016 , pages =

    A Diagram Is Worth a Dozen Images , author =. Computer Vision -- ECCV 2016 , pages =. 2016 , publisher =. doi:10.1007/978-3-319-46493-0_15 , url =

    2016 doi

  20. [28]

    doi:10.48550/arXiv.2407.07895 , url =

    Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan , year =. doi:10.48550/arXiv.2407.07895 , url =. 2407.07895 , archivePrefix =

    doi arXiv

  21. [29]

    and Zhang, Hao and Stoica, Ion , booktitle =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with. 2023 , doi =

    2023

  22. [30]

    International Conference on Learning Representations , year =

    Expert Merging in Sparse Mixture of Experts with Nash Bargaining , author =. International Conference on Learning Representations , year =

  23. [31]

    doi:10.48550/arXiv.2510.14436 , url =

    Miao, Ruijie and Yao, Yilun and Wang, Zihan and Wang, Zhiming and Yi, Bairen and Liu, LingJun and Zhao, Yikai and Yang, Tong , year =. doi:10.48550/arXiv.2510.14436 , url =. 2510.14436 , archivePrefix =

    doi

  24. [32]

    The Annals of Mathematical Statistics , volume =

    On Information and Sufficiency , author =. The Annals of Mathematical Statistics , volume =. 1951 , doi =

    1951

  25. [33]

    Retraining-free Merging of Sparse

    Chen, I-Chun and Liu, Hsu-Shen and Sun, Wei-Fang and Chao, Chen-Hao and Hsu, Yen-Chang and Lee, Chun-Yi , booktitle =. Retraining-free Merging of Sparse. 2025 , publisher =

    2025

  26. [34]

    Advances in Neural Information Processing Systems , volume =

    Agglomerative Information Bottleneck , author =. Advances in Neural Information Processing Systems , volume =. 1999 , url =

    1999

  27. [35]

    doi:10.48550/arXiv.2508.18265 , url =

    Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others , year =. doi:10.48550/arXiv.2508.18265 , url =. 2508.18265 , archivePrefix =

    doi arXiv

  28. [36]

    2024 , eprint =

    Aria: An Open Multimodal Native Mixture-of-Experts Model , author =. 2024 , eprint =. doi:10.48550/arXiv.2410.05993 , url =

    2024 doi arXiv

  29. [37]

    2021 , url =

    Lepikhin, Dmitry and Lee, HyoukJoong and Xu, Yuanzhong and Chen, Dehao and Firat, Orhan and Huang, Yanping and Krikun, Maxim and Shazeer, Noam and Chen, Zhifeng , booktitle =. 2021 , url =

    2021

  30. [38]

    2022 , url =

    Rajbhandari, Samyam and Li, Conglong and Yao, Zhewei and Zhang, Minjia and Aminabadi, Reza Yazdani and Awan, Ammar Ahmad and Rasley, Jeff and He, Yuxiong , booktitle =. 2022 , url =

    2022

  31. [39]

    2023 , url =

    Hwang, Changho and Cui, Wei and Xiong, Yifan and Yang, Ziyue and Liu, Ze and Hu, Han and Wang, Zilong and Salas, Rafael and Jose, Jithin and Ram, Prabhat and Chau, HoYuen and Cheng, Peng and Yang, Fan and Yang, Mao and Xiong, Yongqiang , booktitle =. 2023 , url =

    2023

  32. [40]

    Dai, Damai and Deng, Chengqi and Zhao, Chenggang and Xu, R. X. and Gao, Huazuo and Chen, Deli and Li, Jiashi and Zeng, Wangding and Yu, Xingkai and Wu, Y. and Xie, Zhenda and Li, Y. K. and Huang, Panpan and Luo, Fuli and Ruan, Chong and Sui, Zhifang and Liang, Wenfeng , bookti...

    2024 doi

  33. [41]

    doi:10.48550/arXiv.2402.01739 , url =

    Xue, Fuzhao and Zheng, Zian and Fu, Yao and Ni, Jinjie and Zheng, Zangwei and Zhou, Wangchunshu and You, Yang , year =. doi:10.48550/arXiv.2402.01739 , url =. 2402.01739 , archivePrefix =

    doi arXiv

  34. [42]

    and Koh, Pang Wei and Singh, Amanpreet and Hajishirzi, Hannaneh , booktitle =

    Muennighoff, Niklas and Soldaini, Luca and Groeneveld, Dirk and Lo, Kyle and Morrison, Jacob and Min, Sewon and Shi, Weijia and Walsh, Pete and Tafjord, Oyvind and Lambert, Nathan and Gu, Yuling and Arora, Shane and Bhagia, Akshita and Schwenk, Dustin and Wadden, David and Wet...

    2025

  35. [43]

    International Conference on Learning Representations , year =

    Multilingual Routing in Mixture-of-Experts , author =. International Conference on Learning Representations , year =

  36. [44]

    doi:10.48550/arXiv.2401.14361 , url =

    Xue, Leyang and Fu, Yao and Lu, Zhan and Mai, Luo and Marina, Mahesh , year =. doi:10.48550/arXiv.2401.14361 , url =. 2401.14361 , archivePrefix =

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.