Pith. sign in

REVIEW 4 major objections 6 minor 43 references

SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A soft routing regularizer keeps a multimodal MoE's language skills mostly intact while training on only 2.5% text.

desk verdict A clean, well-specified regularizer with a headline result that mostly evaporates once MBPP is set aside; the method deserves peer review, but the language-retention claim needs better support. read the letter →

arxiv 2506.06406 v2 pith:7IRVDDV5 submitted 2025-06-06 cs.CL cs.AI

Guoyang Xia , Yifeng Ding , Fengfa Li , Lei Ren , Wei Chen , Fangxiang Feng , Xiaojie Wang This is my paper
classification cs.CLcs.AI
keywords mixtureofexpertsmultimodallargelanguagemodelsmodality-awareroutingcapabilityretentionKLdivergenceregularizationvisualinstructiontuningexpertspecializationdistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most routes to turning a pretrained mixture-of-experts language model into a vision-language model either mix in a large share of text-only data or freeze parts of the backbone, both of which cost efficiency or multimodal quality. This paper proposes a regularizer that tracks the routing distribution of image tokens and text tokens over experts and softly enforces how far apart those two distributions can be. On visual instruction tuning with only 2.5% pure text, the method retains 86.6% of the backbone's language score across six benchmarks, against 81.6% without the auxiliary loss and 82.8% with a standard load-balancing loss. The underlying point is that a small, architecture-free loss term can make experts specialize by modality and thereby protect language competence without a large text corpus.

What carries the argument

The central object is the modality routing distribution (MRD): for each modality $m\in\{v,t\}$, the batch-level probability mass $\tilde{q}_m$ over experts, built from the frequency with which each expert is selected by tokens of that modality and the average normalized routing weight. The SMAR loss is a hinge on the symmetric KL divergence $d_{\mathrm{sym\text{-}KL}}(\tilde{q}_v,\tilde{q}_t)$, penalizing distances only when they fall outside $[d_{\min}, d_{\max}]$; small trainable modality-aware biases $b_v, b_t \in \mathbb{R}^E$ are added to the router logits to make the separation learnable. Together these steer the router toward modality-specialized experts while leaving the transformer architecture, visual encoder, and data untouched.

What would settle it

Train the identical model with SMAR, but compute the MRD distance on shuffled modality labels so the loss has no true signal about which tokens are visual or textual; if language retention still reaches about 86.6%, the reported preservation is not caused by modality separation.

Watch

Extended reading notes

Core claim

The paper's claim is that language degradation during multimodal fine-tuning of a MoE model can be curbed by controlling the gap between the routing distributions of vision and text tokens, rather than by adding text data or freezing layers. Concretely, it defines a modality routing distribution (MRD) per modality, measures the symmetric KL divergence between the vision and text MRDs, and adds a hinge penalty whenever that distance leaves a chosen band $[d_{\min}, d_{\max}]$. The resulting SMAR loss steers some experts toward text, some toward images, and leaves others as mixed fusion experts, without hard-assigning experts to modalities. With this loss, the paper reports 86.6% language-capability retention on a six-benchmark average using only 2.5% pure text, while holding or slightly improving most multimodal benchmark scores relative to the baseline and to the load-balancing-only variant.

Load-bearing premise

The load-bearing premise is that making image tokens and text tokens flow through noticeably different sets of experts is what actually preserves the model's language ability, so controlling that gap through a distance penalty is sufficient—rather than the gap being only a by-product of better training or of benchmark selection.

Editorial extensions

If this is right

  • With only 2.5% pure text in the instruction-tuning mix, SMAR reaches 86.6% language retention, beating the 81.6% no-auxiliary-loss baseline and the 82.8% load-balancing-only baseline.
  • SMAR carries over to a second MoE architecture (MoE-LLaVA) when applied during its expert-expansion stage, improving several language benchmarks without hurting multimodal scores.
  • The tolerance band matters: setting $d_{\min}=0.1, d_{\max}=0.5$ causes routing collapse and the worst overall performance, while $[1.5, 2.0]$ gives the best language result.
  • The final configuration drops the standard load-balancing loss; the paper's ablation shows that adding load balancing to SMAR degrades the result.
  • The method requires no architecture changes and no frozen backbone, so it can be plugged into existing MoE-MLLM training loops.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 86.6% retention number is an average over six benchmarks that includes MBPP, where SMAR's gain is much larger than on the other tasks; averaging only knowledge and reasoning benchmarks would make the advantage over baselines much smaller.
  • If modality-separated routing is the causal mechanism, an analogous band on pairwise routing distances could also protect other competences such as code or multilingual skill during multimodal expansion, which the paper does not test.
  • A direct causal test would be to keep the same loss but shuffle the modality labels used to compute the MRDs; if language retention still rose, the effect would not be about true modality separation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SMAR, a regularization loss that penalizes deviations of the symmetric KL divergence between batch-level vision and text routing distributions from a target band, aiming to preserve language capabilities when adapting MoE-based LLMs to multimodal instruction tuning. The method is tested on a Mixtral 8x7B-based VITA-style model using only 2.5% pure-text data, reporting an 86.6% language retention rate versus 81.6% for the no-auxiliary-loss baseline and 82.8% for the load-balancing-loss baseline, and is also applied to MoE-LLaVA. The loss formulation, training recipe, and dataset composition are clearly specified.

Significance. If the retention result were robust, SMAR would be a cheap and practical way to preserve language skills during multimodal adaptation, avoiding the need for large text-only corpora or architectural overhauls. The paper provides a clearly specified loss, a two-stage training protocol, ablation of thresholds and components, and a transfer experiment to another MoE architecture. However, the headline claim is not yet supported because the retention advantage is largely an artifact of benchmark selection (MBPP) and the experiments are single runs without error bars or significance tests.

major comments (4)
  1. [Section 4.3, Table 2] The headline retention advantage (86.6% vs 81.6% and 82.8%) is almost entirely due to the inclusion of MBPP in the six-benchmark average. If MBPP is excluded, the retention ratios become approximately 90.7% (SMAR), 90.2% (no-auxiliary baseline), and 90.3% (load-balancing), a gap of about 0.5 percentage points. Because the paper itself attributes the MBPP gain to 'preservation of code formatting' and instruction-following (Section 4.3), MBPP is not a clean measure of parametric language knowledge. The authors should report retention with and without MBPP, and justify why MBPP is included while HumanEval and IFEval are excluded from the retention average.
  2. [Section 4.4, Table 4] The SMAR hyperparameters [dmin, dmax] and beta are selected by evaluating the same language benchmarks (MMLU, GSM8K, BBH, MBPP, HumanEval) that are used in the headline retention claim. This means the reported 86.6% reflects a fitted choice, not an out-of-sample prediction. The authors should select thresholds on a validation split or report retention averaged over the whole grid, and show the sensitivity of the headline difference to the threshold choice.
  3. [Section 4.2, Table 2] All experiments are single runs with no error bars or significance tests. After removing MBPP, the difference between SMAR and the baselines is only about 0.5 percentage points, which is within typical run-to-run variance for this type of evaluation. At least three seeds, or an appropriate significance test, are needed to support the claim that SMAR 'outperforms' the baselines in language retention.
  4. [Section 3.2, Section 4.4] The proposed causal mechanism—that separating routing distributions preserves linguistic knowledge—is not demonstrated. The ablation in Section 4.4 shows that the relation between the MRD distance band and retention is non-monotonic (best at [1.5,2.0], worse at lower bands, with routing collapse at [0.1,0.5]), and the paper provides no analysis linking the achieved layer-wise MRD distance to the retention numbers. A control experiment (e.g., applying the same KL penalty to a random modality split, or measuring retention against the actually achieved distance) would strengthen the causal interpretation.
minor comments (6)
  1. [Abstract, Section 1, Eqs. (1)-(2)] The claim 'without modifying model architecture' is inconsistent with the introduction of trainable modality-aware bias vectors bv and bt. Please clarify that the core MoE structure is unchanged but that router parameters are added.
  2. [Section 4.3] The sentence listing gains over LLaVA-1.5-13B gives five percentages (5.4%, 7.0%, 7.4%, 12.8%, 3.0%) for six benchmarks (SQA I, MME, MMBench, MM-Vet, VQAT, VQAv2). Please correct the mismatch.
  3. [Table 2, Section 4.3] The 'Avg.' retention ratio is computed as the ratio of the averaged raw scores, not the average of the per-benchmark ratios. Please state this formula explicitly to avoid ambiguity.
  4. [Table 1] The column header 'VQAT' is not defined in the caption; please spell out the benchmark name (e.g., TextVQA) in the caption.
  5. [Section 4.4, Table 4] The threshold grid is coarse and the best setting [1.5,2.0] is at the upper end of the tested range; consider extending the grid to show where performance peaks.
  6. [Section 5, Section 6] The limitations section correctly acknowledges the two hyperparameters and limited sensitivity exploration; consider also noting the benchmark-composition sensitivity of the headline retention number.

Circularity Check

2 steps flagged · score 4.0 of 10

SMAR's headline retention advantage is partly a fitted hyperparameter choice: [dmin,dmax] and beta are selected on the same language benchmarks later reported as the 86.6% retention, and the gap over baselines is dominated by MBPP.

  1. fitted input called prediction [Section 4.4 (Ablation on SMAR Thresholds, Table 4) and Section 4.3 (Table 2 retention ratios); Appendix A Table 7]
    "The best overall language score is obtained for d min = 1.5 and d max = 2.0. ... Using only 2.5% pure-text data, SMAR still preserves 86.6%—clearly surpassing both the no-auxiliary-loss variant (81.6%) and the load-balancing-only variant (82.8%)."

    The headline retention number is not an out-of-sample prediction: the SMAR hyperparameters ([dmin,dmax] = [1.5,2.0], and beta = 0.01 per Table 7) were selected by inspecting Table 4, which reports MMLU, GSM8K, BBH, MBPP, and HumanEval for each threshold pair, and the paper states that the best 'overall language score' determines the choice. The same language benchmarks (MMLU, GSM8K, BBH, MBPP) then feed the Table 2 six-benchmark average used for the 86.6% versus 81.6%/82.8% claim, so part of the reported advantage is a selected-maximum effect rather than an independent consequence of the KL regularizer.

  2. other [Section 4.3, Table 2 and Table 1; paragraph starting 'Notably, in code-related evaluations']
    "we average performance exclusively across six benchmarks (C-EVAL, MMLU, GSM8K, ARC-Challenge, BBH, and MBPP) that have minimal impact on instruction-following capability to compute the retention ratio of language capabilities, as shown in Table 2."

    The choice of which language benchmarks define retention is itself part of what makes the headline claim. The table excludes HumanEval and IFEval, where SMAR also improves, and includes MBPP, where SMAR's gain is nearly double the load-balancing baseline; removing MBPP from the same six-benchmark average leaves only a roughly 0.5-point retention gap. This is not a by-construction circularity, but it makes the reported 86.6% 'retention ratio' sensitive to the paper's own benchmark-averaging rule, so the outperformance claim is not robust to an equally defensible benchmark selection.

full rationale

There is no by-construction equivalence in the method itself: the MRD (Eqs. 6-9) and the SMAR loss (Eqs. 10-13) are defined purely from router logits and token modality labels, not from language-retention scores, so the loss is not a renamed version of the claimed outcome. The paper contains no load-bearing self-citations: the cited prior work (VITA, Mixtral, MoE-LLaVA, Uni-MoE, etc.) is external and is used for architecture and context, not to justify the core claim. The central weakness is a fitted-input issue: Section 4.4 explicitly tunes [dmin,dmax] to maximize language scores on a benchmark set that overlaps heavily with the Table 2 retention average, and the final configuration also omits load-balancing loss because Table 5 showed it 'degrades the results' on those same language benchmarks. The 86.6% retention and the claimed outperformance over the 81.6%/82.8% baselines therefore partly reflect hyperparameter selection on the evaluation set; excluding MBPP from the same averages reduces the gap to under a percentage point. This warrants a 4 rather than a higher score because the selection is acknowledged in the Limitations section, the loss mechanism has independent content, and the multimodal results are compared against external open models.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

SMAR relies on three tuned hyperparameters (distance band, loss weight, learned modality biases) and on the untested premise that routing-distribution separation preserves language ability. The MRD metric is a new paper-specific construct without independent validation.

free parameters (3)
  • MRD distance tolerance band dmin/dmax = [1.5, 2.0] for Mixtral; [1.0, 1.5] for MoE-LLaVA
    Tuned via ablations in Table 4 on the same benchmarks used for the headline claim; controls how far image and text routing distributions are pushed apart.
  • SMAR loss weight beta = 0.01
    Chosen by hand and fixed across experiments; weights the KL penalty in the total loss (Eq. 14).
  • Modality-aware bias vectors bv and bt = Learned during training
    Added to the router logits for vision and text tokens in Eq. (1)-(2); the Table 5 ablation shows they are important to SMAR's language-retention effect.
assumptions (3)
  • domain assumption Increasing the distance between visual and textual routing distributions preserves language capabilities.
    Central premise of the method, motivated in Section 1 and measured through retention ratios; no theoretical guarantee that wider MRD distance protects linguistic knowledge.
  • domain assumption Mini-batch MRD, computed as the normalized product of routing frequency and expected weight, is a reliable estimate of true per-modality routing behavior.
    Used in Eq. (6)-(8); the estimate depends on the sampled mixture of visual and text tokens and has no convergence or variance analysis.
  • domain assumption Symmetric KL divergence is an appropriate distance measure for controlling modality differentiation.
    The loss in Eq. (10)-(13) uses this choice; no comparison with alternative distributional distances or justification for the tolerance band shape.
invented entities (1)
  • Modality Routing Distribution (MRD)
    purpose: Quantifies, per expert, how much routing mass comes from image versus text tokens, forming the target of the SMAR KL constraint.
    Defined in Eq. (6)-(9) and used throughout; it is a paper-specific diagnostic statistic with no separate falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities." pith.science (2026). https://pith.science/paper/7IRVDDV5

@misc{pith2026250606406,
  author       = {Pith},
  title        = {Pith review of: SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7IRVDDV5}},
  note         = {Machine review of arXiv:2506.06406}
}
read the original abstract

Mixture of Experts (MoE) architectures have become a key approach for scaling large language models, with growing interest in extending them to multimodal tasks. Existing methods to build multimodal MoE models either incur high training costs or suffer from degraded language capabilities when adapting pretrained models. To address this, we propose Soft ModalityAware Routing (SMAR), a novel regularization technique that uses Kullback Leibler divergence to control routing probability distributions across modalities, encouraging expert specialization without modifying model architecture or heavily relying on textual data. Experiments on visual instruction tuning show that SMAR preserves language ability at 86.6% retention with only 2.5% pure text, outperforming baselines while maintaining strong multimodal performance. Our approach offers a practical and efficient solution to balance modality differentiation and language capabilities in multimodal MoE models.

Figures

Figures reproduced from arXiv: 2506.06406 by the authors.

Figure 1
Figure 1. Illustration of the proposed Soft Modality-Aware Routing (SMAR) mechanism inside a single Mixtral decoder layer. Left: Visual tokens {V1, . . . , Vm} (orange) and textual tokens {T1, . . . , Tn} (blue) are fed into the shared MoE-FFN. The router selects the Top-K experts for each token (dashed arrows). Right: The token-expert matrix (heat-map) represents the router logits of each token. We calculate the modality rou… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The MRD distance curve of different methods. It is evident that after applying the SMAR method to encourage modality-specific expert differentiation, the MRD curves differ significantly from those observed in methods without SMAR [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The detailed depiction of the proportion of image and text tokens routed to each expert at every layer. (a) illustrates the modality preferences of experts across layers in the model trained solely with the load-balancing loss. (b) demonstrates the effectiveness of the…
Figure 5
Figure 5. Figure 5: The MRD distance curve illustrating the effects of applying Modality-Specific Bias and the load-balancing loss within the SMAR framework. Effect of the Trainable Modality-Aware Bias and the Load–Balancing Loss [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 5 canonical work pages

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  3. [3]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185--24198

    work page 2024

  4. [4]

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  5. [5]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  8. [8]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

    arXiv 2023

Show all 43 references
  1. [9]

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, and 1 others. 2024. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211

    2024 arXiv

  2. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  3. [11]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3...

    2018

  4. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

  6. [14]

    Yongqi Huang, Peng Ye, Chenyu Huang, Jianjian Cao, Lin Zhang, Baopu Li, Gang Yu, and Tao Chen. 2025. Ders: Towards extremely efficient upcycled mixture-of-experts models. arXiv preprint arXiv:2503.01359

    2025 arXiv

  7. [15]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing ...

    2023

  8. [16]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  9. [17]

    Mojan Javaheripi, S \'e bastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio C \'e sar Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, and 1 others. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3

    2023

  10. [18]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  11. [19]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2023 a . Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425

    2023 arXiv

  12. [20]

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, and 1 others. 2024. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993

    2024 arXiv

  13. [21]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023 b . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  14. [22]

    Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. 2025. Uni-moe: Scaling unified multimodal llms with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2025

  15. [23]

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, and 1 others. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947

    2024 arXiv

  16. [24]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  17. [25]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023 a . Aligning large multi-modal model with robust instruction tuning. CoRR

    2023

  18. [26]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 b . Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744

    2023 arXiv

  19. [27]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2023 c . Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2023 arXiv

  20. [28]

    Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2024. A closer look into mixture-of-experts in large language models. arXiv preprint arXiv:2406.18219

    2024 arXiv

  21. [29]

    Jinqiang Long, Yanqi Dai, Guoxing Yang, Hongpeng Lin, Nanyi Fei, Yizhao Gao, and Zhiwu Lu. 2024. Awaker2. 5-vl: Stably scaling mllms with parameter-efficient mixture of experts. arXiv preprint arXiv:2411.10669

    2024 arXiv

  22. [30]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Sy...

    2022

  23. [31]

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. 2024. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202

    2024 arXiv

  24. [32]

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. 2023. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226

    2023 arXiv

  25. [33]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  26. [34]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and 1 others. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  27. [35]

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574

    2023 arXiv

  28. [36]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  29. [37]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, and 1 others. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302

    2024 arXiv

  30. [38]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  31. [39]

    Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. 2024. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. arXiv preprint arXiv:2410.08245

    2024 arXiv

  32. [40]

    Bo Zhao, Boya Wu, and Tiejun Huang. 2023. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087

    2023 arXiv

  33. [41]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  34. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.