Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One vision encoder, split into routed experts, beats multi-task training

desk verdict Solid idea with a clean architecture, but the evaluation protocol leaks test-set signal into router tuning; worth a serious referee with major revisions. read the letter →

arxiv 2505.24541 v1 pith:WEHN33J2 submitted 2025-05-30 cs.CV cs.AI

Xin He , Xumeng Han , Longhui Wei , Lingxi Xie , Qi Tian This is my paper
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsmixture-of-expertsvisionencoderdomainconflictdynamicroutingvisualinstructiontuningefficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the performance ceiling of a multimodal LLM on diverse visual tasks comes less from encoder capacity than from how the encoder's parameters are organized. Its proposal, Mixpert, keeps a single jointly fine-tuned vision encoder but restructures the deep layers and projector into multiple domain-specific experts, with a lightweight router sending each image to the most suitable expert. This is claimed to preserve the shared-knowledge benefits of joint training while removing the inter-domain conflicts that hold a single encoder back, and to do so at nearly single-encoder inference cost. On LLaVA-OV, InternVL2, and Qwen2-VL, the paper reports consistent gains on chart, document, OCR, math, and general benchmarks, with router category accuracy around 89–95% on validation samples.

What carries the argument

The load-bearing object is the mixture-of-vision-experts layer stack: the last two ViT transformer layers and the projector of an existing vision encoder are replicated into multiple domain experts, all fed by the same frozen shallow trunk, with a lightweight two-layer MLP router that sees global-average-pooled trunk features and a score-difference threshold that sends ambiguous images to the versatile expert. This design lets the model keep one shared representation at the bottom, specialize only at the top, and never activate more than one expert plus the router per image.

What would settle it

Take a collection of images that mixes the five domains in unseen proportions, including hybrid images such as a math chart or a scanned document with handwriting that the router is likely to find ambiguous, and compare Mixpert's end-to-end benchmark accuracy against the single-encoder baseline at the same compute budget. If the images whose top-two router scores differ by less than the threshold perform no better than random expert selection, or if accuracy on correctly routed images does not exceed the baseline, then the central claim that routing delivers the gains is falsified.

Watch

Extended reading notes

Core claim

Starting from a jointly supervised-fine-tuned MLLM, Mixpert partitions the vision encoder into a frozen shared trunk (the shallow ViT layers) and a mixture of vision experts (the projector plus the last two ViT layers), where each of five experts is trained on one manually defined domain (chart, document, math, OCR, general) and a sixth 'versatile' expert keeps the original joint-SFT weights as a fallback. A two-layer MLP router, fed by average-pooled features from the shared trunk, picks the highest-scoring expert unless the gap between the top two routing scores falls below a threshold, in which case the versatile expert handles the image. The paper's central claim is that this restructured encoder reconciles joint optimization with task specialization: it inherits common knowledge from joint SFT, removes per-domain conflicts through decoupled expert tuning, and adds only about 0.001 G FLOPs of routing overhead at inference. On benchmarks this yields improvements such as +0.6 on ChartQA and DocVQA, +31 on OCRBench, +34 on MME, and gains on MathVista, MathVerse, AI2D, and MMBench over LLaVA-OV-7B, with analogous gains when the same recipe is applied to InternVL2-8B and Qwen2-VL-7B.

Load-bearing premise

The load-bearing premise is that the five hand-defined domain categories capture the real distribution of images and that the router's roughly 89–95% category accuracy on benchmark validation samples carries over to every new image, since routing mistakes are never measured end-to-end and would, if frequent or harmful, erase the reported gains.

Editorial extensions

If this is right

  • Any MLLM with a ViT-style vision encoder and an MLP projector can adopt Mixpert by restructuring only the projector and last two encoder layers, so the recipe transfers without changing the LLM or the training data.
  • Because each expert is fine-tuned independently on its own domain, data-balance concerns in multi-task SFT disappear: a rare domain can be trained as aggressively as a common one without harming other tasks.
  • Inference stays close to single-encoder cost because only the router (about 1.3M parameters and 0.001 G FLOPs) is added per image, while total stored parameters grow by about 237M.
  • Images that mix multiple domain characteristics are handled by the versatile expert rather than forced into a single category, so routing mistakes on ambiguous inputs degrade performance less than direct top-1 routing.
  • When dynamic resolution splits an image into sub-images, applying the routing decision independently to each sub-image yields further gains, extending the method to high-resolution and multi-image settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper measures router accuracy only as a classifier on validation samples from the same task families used in training; a natural extension would measure end-to-end answer quality conditional on the routing decision, which would reveal whether the versatile expert fully absorbs misroutes.
  • The manual five-way taxonomy bundles images into chart, document, math, OCR, and general, but the same architecture could be probed with a coarser or finer partition, or with categories discovered from data clusters instead of human judgment, and the gains might shift accordingly.
  • Because the router uses a global average-pooled representation of the whole image, very small but decisive visual details such as a single formula or a printed stamp may be underweighted; a patch-level or attention-weighted routing signal is a testable variant the paper does not explore.
  • If the benefit stems from decoupled expert tuning rather than routing itself, a single encoder fine-tuned jointly on all domains with per-domain low-rank adapters should show a similar pattern, and comparing those two mechanisms would isolate the true source of the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Mixpert, a mixture-of-vision-experts architecture for multimodal LLMs. The vision encoder is split into a frozen shared shallow component and a set of domain experts (chart, document, math, OCR, general) plus a versatile fallback expert, initialized from the joint-SFT checkpoint. A lightweight two-layer MLP router, operating on global-average-pooled features from the shared component, selects the expert at inference using a score-difference threshold. Experts are fine-tuned on domain-specific data while the shared component and LLM remain frozen. Experiments on LLaVA-OV, InternVL2-8B, and Qwen2-VL-7B report consistent gains on ChartQA, DocVQA, OCRBench, MME, MathVerse, MathVista, and MMBench, with minimal additional activated parameters and FLOPs. The paper also includes ablations on the number of MoE layers, router training data size, routing strategies, and the routing threshold.

Significance. If the results hold, Mixpert provides a practical way to mitigate domain conflicts in MLLMs at near-single-encoder inference cost, which would be a useful contribution. The paper gives a clear motivation experiment (Fig. 2), a thorough efficiency analysis (Table 6), and ablations for layer placement, routing strategy, and threshold. It also attempts a control for extra fine-tuning (Table 7). However, the central evaluation protocol has a selection leak: the router validation set and the threshold sweep use the same test benchmarks that later measure the headline gains. This makes the reported improvements difficult to interpret as evidence of general routing capability. The open-weights comparisons additionally introduce 5.5M training samples not available to the baselines, weakening the integration claims. These issues are fixable with re-designed evaluation and controls, but they are load-bearing for the paper's main claims.

major comments (4)
  1. [Sec 4.1, Tables 4–5, Table 8] The router validation set is sampled from the final evaluation test distributions. Specifically, Sec 4.1 constructs the 25K router validation set from ChartQA test/val, DocVQA test, MathVerse/MathVista test, IIIT5K/HME100K test, and COCO Caption test; Table 8 then evaluates on ChartQA test, DocVQA test, MathVerse mini-vision, and MathVista testmini. Moreover, the routing threshold tau=0.6 in Table 5 is selected by sweeping on ChartQA test, DocVQA test, OCRBench test, MME test, and MathVista testmini. This means the reported router accuracy (~89–95%) and the chosen threshold are fitted to the evaluation distribution. The paper should hold out the final benchmarks entirely from router validation and threshold selection, for example by using a leave-one-benchmark-out procedure or by training the router only on the training corpora and selecting tau on a separate validation split that is disjoint from the test sets used in Table 8.
  2. [Sec 3.3, Table 3, Sec 4.3] The end-to-end effect of routing errors is never measured. Table 3 reports router classification accuracy on the 25K validation set, but this accuracy is not linked to final MLLM performance. The paper assumes that the versatile expert handles low-confidence and misrouted images, but no experiment quantifies what happens when the router misroutes an image to a specialized expert. I ask for an oracle-routing versus router-routing comparison on the final benchmarks, and ideally an explicit corruption experiment (e.g., forcing random or systematically wrong expert assignments) to show the robustness of the fallback mechanism. Without this, the reported gains could be driven by the router's benchmark-specific behavior rather than a general solution.
  3. [Sec 4.4, Table 8, Table 7] The open-weights comparisons are not controlled for the extra training data. For InternVL2-8B and Qwen2-VL-7B, Mixpert is trained on an additional 5.5M collected samples plus 3M router training images, while the base models are evaluated without these data. Table 7 provides a fine-tuned control for LLaVA-OV-7B and InternVL2-8B, but no corresponding control is shown for Qwen2-VL-7B. Even for InternVL2-8B, the control result is reported only in Table 7 and, as presented, does not establish that the improvement in Table 8 is due to the MoE routing rather than the extra fine-tuning data. To support the claim that Mixpert 'integrates seamlessly into any MLLM,' the authors should report a same-data, single-expert fine-tuned control for each backbone, with identical training budgets and data mixtures.
  4. [Table 2, Sec 4.3] The layer-selection ablation uses manual expert assignment based on the test benchmarks: the caption of Table 2 states that 'based on the characteristics of testing samples, we manually selected the specific expert for each evaluation benchmark.' This introduces test-set information into the architecture choice (number of MoE layers and whether to include the projector). Since the chosen configuration (projector + last two ViT layers) is derived from this ablation, the reported efficiency/performance trade-off is partly tuned on the final test sets. The authors should either use a held-out validation set for this selection or clearly acknowledge that the layer choice is optimized on the evaluation benchmarks and show that the conclusion is stable under this selection.
minor comments (5)
  1. [Sec 4.3] The phrase 'some navie routing strategies' contains a typo; it should be 'naive routing strategies.'
  2. [Sec 4.3, 'Additional Cost' paragraph] The sentence 'during inference phrase' should read 'during inference phase.'
  3. [Sec 4.1] The dataset name 'IIITK5K' appears in Sec 4.1, but the reference [42] is titled 'Scene Text Recognition Using Higher Order Language Priors' for the IIIT5K dataset; the paper uses 'IIITK5K' inconsistently.
  4. [Sec 3.3, Eq. (1)–(2)] The routing equations use the notation s_d for the score difference but do not define s^(1) and s^(2) explicitly as the highest and second-highest softmax scores; please add a brief definition for clarity.
  5. [Table 1] The dataset names in Table 1, such as 'K12Printing,' 'MA VIS,' and 'RoBUT WTQ,' are not self-explanatory; consider adding a footnote or reference for each unusual dataset name.

Circularity Check

2 steps flagged · score 4.0 of 10

Router validation and τ threshold selection use the same test benchmarks as the final evaluation, so some of the reported gains are fitted rather than independently predicted; the core architecture comparison remains a genuine control.

  1. fitted input called prediction [Sec 4.3, Table 5, and Sec 4.4, Table 8]
    "Notably, in the above experiments, we set the score-difference threshold τ as 0.6. To further verify its influence, we further conduct ablations with different values of τ. As shown in Table 5, while τ is set as a small number (e.g., 0.1), our score-difference strategy approximates direct routing and achieves similar results. As τ increases beyond a certain threshold (e.g., 0.5), the routing strategy becomes less sensitive to the value changes of τ, and the results across various tasks remains nearly consistent."

    Table 5 sweeps τ on ChartQA test, DocVQA test, OCRBench test, MME test, and MathVista testmini, which are the same benchmarks used to report Mixpert's headline results in Table 8. The Mixpert (LLaVA-OV-7B) entries in Table 8 for those five benchmarks are exactly the τ=0.6 row of Table 5 (79.4, 87.5, 732, 2143, 56.7). Thus the threshold was selected by optimizing on the evaluation distribution, and the final reported gains on those benchmarks are the fitted value rather than an independent out-of-sample prediction.

  2. other [Sec 4.1, 'Training and Evaluation Datasets for Router', and Sec 4.4, Table 8]
    "In addition, we sample 5K chart samples from the ChartQA [39] test and validation set, 5K document samples from the DocVQA [40] test set, 5K math samples from the MathVerse [58] and MathVista [38] test sets, 5K OCR samples from the IIITK5K [42] and HME100K [56] test sets, 5K general samples from the COCO Caption [3] test set, to form a validation dataset consisting of 25K samples for evaluating its accuracy of router in assigning the appropriate expert for each visual input."

    The router's validation set is drawn from the same test/val corpora that later appear as final evaluation benchmarks in Table 8 (ChartQA test, DocVQA test, MathVerse/MathVista, OCR-type IIIT5K/HME100K, and COCO Caption). The Table 3 routing-accuracy numbers therefore measure performance on the evaluation distribution itself, not generalization to a held-out distribution of novel visual inputs. Because final end-task scores depend on the router's choices, the reported gains on those benchmarks are partly an artifact of validating (and, via τ, tuning) the routing mechanism on the benchmarks being evaluated rather than a fully independent prediction.

full rationale

The paper's derivation of Mixpert is mostly self-contained. The experts are initialized from the joint-SFT model and fine-tuned on manually categorized data; the router is a separate MLP trained on 1M labeled samples; and the main control in Table 7, comparing Mixpert against fine-tuning the same last-two-layer-plus-projector on all data, rules out the 'extra fine-tuning' explanation. The central architectural claim does not reduce to the input definitions or to the authors' prior work, and no load-bearing self-citation is present. The significant weakness is evaluative circularity: Sec 4.3 sets the routing threshold τ by sweeping on ChartQA test, DocVQA test, OCRBench test, MME test, and MathVista testmini, and Table 8 then reports Mixpert's results on those same benchmarks with the chosen τ, with the numbers matching the τ=0.6 row of Table 5. Consequently, those headline improvements are partially fitted choices rather than independent predictions. Additionally, the router's validation accuracy in Table 3 is measured on samples from the same test corpora used in the final evaluation, so it does not independently establish routing generalization to arbitrary new images. These issues are mitigated by the Table 7 control, by gains on benchmarks not used in the τ sweep (AI2D, InfoVQA, MMBench), and by consistent improvements on InternVL2-8B and Qwen2-VL-7B, which keeps the central claim from being wholly circular. Score 4 reflects partial, not wholesale, circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or external entities; the experts and router are model components. The main uncharged assumptions are the manual domain taxonomy, the sufficiency of shared shallow features for routing, the versatile expert as a fallback, and the safety of freezing the LLM and shared encoder. The free parameters are design choices selected using the evaluation benchmarks, most notably the routing threshold tau.

free parameters (4)
  • score-difference routing threshold tau = 0.6
    Selected from an ablation measured directly on the final evaluation test sets (Table 5); the reported main results in Table 8 use this threshold.
  • number of deep vision layers converted to MoE = 2 (plus projector)
    Chosen from the MoE layer scanning ablation in Table 2 to balance efficiency and performance; layer count and layer choice are based on benchmark results.
  • domain taxonomy (five categories) = chart, document, math, OCR, general
    Manually predefined in Sec 3.3 and used both for expert assignment and for router labels; this categorization is a hand-chosen modeling choice on which the whole method depends.
  • router training set size = 1M
    Chosen based on Table 3, which shows that router accuracy improves with more training data; the final router uses 1M samples.
assumptions (4)
  • domain assumption Images can be exhaustively grouped into five predefined visual domains (chart, document, math, OCR, general) plus a fallback class.
    Introduced in Sec 3.3 and used to define the experts and to label the router training data; no evidence is given that this taxonomy is complete or stable across real-world images.
  • domain assumption The shallow layers of the vision encoder provide features that are sufficient both for routing and for all expert specializations.
    The router takes input from the shared shallow features (Sec 3.3), and all experts receive the same shared features, so the success of the method depends on this assumption.
  • ad hoc to paper A versatile expert initialized from the joint-SFT checkpoint adequately handles ambiguous images when the router's top-two confidence gap is small.
    The versatile expert is introduced in Sec 3.3 as a fallback, but no dedicated experiment measures whether the fallback actually corrects routing errors or simply leaves the joint-SFT behavior unchanged.
  • domain assumption Freezing the LLM and the shared encoder while tuning only expert layers preserves language and reasoning capabilities.
    The disentangled tuning strategy in Sec 3.3 and 4.2 freezes all common modules; the paper does not analyze whether this freeze limits joint optimization in ways that a full fine-tune would not.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts." pith.science (2026). https://pith.science/paper/WEHN33J2

@misc{pith2026250524541,
  author       = {Pith},
  title        = {Pith review of: Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEHN33J2}},
  note         = {Machine review of arXiv:2505.24541}
}
read the original abstract

Multimodal large language models (MLLMs) require a nuanced interpretation of complex image information, typically leveraging a vision encoder to perceive various visual scenarios. However, relying solely on a single vision encoder to handle diverse task domains proves difficult and inevitably leads to conflicts. Recent work enhances data perception by directly integrating multiple domain-specific vision encoders, yet this structure adds complexity and limits the potential for joint optimization. In this paper, we introduce Mixpert, an efficient mixture-of-vision-experts architecture that inherits the joint learning advantages from a single vision encoder while being restructured into a multi-expert paradigm for task-specific fine-tuning across different visual tasks. Additionally, we design a dynamic routing mechanism that allocates input images to the most suitable visual expert. Mixpert effectively alleviates domain conflicts encountered by a single vision encoder in multi-task learning with minimal additional computational cost, making it more efficient than multiple encoders. Furthermore, Mixpert integrates seamlessly into any MLLM, with experimental results demonstrating substantial performance gains across various tasks.

Figures

Figures reproduced from arXiv: 2505.24541 by the authors.

Figure 1
Figure 1. Vision encoder structures. (a) A single vision en￾coder handles various task scenarios. (b) Integrating multiple task￾specific vision encoders to enhances data perception but introduces additional costs. (c) Our efficient mixture-of-vision-experts frame￾work, Mixpert, assigns each expert to a specific domain. The in￾put is routed to the expert most aligned with its type, incurring minimal computational overhead comp… view at source ↗
Figure 2
Figure 2. (Left) Multimodal data encompasses a variety of types, with images differing widely in content and structure. (Right) Building upon the joint SFT model, further domain-specific specialization can enhance performance on the respective task. This also demonstrates that joint optimization, due to balancing multiple objectives, limits the ability to achieve optimal performance in each individual domain. However, SFT on … view at source ↗
Figure 3
Figure 3. Architecture of Mixpert. We build a multi-expert component within the vision encoder, with each expert responsible for a specific task. This allows for specialization even when there are significant differences between domains, effectively mitigating potential conflicts. For images with ambiguous types or multiple characteristics, the router struggles to make a decisive prediction, thus directing them to the versati… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 17 canonical work pages

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  2. [2]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 4

    arXiv 2023

  3. [3]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 5

    arXiv 2015

  4. [4]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024. 2, 8

    arXiv 2024

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2, 8

    work page 2024

  6. [6]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 5

    arXiv 2010

  8. [8]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InIn- ternational Conference on Machine Learning, pages 5547–

    work page

Show all 59 references
  1. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  2. [10]

    Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.Journal of Machine Learning Re- search, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.Journal of Machine Learning Re- search, 23(120):1–39, 2022. 2

    2022

  3. [11]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023. 5, 6, 7, 8

    2023 arXiv

  4. [12]

    G-llava: Solving geometric prob- lem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wan- jun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric prob- lem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023. 4

    2023 arXiv

  5. [13]

    Mixture of cluster-conditional lora experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379, 2023

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379, 2023. 2, 4

    2023 arXiv

  6. [14]

    Vimoe: An empirical study of designing vision mixture-of-experts.arXiv preprint arXiv:2410.15732, 2024

    Xumeng Han, Longhui Wei, Zhiyang Dou, Zipeng Wang, Chenhui Qiang, Xin He, Yingfei Sun, Zhenjun Han, and Qi Tian. Vimoe: An empirical study of designing vision mixture-of-experts.arXiv preprint arXiv:2410.15732, 2024. 2

    2024 arXiv

  7. [15]

    In- corporating visual experts to resolve the information loss in multimodal large language models.arXiv preprint arXiv:2401.03105, 2024

    Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. In- corporating visual experts to resolve the information loss in multimodal large language models.arXiv preprint arXiv:2401.03105, 2024. 1, 2

    2024 arXiv

  8. [16]

    Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neu- ral computation, 3(1):79–87, 1991. 2, 3

    1991

  9. [17]

    Hierarchical mixtures of experts and the em algorithm.Neural computation, 6(2): 181–214, 1994

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm.Neural computation, 6(2): 181–214, 1994. 2, 3

    1994

  10. [18]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

  11. [19]

    Chart-to-text: A large-scale benchmark for chart sum- marization.arXiv preprint arXiv:2203.06486, 2022

    Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart sum- marization.arXiv preprint arXiv:2203.06486, 2022. 4

    2022 arXiv

  12. [20]

    Geomverse: A systematic evalu- ation of large models for geometric reasoning.arXiv preprint arXiv:2312.12241, 2023

    Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evalu- ation of large models for geometric reasoning.arXiv preprint arXiv:2312.12241, 2023. 4

    2023 arXiv

  13. [21]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–

    2016

  14. [23]

    Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020. 2

    2006 arXiv

  15. [24]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 3, 5, 7, 8

    2024 arXiv

  16. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 1, 2 9

    2023 arXiv

  17. [26]

    Cumo: Scaling multimodal llm with co-upcycled mixture- of-experts.arXiv preprint arXiv:2405.05949, 2024

    Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal llm with co-upcycled mixture- of-experts.arXiv preprint arXiv:2405.05949, 2024. 8

    2024 arXiv

  18. [27]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    arXiv

  19. [28]

    Moe-llava: Mixture of experts for large vision- language models.arXiv preprint arXiv:2401.15947, 2024

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- language models.arXiv preprint arXiv:2401.15947, 2024. 2, 4

    2024 arXiv

  20. [29]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models.arXiv preprint arXiv:2311.07575, 2023

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models.arXiv preprint arXiv:2311.07575, 2023. 1, 2

    2023 arXiv

  21. [30]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024. 2, 8

    2024 arXiv

  22. [31]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    arXiv

  23. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  24. [33]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 7

    2024

  25. [34]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 5, 6, 7

    2024

  26. [35]

    Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, pages 216–233. Springer, 2025. 5

    2025

  27. [36]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  28. [37]

    Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding.arXiv preprint arXiv:2403.05525,

    arXiv

  29. [38]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 5, 6, 7

    arXiv

  30. [39]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 4, 5, 6, 7, 8

    2022 arXiv

  31. [40]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 4, 5, 6, 7, 8

    2021

  32. [41]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 5

    2022

  33. [42]

    Scene text recognition using higher order language priors

    Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. InBMVC- British machine vision conference. BMV A, 2012. 5

    2012

  34. [43]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. InICDAR, 2019. 4

    2019

  35. [44]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 1

    2018

  36. [45]

    Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019. 1

    2019

  37. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  38. [47]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. InEuropean conference on computer vision, pages 146–162. Springer, 2022. 4

    2022

  39. [48]

    The sparsely-gated mixture-of-experts layer

    N Shazeer, A Mirhoseini, K Maziarz, A Davis, Q Le, G Hin- ton, and J Dean. The sparsely-gated mixture-of-experts layer. Outrageously large neural networks, 2017. 2

    2017

  40. [49]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 1, 8

    2024 arXiv

  41. [50]

    Vi- sualmrc: Machine reading comprehension on document im- ages

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Vi- sualmrc: Machine reading comprehension on document im- ages. InProceedings of the AAAI Conference on Artificial Intelligence, pages 13878–13888, 2021. 4

    2021

  42. [51]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024. 1, 2, 8

    2024 arXiv

  43. [52]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  44. [53]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, 10 Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  45. [54]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 8

    2024 arXiv

  46. [55]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 1, 2, 8

    2024 arXiv

  47. [56]

    Syntax-aware network for handwritten mathematical expression recognition.arXiv preprint arXiv:2203.01601, 2022

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition.arXiv preprint arXiv:2203.01601, 2022. 4, 5

    2022 arXiv

  48. [57]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1

    2023

  49. [58]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2025

  50. [59]

    Mixture-of-experts with expert choice routing.Ad- vances in Neural Information Processing Systems, 35:7103– 7114, 2022

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing.Ad- vances in Neural Information Processing Systems, 35:7103– 7114, 2022. 2

    2022

  51. [60]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 1, 2 11

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.