Pith. sign in

REVIEW 3 major objections 6 minor 56 references

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VisionFuse claims that fusing same-family MLLMs by concatenating vision tokens and merging delta parameters improves multimodal benchmark scores by over 4% on average, with no training.

desk verdict Training-free fusion of same-family MLLMs is a real and useful trick, but the headline gain is inflated by test-set-tuned hyperparameters. read the letter →

arxiv 2412.01289 v2 pith:GFCA7FEC submitted 2024-12-02 cs.CV cs.AI

Zhuokun Chen , Jinwu Hu , Zeshuai Deng , Yufeng Wang , Bohan Zhuang , Mingkui Tan This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodallargelanguagemodelstraining-freefusionvisionencoderensembledeltaparametermergingtaskarithmeticvisualperceptionMLLMfamilytokenconcatenation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VisionFuse tries to establish that the visual perception of multimodal large language models (MLLMs) can be improved without any training by combining off-the-shelf models that share the same pretrained language model. The paper claims that two operations are enough: concatenate the visual tokens produced by the different models' vision encoders, and merge the language models' delta parameters by weighted interpolation toward their shared base model. Fusing Mini-Gemini-8B and SLIME-8B this way raises average performance by more than 4% over the stronger individual model across nine multimodal benchmarks, at a cost of only 3.4% more parameters from the added encoder. If the claim holds, it matters because it sidesteps the expensive re-alignment step that normally accompanies any new or additional vision encoder.

What carries the argument

The load-bearing mechanism is a two-part, training-free pipeline. A vision ensemble runs each member's preprocessing, encoder, and projector to obtain visual tokens $V_i$, then feeds the concatenation $V_F = [V_1; \dots; V_n]$ into the language model alongside text tokens. In parallel, an LLM-merging step forms $\Theta_{\mathrm{merged}} = \Theta_{\mathrm{pre}} + \lambda \sum_{i=1}^M (\Theta_i - \Theta_{\mathrm{pre}})$, the task-arithmetic rule of adding scaled fine-tuning deltas to the shared base model. The identity that carries the argument is this linear interpolation: it is what aligns one language model with several vision encoders that were never jointly trained, and its validity is what confines the method to a single MLLM family.

What would settle it

Take two same-family MLLMs with deliberately divergent visual preprocessing, for example one crop-based and one patch-based, and large pairwise delta differences; if a tuned $\lambda$ cannot keep TextVQA accuracy above the better individual model, the compatibility premise fails. The paper's own cross-family result, MGM-8B + VILA-8B dropping TextVQA from 67.6 to 33.0, is a partial falsifier; a systematic scan across same-family pairs with measured delta distances would settle how far the claim extends.

Watch

Extended reading notes

Core claim

The central discovery is that within an MLLM family, defined as models built on the same pretrained LLM, two previously separate techniques become compatible: visual-token concatenation and delta-parameter merging. The paper's three observations are that different MLLMs attend to different image regions for the same query, that same-family vision encoders produce feature distributions similar enough to be concatenated into one visual context, and that interpolating the language models' deltas, $\Theta_{\mathrm{merged}} = \Theta_{\mathrm{pre}} + \lambda \sum_i (\Theta_i - \Theta_{\mathrm{pre}})$, lets a single LLM parse tokens from multiple encoders. The authors show that on TextVQA the interpolated model peaks near $\alpha = 0.5$ between the two 8B models, and ablation evidence indicates the gain comes from genuinely richer visual information rather than from longer token sequences, since duplicating one model's tokens does not improve accuracy.

Load-bearing premise

The load-bearing premise is that linearly averaging the fine-tuning differences of same-family models produces a single language model that can interpret every member's visual tokens without any alignment training; the paper establishes this empirically from one family pair and observes that it fails when deltas are large or models come from different families.

Editorial extensions

If this is right

  • Fusing MGM-8B and SLIME-8B outperforms both individual models on every benchmark reported, and reaches parity with MGM-8x7B, a model with over six times as many parameters.
  • Combining a high-resolution encoder (MGM-HD-8B) with a low-resolution one (SLIME-8B) still improves overall accuracy, so the low-resolution encoder supplies complementary information rather than redundancy.
  • Both components are necessary: merging deltas without extra encoders gives small gains, while adding extra encoders without merging deltas hurts alignment and accuracy.
  • Naively concatenating tokens from more than two 7B models degrades performance because sequence lengths exceed the training distribution, but the same three-model fusion on 8B models, or with about 1,000 tokens pruned, improves over the two-model fusion.
  • The recipe transfers beyond the MGM-SLIME pair, improving MGM+Eagle-8B across TextVQA, MMBench, MMMU, Ai2d, OCRBench, and RealworldQA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If same-family compatibility is the active ingredient, then the useful resource is families of models sharing a base LLM; quantitative measures of delta-parameter distance could predict which pairs will fuse well before running benchmarks.
  • The success of simple $\lambda$-scaled delta interpolation suggests the visual-alignment knowledge in these MLLMs is concentrated in small parameter changes; testing sparse-merging variants, such as pruning or conflict-resolving merges, may make the fusion more robust and cheaper to deploy.
  • The token-length failure mode points to a natural extension the paper leaves open: instead of concatenating all tokens, select a per-encoder subset of the most informative or complementary visual tokens, which could preserve gains while cutting inference cost.
  • Because the authors' cross-family experiments show sharp drops, the method doubles as a probe: a family's merge success rate could serve as an empirical measure of how much two MLLM fine-tunings share in their visual alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. VisionFuse is a training-free framework that improves the visual perception of multimodal LLMs by integrating multiple MLLMs that share the same pretrained language model. The method concatenates visual tokens from the vision encoders of several family members and merges their LLM delta parameters as Θₘₑᵣ₉ₑₐ = Θₚᵣₑ + λ·Σ(Θᵢ − Θₚᵣₑ). The central claim is that, without additional training, fusing MiniGemini-8B and SLIME-8B yields an average relative improvement of over 4% across nine multimodal benchmarks, with ablations showing that both token concatenation and LLM merging are necessary. The paper also reports results for 7B models, a high-resolution variant, an Eagle-8B combination, and three-model integration with token pruning.

Significance. If the reported gains are robust, VisionFuse offers a practical, low-overhead route to improving MLLM perception by reusing existing checkpoints, and it provides useful empirical evidence about the compatibility of vision encoders and LLM deltas within an MLLM family. The paper's strengths include consistent improvements across a wide benchmark suite, explicit ablations isolating the two components (Table 2), a negative control showing that simply duplicating tokens does not help (Figure 5), and a striking cross-family failure (Table 9) that supports the family-condition hypothesis. The main reservation is the selection of the merging coefficient λ on the evaluation benchmarks themselves, which affects the credibility of the headline quantitative claim.

major comments (3)
  1. [Appendix A.2, Eq. (4), Figure 4c, Table 1] The merging coefficient λ is selected by maximizing accuracy on TextVQA (Figure 4c), and the hyperparameter search in Table 5 is performed over the same benchmarks that are later averaged in Table 1. The paper does not state that a held-out validation split is used. Because TextVQA is one of the nine benchmarks in the headline average, the reported "over 4%" improvement is partly a consequence of test-set selection. Please either choose λ on a held-out validation set (e.g., a few tasks not in the evaluation suite) or report the full benchmark results across the entire λ range to show that the average improvement is not an artifact of peaking on TextVQA.
  2. [Section 5, Table 1] The average "increase of over 4%" is an arithmetic mean of per-benchmark relative gains that range from +1.0% (MMBench) to +14.7% (Vizwiz), with MME scores and accuracy percentages mixed in the same average. With λ tuned on one of these benchmarks, the mean is not a stable statistic. Please clarify exactly how the average is computed, report the median and the range, and, ideally, provide bootstrap confidence intervals or a sensitivity table across λ to establish the stability of the central claim.
  3. [Appendix D.2, Figure 8c, Table 7] The three-model integration result relies on randomly dropping 1000 visual tokens, and this drop count is selected by maximizing TextVQA accuracy in Figure 8c. This is another free parameter chosen on the evaluation set. The three-model improvement over the two-model baseline should therefore be reported with the same caveat, and a sensitivity analysis over the number of dropped tokens should be provided so that the reader can gauge the robustness of that result.
minor comments (6)
  1. [Section 3, title] The heading "Empircal Insights" appears to be a typo and should read "Empirical Insights."
  2. [Table 3, caption] The word "Comparasions" in the caption should be "Comparisons."
  3. [Appendix D.1] The sentence "This limitation leeds to degraded performance" contains a typo; "leeds" should be "leads."
  4. [Section 5, paragraph on main results] The phrase "in both the the combinations" contains a duplicated article and should be "in both the combinations."
  5. [Figure 4c] Please clarify in the caption whether the accuracy curve corresponds to the merged model with both vision encoders or to a single-encoder model; the text is ambiguous about what is being interpolated.
  6. [Algorithm 1] The comment "Merge language model parameters using Eq. (4)" is placed as a step; consider making it a formal numbered step to match the algorithmic description in the text.

Circularity Check

1 steps flagged · score 4.0 of 10

The reported TextVQA gain is not clearly out-of-sample: the merge coefficient α/λ is selected using TextVQA accuracy, and the same TextVQA column is then part of the headline 4% average; no validation split is described.

  1. fitted input called prediction [Section 3, Observation 3 / Figure 4c; Section 4.3 Eq. (4); Appendix A.2 Table 5; Section 5 Table 1]
    "(c) Accuracy on TextVQA when interpolating MGM and SLIME with different α. … Table 5 shows the searching range of the parameters of sev-eral merging methods."

    The method's only free constant is λ in Eq. (4), Θmerged = Θpre + λ·Σ(Θi − Θpre). For Task Arithmetic, the paper searches λ over {0.1, 0.3, 0.5, 0.7, 0.9, 1.0}, and Observation 3 explicitly selects α by maximizing accuracy on TextVQA. The same TextVQA benchmark then appears in Table 1 as a +3.6% gain for MGM-SLIME-8B and is included in the 'over 4%' average improvement. Because the paper never states that this search was performed on a validation split, the reported TextVQA number is not an independent prediction: it is the result of choosing the coefficient on that very benchmark. At least this component of the headline improvement is a fitted input presented as an achieved result, so the magnitude of the claimed gain is not established by the paper as written.

full rationale

VisionFuse is an empirical integration method rather than a derived first-principles result: token concatenation and delta-parameter interpolation are taken from prior external work (Eagle, Task Arithmetic), and the headline performance is measured on standard external MLLM benchmarks. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no equation that defines the claimed output in terms of its own inputs. Observations 1-3 are empirical motivations collected from the very models being fused; they are descriptive rather than independent evidence, but they do not constitute a circular derivation. The one defensible circularity-adjacent step is the selection of α/λ using TextVQA accuracy while TextVQA is then reported as a benchmark gain, with no validation split described. That undermines the strength of the reported improvement on that benchmark and slightly inflates the 'over 4%' average. The central comparison against individual MLLMs nevertheless retains independent external content, so the paper is only partially compromised rather than circular by construction.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claim depends on one fitted scalar λ and the empirical assumption that shared-base MLLMs have compatible visual features and compatible deltas. No new physical or architectural entities are introduced.

free parameters (2)
  • merging coefficient λ = searched over {0.1, 0.3, 0.5, 0.7, 0.9, 1.0}; best on TextVQA
    Appears in Eq. (4) and controls the weight of the summed delta parameters; chosen by search on evaluation benchmarks (Appendix A.2), so the reported gains partly reflect this selection.
  • token drop count for three-model integration = 1000 tokens
    Appendix D.2: the number of randomly dropped visual tokens is chosen based on TextVQA accuracy, and affects only the three-model integration results.
assumptions (2)
  • domain assumption Linear interpolation of delta parameters (Task Arithmetic) preserves fine-tuned capabilities and can combine them into one model
    Used in Eq. (4) for the language model merge; follows [19] but extends the claim to multimodal alignment, where the merged model must interpret tokens from multiple encoders.
  • domain assumption Visual tokens from encoders of MLLMs sharing the same base LLM are feature-space compatible for direct concatenation
    Observation 2 (Figure 4b) shows t-SNE overlap, but joint usability is only validated empirically on benchmarks; the paper does not provide a formal bound or analytic justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion." pith.science (2026). https://pith.science/paper/GFCA7FEC

@misc{pith2026241201289,
  author       = {Pith},
  title        = {Pith review of: Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFCA7FEC}},
  note         = {Machine review of arXiv:2412.01289}
}
read the original abstract

Multimodal LLMs (MLLMs) equip language models with visual capabilities by aligning vision encoders with language models. Existing methods to enhance the visual perception of MLLMs often involve designing more powerful vision encoders, which requires exploring a vast design space and re-aligning each potential encoder with the language model, resulting in prohibitively high training costs. In this paper, we introduce VisionFuse, a novel integration framework that efficiently utilizes multiple vision encoders from off-the-shelf MLLMs to enhance visual perception without requiring additional training. Our approach is motivated by the observation that different MLLMs tend to focus on distinct regions given the same query and image. Moreover, we find that the feature distributions of vision encoders within an MLLM family, a group of MLLMs sharing the same pretrained LLM, are highly aligned. Building on these insights, VisionFuse enriches the visual context by concatenating the tokens generated by the vision encoders of selected MLLMs within a family. By merging the parameters of language models from these MLLMs, VisionFuse allows a single language model to align with various vision encoders, significantly reducing deployment overhead. We conduct comprehensive evaluations across multiple multimodal benchmarks using various MLLM combinations, demonstrating substantial improvements in multimodal tasks. Notably, when integrating MiniGemini-8B and SLIME-8B, VisionFuse achieves an average performance increase of over 4%.

Figures

Figures reproduced from arXiv: 2412.01289 by the authors.

Figure 1
Figure 1. To enhance the perception capabilities of MLLMs, ex [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Different MLLMs exhibit varying visual perception capabilities. We visualize the average cross-attention maps across all layers for two MLLMs - MGM and SLM, as well as for our method that integrates these two models, using an example to observe which areas the models focus on. It shows that our VisionFuse attention is more accurate, integrating the perceptual abilities of both MGM and SLM. Here, “MGM” represents Min… view at source ↗
Figure 3
Figure 3. Overview of VisionFuse. VisionFuse merges the language model parameters from [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Summary of our exploration and observations: (a) demonstrates that different MLLMs focus on distinct image regions for the [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparision with directly dupli￾cating original tokens many times. MGM SLIME ConcatAdd global Intepolate-add 62 64 66 68 70 Accuaracy(%) on TextVQA 67.6 64.8 70.0 67.8 64.5 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Exploration of the impact of sequence length: (a) demonstrates the significant performance degradation caused by visual se [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Exploration of redundancy in visual tokens: (a) and (b) examine the redundancy of visual tokens in SLIME-7B and LLaVA [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Enhancement of the perception regions. What letters are on the hat? MGM G. SLM Gi. Ours SF. What's the authors initials? Ours J.L. MGM J.K. SLM G.F [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Enhancement of fine-grained perception 7 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Enhancement of colors perception Please describe this image in as much detail as possible. The image captures a variety of fresh produce and food items arranged on a wooden cutting board. The produce includes tomatoes, basil, mushrooms, and broccoli, while the food it…
Figure 12
Figure 12. Figure 12: A sample for detailed descriptions. 8 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: A sample for detailed descriptions. 9 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: A sample for detailed descriptions. 10 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: A sample for detailed descriptions. 11 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: A sample for detailed descriptions. 12 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 24 canonical work pages

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. 2, 3, 4, 6

    work page 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35: 23716–23736, 2022. 1

    2022

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 6

    arXiv 2023

  4. [4]

    To- ken merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. To- ken merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. 3

    arXiv 2022

  5. [5]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024. 1, 3

    work page 2024

  6. [6]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 6

    arXiv 2023

  7. [7]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024. 6, 3, 4

    work page 2024

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march

Show all 56 references
  1. [9]

    Fusing finetuned models for better pretraining

    Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044, 2022. 8

    2022 arXiv

  2. [10]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 6

    2023 arXiv

  3. [11]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  4. [12]

    Epsilon sampling rocks: Investigating sampling strategies for minimum bayes risk decoding for machine translation

    Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investigating sampling strategies for minimum bayes risk decoding for machine translation. arXiv preprint arXiv:2305.09860, 2023. 2

    2023 arXiv

  5. [13]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 4, 6

    2024

  6. [14]

    Con- vllava: Hierarchical backbones as visual encoder for large multimodal models

    Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, and Bo Zheng. Con- vllava: Hierarchical backbones as visual encoder for large multimodal models. arXiv preprint arXiv:2405.15738, 2024. 3

    2024 arXiv

  7. [15]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, pages 6904–6913, 2017. 4

    2017

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, pages 6904–6913, 2017. 6

    2017

  9. [17]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018. 6

    2018

  10. [18]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xi- angyu Yue. Onellm: One framework to align all modalities with language. In CVPR, pages 26584–26595, 2024. 1

    2024

  11. [19]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2022. 5, 6, 7, 8, 1

    2022

  12. [20]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco T ´ulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR. OpenRe- view.net, 2023. 4

    2023

  13. [21]

    Llm- blender: Ensembling large language models with pair- wise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm- blender: Ensembling large language models with pair- wise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023. 2

    2023 arXiv

  14. [22]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2022. 4

    2022

  15. [23]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235–251. Springer,

  16. [24]

    In- troducing idefics: An open reproduction of state-of-the-art visual language model, 2023

    Hugo Laurencon, Daniel van Strien, Stas Bekman, Leo Tron- chon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. In- troducing idefics: An open reproduction of state-of-the-art visual language model, 2023. 6

    2023

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 1

    2023

  18. [26]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 5, 6

    2023 arXiv

  19. [27]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  20. [28]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1

    2023 arXiv

  21. [29]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pages 26689–26699, 2024. 1, 6, 4

    2024

  22. [30]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 8

    2014

  23. [31]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In CVPR,

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 6

    2024

  25. [33]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 3, 1

    2024

  26. [34]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 1

    2024

  27. [35]

    Bitdelta: Your fine-tune may only be worth one bit

    James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. arXiv preprint arXiv:2402.10193, 2024. 5

    2024 arXiv

  28. [36]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 6

    2023 arXiv

  29. [37]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 1

    2023 arXiv

  30. [38]

    An empirical study of scal- ing instruct-tuned large multimodal models

    Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jian- feng Gao, and Yelong Shen. An empirical study of scal- ing instruct-tuned large multimodal models. arXiv preprint arXiv:2309.09958, 2023. 1

    2023 arXiv

  31. [39]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 3

    2021

  32. [40]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 1, 2, 3

    2024 arXiv

  33. [41]

    Animating rotation with quaternion curves

    Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques , pages 245–254, 1985. 4, 8

    1985

  34. [42]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 4, 6

    2019

  35. [43]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9 (11), 2008. 4

    2008

  36. [44]

    Knowledge fusion of large language models

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491, 2024. 2

    2024 arXiv

  37. [45]

    Grok-1.5 Vision Preview, 2024

    xAI. Grok-1.5 Vision Preview, 2024. Accessed: November

    2024

  38. [46]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 1

    2024 arXiv

  39. [47]

    Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. arXiv preprint arXiv:2306.01708 , 2023. 4, 5, 8, 1

    2023 arXiv

  40. [48]

    Law of vision represen- tation in mllms

    Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. Law of vision represen- tation in mllms. arXiv preprint arXiv:2408.16357, 2024. 2, 6

    2024

  41. [49]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 1

    2023 arXiv

  42. [50]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 5

    2023 arXiv

  43. [51]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024. 4, 8, 1

    2024

  44. [52]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In CVPR, pages 9556–9567, 2024. 6

    2024

  45. [53]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 1

    2023 arXiv

  46. [54]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 1, 2, 3, 4, 6

    2024 arXiv

  47. [55]

    Svit: Scaling up visual instruction tuning

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023. 1

    2023 arXiv

  48. [56]

    Sparsity

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1 10 Appendix Contents A . Details of Experimental Settings 1 A.1 . Details of ...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.