Pith. sign in

REVIEW 3 major objections 5 minor 73 references

LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LaVi matches LLaVA-class accuracy by injecting vision-conditioned deltas into LayerNorm's scale and shift parameters, eliminating visual tokens from self-attention at a 94% FLOP reduction.

desk verdict A genuinely useful architecture trick that deserves a close look, but the paper's headline efficiency numbers are not yet auditable and the 'state-of-the-art' wording outstrips its own tables. read the letter →

arxiv 2506.16691 v1 pith:GM2RMMLV submitted 2025-06-20 cs.CV

Tongtian Yue , Longteng Guo , Yepeng Tang , Zijia Zhao , Xinxin Zhu , Hua Huang , Jing Liu This is my paper
classification cs.CV
keywords vision-languagemodelsfeaturemodulationinjectionlayernormalizationinferenceefficiencyvideounderstandingmultimodalintegrationvisualtokencompressionreal-timereasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a 7-billion-parameter vision-language model can match the accuracy of LLaVA-style baselines on fifteen image and video benchmarks without ever placing visual tokens into the language model's self-attention. Instead of concatenating visual tokens into the context or inserting cross-attention layers, LaVi conditions the scale and shift parameters of selected layer-normalization layers on visual features, so each text token's hidden state is modulated by a vision-dependent delta. If the claim holds, the quadratic cost of long visual contexts is bypassed: high-resolution images and long videos cost almost the same as text-only sequences. The reported trade-off is a 94.0% reduction in FLOPs, a 3.1x inference speedup, and a memory cut roughly in half, at accuracy parity on average across the benchmarks.

What carries the argument

The central object is Vision-Infused Layer Normalization (ViLN), an extension of standard LayerNorm in which the affine parameters are shifted by vision-conditioned deltas: $ViLN(t,v)=(\alpha+\Delta\alpha_v)\odot \hat{t}+(\beta+\Delta\beta_v)$. The deltas come from a conditioning module that reads the text token and the visual features—three variants are tested, MLP-based, convolution-based, and attention-based—followed by a Swish-activated projection that is zero-initialized, so the model starts as the original LLM and learns to steer the hidden states with visual information. Only 25% of the LLM's layers are modulated, evenly spaced, which the ablations show is the best frequency. Because no visual token enters self-attention, the sequence length stays equal to the text length, and the computational savings grow with the number of visual tokens that would otherwise be fed in.

What would settle it

Run LaVi at matched resolution on a dense-OCR benchmark such as TextVQA or on a spatial-localization task with many tiles against the same-backbone LLaVA-OV: if accuracy falls several points and the gap widens as the number of visual tiles increases, the affine modulation channel is provably lossy and the parity claim does not generalize.

Watch

Extended reading notes

Core claim

The central claim is that token-wise affine modulation of LayerNorm is a complete enough channel for vision-language alignment that a 7B LLM can answer image and video questions as well as the same LLM given explicit visual tokens. Concretely, LaVi replaces a quarter of the LLM's LayerNorm modules with Vision-Infused Layer Normalization, $ViLN(t,v)=(\alpha+\Delta\alpha_v)\odot \hat{t}+(\beta+\Delta\beta_v)$, where the deltas are produced by a lightweight conditioning module (default: cross-attention of the text token over visual features) and a zero-initialized projection so that the pre-trained LLM's forward pass is initially unchanged. Trained on the same data as LLaVA-v1.5, LLaVA-v1.6, and LLaVA-OneVision respectively, the three LaVi variants match or slightly exceed the average accuracy of those baselines while shrinking the number of tokens the LLM processes to the text alone.

Load-bearing premise

The load-bearing premise is that a token-wise scale-and-shift of LayerNorm outputs can carry all the visual detail the LLM needs, so nothing essential is lost by keeping visual tokens out of self-attention.

Editorial extensions

If this is right

  • High-resolution images and long videos no longer cost quadratic compute: the paper reports that at 128 frames, LaVi saves 92.0% of FLOPs and 61.1% of memory compared with Video-LLaVA at the same frame count.
  • A 7B vision-language model becomes a real-time system: single-image latency drops from 612.5 ms (LLaVA-OV-7B) to 198.1 ms on one A100, and FLOPs fall below those of LLaVA-OV-0.5B while average accuracy rises by 15.5 points.
  • Language-only capability degrades less than with cross-attention or hyper-attention injection, since the base LLM's forward pass is preserved at initialization and feature drift stays small.
  • Video understanding scales with sampled frames in near-linear cost: going from 8 to 32 frames adds modest FLOPs while improving performance on all six video benchmarks, with the 32-frame model roughly matching LLaVA-OV's video accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the modulation channel is as lossless as parity suggests, then visual tokens could be dropped after conditioning, meaning the LLM only ever processes text; a direct test would be measuring whether downstream accuracy survives deleting the visual KV cache.
  • The success of 25% evenly-spaced modulation hints that LayerNorm is a low-dimensional control surface for the LLM; one could estimate the rank of the delta matrices to see how many independent visual dimensions are actually transmitted.
  • The mechanism is modality-agnostic: the same affine modulation could carry audio, depth, or structured table features, so the efficiency gain may transfer to other long-context multimodal settings.
  • A testable scaling prediction is that fine-grained tasks with dense spatial detail (e.g., OCR-heavy images) will deviate from parity as the number of tiles grows, exposing the information bottleneck of the affine channel.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces LaVi, a 7B-scale LVLM that fuses visual information into an LLM by replacing selected LayerNorm layers with Vision-Infused Layer Normalization (ViLN), which injects token-wise vision-conditioned scale and shift deltas. The authors claim that this Feature Modulation Injection (FMI) avoids both architectural disruption and long visual contexts, achieving accuracy comparable to LLaVA-style models while reducing FLOPs by 94%, improving inference speed by 3.1x, and halving memory consumption. The paper presents three conditioning variants (MLP-based, Conv-based, attention-based), ablations over modulation sublayers, parameters, frequency, and location, and evaluations on 9 image and 6 video benchmarks, alongside a controlled comparison of injection paradigms under identical data and backbone settings.

Significance. If the measured efficiency numbers are reproducible, the contribution is significant: it demonstrates a practical alternative to in-context visual token injection, with a structural FLOP advantage that follows from keeping the text-only sequence length in self-attention. The controlled comparison in Table 4, where all injection paradigms share the same data and backbone, is the strongest part of the paper and supports the accuracy/efficiency trade-off claim. The paper also provides useful ablations and initial evidence on linguistic-prior preservation. The main caveats are that the headline efficiency table lacks input specifications and the 'state-of-the-art' wording is not backed by per-benchmark results; these are fixable reporting issues rather than fundamental flaws, but they currently prevent verification of the paper's central efficiency claim.

major comments (3)
  1. [§4.1, Tables 1 and 2; §A.3] The central efficiency claim is not auditable. The FLOPs and latency numbers in Table 1 (e.g., LLaVA-OV at 60.4 T / 612.5 ms vs. LaVi at 3.6 T / 198.1 ms) and Table 2 (e.g., LLaVA-OV at 129.6 T / 1215.6 ms vs. LaVi at 18.6 T / 401.5 ms) are reported without the input specification used for profiling: image resolution, number of dynamic-resolution tiles, text sequence length, number of generated tokens, and whether the vision encoder and conditioning module are included in the measurements. Section A.3 only states that DeepSpeed FLOPs profiling on one A100 was used. Without these settings, the 94.0% FLOP reduction and 3.1x speedup may be an artifact of different input configurations rather than an architectural fact. Please provide the exact profiling configuration for every model in Tables 1 and 2, and ideally release the profiling script.
  2. [Abstract, §1, §4.2 (Tables 1 and 2)] The 'state-of-the-art' and 'consistently surpasses or matches' claims are stronger than the data. In Table 1, LaVi (Qwen2-7B) trails LLaVA-OV on VQAv2 (84.0 vs. 84.5), ScienceQA-IMG (95.4 vs. 96.0), POPE (87.1 vs. 87.4), and MMBench (79.3 vs. 80.8), and the average advantage is only +0.5 (77.7 vs. 77.2) with no significance testing or variance estimates. In Table 2, LaVi at 32 frames trails LLaVA-OV on EgoSchema, MLVU, VideoMME, and Video-ChatGPT. Please either support the superiority claims with statistical evaluation or reword them to 'competitive accuracy' throughout the abstract, introduction, and conclusion.
  3. [§3.3, Table 4] The controlled comparison in Table 4 is a strength, but the in-context injection baseline is a custom 'Concat' model rather than LLaVA-OV or LLaVA-v1.5, and it is trained on a 4M subset of the pretraining data plus LLaVA-665K. The authors should clarify whether this baseline uses a comparable connector and whether the same data schedule and backbone are used for all rows. Without this clarification, readers cannot tell how much of the reported FLOPs and accuracy difference between FMI and in-context injection is due to the injection paradigm versus differences in the baseline setup.
minor comments (5)
  1. [Appendix B] The PyTorch-style pseudocode for the MLP-based conditioning module is incorrect: in the forward method, the line 'y = self.mlp_channel(y)' uses an undefined variable y, and should operate on x (the channel-mixed output of the token-mixing stage). Additionally, the attention-based conditioning module references 'self.dk' without defining it in __init__; please fix these code-level typos.
  2. [§A.1 and §4.3] There are several typos: the appendix heading 'Trianing Details' should be 'Training Details', and Table 5's discussion says 'both addictive and multiplicative operations' where 'addictive' should be 'additive'.
  3. [§3.3, Eq. (7)] The zero-initialization description is ambiguous: the text says the projection is zero-initialized so that the initial deltas are zero, but it is not clear whether this refers to the final linear layer (W, b) only, or to the entire conditioning module, and whether the same initialization is used in both training stages. Please clarify.
  4. [§A.3] The paper says code and models will be released soon, but no link or release venue is given. Since the main efficiency claims cannot be independently checked without the profiling script and the trained models, please add a release link or state an expected release date.
  5. [Figure 1] The red dashed line is described as a linear fit to all models except LaVi, but no fit equation or R² is reported. Please either report the fit statistics or remove the line, as the current presentation does not convey quantitative information.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: architecture and benchmark claims are empirically grounded, with only an auditability caveat on FLOPs/latency settings.

full rationale

The paper makes empirical architecture claims: LaVi's accuracy is measured on 15 external benchmarks, and its efficiency is measured by FLOPs, latency, and memory. The core formula ViLN(t,v) = (alpha + delta_alpha_v) * t_hat + (beta + delta_beta_v) (Eq. 6) is a proposed mechanism, not derived from the target results. The reported 94.0% FLOP reduction and 3.1x speedup follow structurally from removing visual tokens from self-attention; that is the intended architectural comparison, not a fitted input renamed as a prediction. There is no self-citation chain used to justify the central claim, no imported uniqueness theorem, and no quantity is fit to a subset of benchmarks and then called a prediction on those benchmarks. The main caveat is that the paper does not state the input settings behind Table 1 (resolution, number of tiles, text sequence length, generated tokens, inclusion of the vision encoder), so the efficiency ratios are not fully auditable; this is a correctness or reproducibility concern rather than circularity. Consistent with the default expectation for an empirical architecture paper, no significant circularity is found.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The paper is empirical, so the ledger mostly records design choices that affect benchmark interpretation: the frozen pretrained LLM and vision encoder are inputs from prior work, while the ViLN frequency and conditioning type are tuned on the same evaluation suite. No free constants are derived from first principles.

free parameters (2)
  • ViLN modulation frequency = 25% of transformer layers
    Table 6 selects 25% because it gives the best average score on the evaluation benchmarks; the final model uses this tuned value, so the benchmark results are not fully out-of-sample for this architectural choice.
  • Conditioning module type = Attention-based by default
    Among MLP, Conv, and Attention conditioning, the attention variant is chosen as default after comparison on the same benchmark suite (Table 4), which introduces a mild selection effect.
assumptions (3)
  • domain assumption LayerNorm is the appropriate and sufficient intervention point for injecting visual information into a pretrained LLM without destroying linguistic priors.
    Section 3.2 justifies the choice via the ubiquity of LayerNorm, but it is an assumption that affine modulation can carry the visual information needed for complex tasks.
  • domain assumption A token-wise conditioning module can compress the entire visual input into per-token deltas without information loss that is critical for the target tasks.
    This is the information bottleneck assumption behind Eq. 6 and the whole architecture; it is tested empirically but not derived.
  • domain assumption Quadratic self-attention cost is the dominant efficiency bottleneck, so removing visual tokens from the context is the correct way to compare efficiency.
    The FLOP and latency comparisons assume that the computational cost of the conditioning module and vision encoder is small relative to LLM self-attention, and that excluding visual tokens from the LLM is a fair architectural trade-off.
invented entities (1)
  • Vision-Infused Layer Normalization (ViLN)
    purpose: Injects token-wise vision-conditioned deltas into LayerNorm scale and shift parameters to modulate LLM hidden states without extending the context length.
    ViLN is a new module proposed by the paper; its support comes from the paper's own ablations and benchmarks. No external replication or released implementation exists yet, so independent evidence is not yet available.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation." pith.science (2026). https://pith.science/paper/GM2RMMLV

@misc{pith2026250616691,
  author       = {Pith},
  title        = {Pith review of: LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GM2RMMLV}},
  note         = {Machine review of arXiv:2506.16691}
}
read the original abstract

Despite the impressive advancements of Large Vision-Language Models (LVLMs), existing approaches suffer from a fundamental bottleneck: inefficient visual-language integration. Current methods either disrupt the model's inherent structure or introduce severe long-context computational burden, severely limiting scalability and efficiency. In this paper, we rethink multimodal integration and present LaVi, a novel LVLM that enables seamless and efficient vision-language fusion through internal feature modulation within the Large Language Models (LLMs). Unlike dominant LVLMs that rely on visual token concatenation, LaVi bypasses long-context expansion by introducing a lightweight and adaptive transformation, which incorporates visual context by injecting token-wise vision-conditioned deltas into the affine parameters of layer normalization. This mechanism directly modulates linguistic hidden states based on visual input, ensuring precise vision-language alignment while preserving the LLM's linguistic priors and drastically reducing computational costs. Extensive evaluations across 15 image and video benchmarks demonstrate that LaVi not only achieves state-of-the-art multimodal performance but also dramatically enhances efficiency. Compared to LLaVA-OV-7B, LaVi reduces FLOPs by 94.0%, improves inference speed by 3.1 times, and cuts memory usage in half - establishing LaVi as a scalable and practical solution for real-time multimodal reasoning. The code and models will be released soon.

Figures

Figures reproduced from arXiv: 2506.16691 by the authors.

Figure 1
Figure 1. Comparison between LaVi and open-source LVLMs on image understanding benchmarks. We re￾port the average accuracy on MMBench [42], MME [19], TextVQA [53], and GQA [25]. For MME, scores are nor￾malized to percentages. The red dashed line represents the linear fit to all models except LaVi. Benefiting from a significantly reduced context length and a lightweight yet ef￾fective visual-language integration strat￾egy, LaV… view at source ↗
Figure 2
Figure 2. Comparisons of various vision integration techniques for LVLMs. (a) Architectural injection: additional layers are inserted into LLM for cross-modal interaction; (b) In-context injection: visual tokens are concatenated before the text sequence as the initial context. (c) Feature modulation injection (Ours): the internal hidden states are modulated by the vision-guided affine transformation. pretrained LLMs. Such mod… view at source ↗
Figure 3
Figure 3. An illustrative diagram of the overall model architecture. For a LLM block equipped with ViLN, visual and textual features are fed into the conditioning module to obtain token-wise visual conditions. Through a lightweight MLP, these conditions are then transformed into scale and shift parameters, which modulate the internal linguistic features of the LLM. 3.3 LaVi: A Highly Efficient LVLM The overall architecture of… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Feature drifts com￾pared with base LLM. Our method preserves best linguistic capabilities. 1 5 10 14 19 23 28 Layer 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Cosine Distance Mean Min-Max Range [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 9
Figure 9. Figure 9: Memory compari￾son across frame counts. LaVi achieves significant Memory re￾duction across all frames. 4.4 Visualization and Analysis Effective Preservation of Linguistic Capabilities. For the baselines compared in [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Implementation of three conditioning modules in PyTorch. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 10 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning.ArXiv, abs/2204.14198, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, R...

    arXiv 2022

  3. [3]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018

    2018

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 2024

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 2024

    2024

  5. [5]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  6. [6]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arX...

    arXiv 2023

  7. [7]

    Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Jimmy Lei Ba. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  8. [8]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

Show all 73 references
  1. [9]

    Nltk: the natural language toolkit

    Steven Bird. Nltk: the natural language toolkit. InProceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72, 2006

    2006

  2. [10]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021

    2021

  3. [11]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  4. [12]

    Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024. 10

    2024 arXiv

  5. [13]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InEuropean conference on computer vision, pages 104–120. Springer, 2020

    2020

  6. [14]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  7. [15]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  8. [16]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  9. [17]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  10. [18]

    Mobilevlm v2: Faster and stronger baseline for vision language model.arXiv preprint arXiv:2402.03766, 2024

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model.arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  11. [19]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  12. [20]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  13. [21]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  14. [22]

    Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. InEuropean Conference on Computer Vision, pages 390–406. Springer, 2024

    2024

  15. [23]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018

    2018

  16. [24]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  17. [25]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  18. [26]

    Revisiting visual question answering baselines

    Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Revisiting visual question answering baselines. InEuropean conference on computer vision, pages 727–739. Springer, 2016. 11

    2016

  19. [27]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  20. [28]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  21. [29]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  22. [30]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  23. [31]

    Align before fuse: Vision and language representation learning with momentum distillation.Advances in neural information processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation.Advances in neural information processing systems, 34:9694–9705, 2021

    2021

  24. [32]

    Videochat: Chat-centric video understanding, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2023

    2023

  25. [33]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  26. [34]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  27. [35]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  28. [36]

    Moe-llava: Mixture of experts for large vision-language models.arXiv preprint arXiv:2401.15947, 2024

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models.arXiv preprint arXiv:2401.15947, 2024

    2024 arXiv

  29. [37]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  30. [38]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    2024 arXiv

  31. [39]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  32. [40]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  33. [41]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  34. [42]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  35. [43]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.arXiv preprint arXiv:1908.02265, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.arXiv preprint arXiv:1908.02265, 2019. 12

    1908 arXiv

  36. [44]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  37. [45]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training.arXiv preprint arXiv:2410.08202, 2024

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training.arXiv preprint arXiv:2410.08202, 2024

    arXiv

  38. [46]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  39. [47]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  40. [48]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.Meta AI Blog

    AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.Meta AI Blog. Retrieved December, 20:2024, 2024

    2024

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  42. [50]

    Language models are unsupervised multitask learners.OpenAI blog, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 2019

    2019

  43. [51]

    Searching for activation functions.arXiv preprint arXiv:1710.05941, 2017

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions.arXiv preprint arXiv:1710.05941, 2017

    2017 arXiv

  44. [52]

    Cinepile: A long video question answering dataset and benchmark.arXiv preprint arXiv:2405.08813, 2024

    Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark.arXiv preprint arXiv:2405.08813, 2024

    2024 arXiv

  45. [53]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  46. [54]

    Flops profiler, 2025

    DeepSpeed Team. Flops profiler, 2025. Accessed: 2025-02-11

    2025

  47. [55]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  48. [56]

    Mlp- mixer: An all-mlp architecture for vision.Advances in neural information processing systems, 34:24261–24272, 2021

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mlp architecture for vision.Advances in neural information processing systems, 34:242...

    2021

  49. [57]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    2024 arXiv

  50. [58]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  51. [59]

    Patches are all you need?arXiv preprint arXiv:2201.09792, 2022

    Asher Trockman and J Zico Kolter. Patches are all you need?arXiv preprint arXiv:2201.09792, 2022

    2022 arXiv

  52. [60]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 13

    2017

  53. [61]

    Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference.arXiv preprint arXiv:2406.18139, 2024

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference.arXiv preprint arXiv:2406.18139, 2024

    2024 arXiv

  54. [62]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2025

    2025

  55. [63]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  56. [64]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  57. [65]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2024

  58. [66]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  59. [67]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  60. [68]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  61. [69]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  62. [70]

    Lmms-eval: Reality check on the evaluation of large multimodal models.arXiv preprint arXiv:2407.12772, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models.arXiv preprint arXiv:2407.12772, 2024

    2024 arXiv

  63. [71]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  64. [72]

    Wings: Learning multimodal llms without text-only forgetting.CoRR, abs/2406.03496, 2024

    Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal llms without text-only forgetting.CoRR, abs/2406.03496, 2024

    2024 arXiv

  65. [73]

    U n e x p e c t e d # visual tokens

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024. 14 A Implementation Details A.1 Trianing Details....

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.