Pith. sign in

REVIEW 4 major objections 5 minor 45 references

D-Attn: Decomposed Attention for Large Vision-and-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decomposing attention into separate text and image streams makes vision-language models up to five times faster.

desk verdict A mathematically clean attention decomposition whose debiased positional encoding delivers most of the gains, while the lossless-efficiency claim for V2V diagonalization remains under-supported. read the letter →

arxiv 2502.01906 v2 pith:IOBSLV6W submitted 2025-02-04 cs.CV

Chia-Wen Kuo , Sijie Zhu , Fan Chen , Xiaohui Shen , Longyin Wen This is my paper · ORCID
classification cs.CV
keywords decomposedattentionvision-languagemodelsvisual-to-visualpositionalencodingdebiasingdiagonalizationalpha-weightingefficientmultimodalLLMimagequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual practice of feeding visual and textual tokens through one shared causal self-attention is unnecessarily restrictive. It introduces Decomposed Attention (D-Attn), which splits that attention into visual-to-visual, textual-to-visual, and textual-to-textual parts, then recombines them with an analytically derived weighting. The split lets the authors remove a positional bias that made text attend unevenly across the image, and replace the quadratic visual-to-visual attention with a diagonal form in which each visual token attends only to itself. The stated result is better performance on a range of image question-answering benchmarks while training about five times faster. If correct, the claim matters because it suggests large language models can process images more efficiently without giving up their pre-trained language abilities.

What carries the argument

The central object is the attention decomposition identity: for a causal sequence with visual tokens $V$ followed by text $T$, $\mathrm{Attn}(t,[V,T]) = \alpha_V \mathrm{XA}(t,V) + \alpha_T \mathrm{SA}(t,T)$, where $\alpha_V = \mathrm{Sigmoid}(S_V - S_T)$ and $\alpha_T = 1 - \alpha_V$. This identity carries the argument because it rewrites the original self-attention algebraically rather than approximating it, so the text-to-text branch behaves as in the pre-trained LLM while the visual branch can be modified. On top of it sit two operations: debiased positional encoding, which assigns every visual token the same position in the T2V attention, and V2V diagonalization, which replaces the softmax attention matrix with the identity and reduces visual-to-visual self-attention to two fully connected layers.

What would settle it

Inspect the learned visual-to-visual attention matrices of a standard LVLM and compute the diagonal mass: if the average diagonal concentration is not close to dominant, or if replacing V2V attention with the identity plus debiased positions lowers accuracy on relation-heavy visual question benchmarks such as spatial relations, counting, and object localization, then the central efficiency claim fails.

Watch

Extended reading notes

Core claim

D-Attn decomposes a causal self-attention block, when visual tokens precede text, into three blocks: V2V self-attention, T2V cross-attention, and T2T self-attention. For a text token, the original output is exactly a weighted sum of T2V and T2T, with weights $\alpha_V = \mathrm{Sigmoid}(S_V - S_T)$ and $\alpha_T = 1 - \alpha_V$, where $S_V$ and $S_T$ are log-sum-exp scores of the text query against visual and textual keys. This exact equivalence, says the paper, is why the architecture keeps the pre-trained LLM's language capabilities intact. With this decomposition, the authors set all visual token positions to the same value in the T2V cross-attention, removing a bias that made text attend more to the lower part of an image, and force the V2V attention matrix to the identity, dropping complexity from $O(|V|^2)$ to $O(|V|)$. The paper reports consistent gains over the same models trained with standard self-attention on ten image benchmarks, along with up to 8x more visual tokens and 5x faster training.

Load-bearing premise

The load-bearing premise is that image tokens already contain their context, so letting each image token look only at itself loses nothing; the ablation data show that this swap alone hurts some scores and helps only after the positional-bias fix is also applied.

Editorial extensions

If this is right

  • Video and high-resolution image inputs can use up to 8x more visual tokens before memory limits, or train about 5x faster, at equal batch size.
  • Visual token operations can be redesigned independently of the language stream, without retraining or degrading the pre-trained LLM's text-to-text attention.
  • The positional bias that made text attend more to bottom-of-image tokens is removed, which should improve questions that require full-image reasoning.
  • The quadratic $O(|V|^2)$ cost of visual-to-visual attention disappears, leaving linear $O(|V|)$ cost in the number of visual tokens.
  • The merge step adds no learned parameters, unlike gated or cascaded alternatives, so the benefits come without a parameter overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore this, but the decomposition identity is token-type-agnostic, so the same alpha-weighted split could apply to audio, video, or structured tokens inserted before text.
  • A concrete testable extension is to use per-head, per-layer alpha values as a routing signal to skip visual computation where the text stream already relies on language context.
  • If diagonalization works because vision encoders pre-contextualize tokens, then stronger encoders should shrink any remaining gap between diagonal and full visual attention; that is a prediction a reader could test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes D-Attn, a decomposed attention architecture for large vision-and-language models (LVLMs). Standard causal self-attention is split into visual-to-visual (V2V), textual-to-visual (T2V), and textual-to-textual (T2T) attention, with the T2V and T2T outputs merged by an analytically derived alpha-weighting. This decomposition enables two modifications: debiased positional encodings in T2V attention and diagonalization of V2V attention, reducing complexity from O(|V|^2) to O(|V|). Experiments on ten image benchmarks with Mistral-7B and Gemma-9B backbones report consistent gains over the S-Attn baseline, along with up to a 5x training speedup.

Significance. If the central claims hold, this is a useful contribution: it shows how to obtain cross-attention-like flexibility without the typical performance penalty of adding separate cross-attention modules, and it does so with a parameter-free, analytically derived merging rule. The mathematical decomposition in Section 3.3 is correct, and the alpha weights are computed from the model's own attention log-sum-exp values rather than fitted to benchmarks, so the merging strategy is not circular. The experimental setup is fair in using the same training data and protocol for D-Attn and S-Attn, and the paper reports results on a wide range of public benchmarks with two LLM backbones. However, the claim that V2V diagonalization is lossless is not fully supported by the ablations as reported, because the control isolating diagonalization under the debiased positional encoding is missing.

major comments (4)
  1. [Sec. 4.3, Table 2] The claim that V2V diagonalization reduces complexity 'without compromising performance' (Abstract, Sec. 3.2) is not established by the reported ablations. Table 2 contains N/N, Y/N (diagonalization only, original positional encoding), and Y/Y (both modifications), but not N/Y (debiased positional encoding with full V2V attention). The Y/N versus N/N comparison shows that diagonalization alone degrades MME (1533.1 to 1507.6), MMB (70.3 to 68.8), and MMStar (36.8 to 32.6). Without the N/Y row, the net gains of Y/Y over N/N could be entirely attributable to the debiased positional encoding, with diagonalization still hurting under the final configuration. Please add the N/Y ablation or otherwise isolate the effect of diagonalization in the debiased setting.
  2. [Sec. 3.2, Eq. (1)] The premise for the O(|V|) efficiency gain is that V2V attention weights are 'concentrated at the diagonal,' so each visual token needs to attend only to itself. This is asserted without quantitative evidence; no attention-matrix statistics (e.g., diagonal mass fraction) are reported for any layer, head, or resolution. Given that Table 2 shows diagonalization alone degrades several benchmarks, the claim that the identity approximation is lossless needs direct empirical support. Please report diagonal-concentration measurements, or provide another quantitative justification for the approximation.
  3. [Sec. 3.3] The paper overstates that D-Attn 'does not introduce architectural and operational changes to the causal self-attention mechanism' (Sec. 3.3 and Abstract). D-Attn explicitly changes T2V attention by assigning all visual tokens the same position (Sec. 3.1) and replaces the softmax attention matrix in V2V with an identity (Sec. 3.2). The intended meaning appears to be that the T2T attention is unchanged; as written, the claim is incorrect and should be qualified to refer only to the textual-to-textual component.
  4. [Tables 1 and 2] No error bars or repeated-run statistics are reported for any benchmark. Several differences, e.g., MMB 70.3 to 71.3 on Mistral and GQA 61.8 to 64.4 in the ablation, could fall within benchmark noise. Please report results across at least three seeds, or provide established variance estimates for the evaluation protocol, so that the significance of the reported gains can be assessed.
minor comments (5)
  1. [Sec. 1, Contributions] There are typos: 'diagonized' should be 'diagonalized' and 'siginicant' should be 'significant'.
  2. [Sec. 3.1] 'postional' should be 'positional' in the opening sentence.
  3. [Table 3] The Cascade merging strategy adds about 1.4B parameters (9.0B vs 7.6B), so the comparison against the parameter-free alpha-weighting is not fully matched. Please clarify whether the parameter increase is essential to the cascade design or provide a parameter-matched variant.
  4. [Related Work] The naming 'LLaV A' and 'LLaV A-1.5' has inconsistent spacing; use a single consistent convention throughout.
  5. [Sec. 4.3] The statement that 'V2V Diagonal-Attn performs comparably to conventional LVLMs' is supported only by a single Mistral configuration in Table 2; please specify the backbone and note the scope of the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: alpha-weighting and V2V diagonalization are derived from attention algebra and validated on external benchmarks.

full rationale

The central derivation chain is self-contained. Section 3.3 derives the decomposition of causal self-attention into V2V, T2V, and T2T components as an exact algebraic identity: Equation (8) expresses the textual-token attention output as alpha_V * XA(t,V) + alpha_T * SA(t,T), where alpha_V and alpha_T are computed from log-sum-exp values of the visual and textual attention logits (Equations 6 and 7). This is a mathematical consequence of the softmax definition, not a fitted parameter or a benchmark-driven prediction. The two proposed improvements are applied on top of this decomposition: debiased positional encodings modify the T2V inputs, and V2V diagonalization replaces the V2V softmax attention matrix with the identity in Equation 1, reducing complexity from O(|V|^2) to O(|V|). Neither modification defines its own success criterion in terms of the model's outputs; effectiveness is measured against ten external benchmarks (Table 1) and ablations (Tables 2 and 3) using public datasets and standard training procedures. The only self-citation, CuMo [19], appears in related work and is not load-bearing to any claim. The V2V diagonalization motivation is an empirical assertion (Section 3.2: 'attention weights matrix of V2V SA are concentrated at the diagonal') that is not supported by reported attention statistics, and Table 2 lacks an N/Y ablation (debiased PE without diagonalization) that would cleanly isolate the diagonalization's cost on MME, MMB, and MMStar; these are evidence gaps, not circular reasoning. No parameters are fitted to the evaluation benchmarks, no same-author uniqueness theorem is invoked, and no ansatz is smuggled in via citation. Thus the derivation chain does not reduce to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The only hand-chosen design element is the uniform visual position assignment. The alpha-weighting itself is parameter-free. The main risk is the domain assumption that V2V attention is redundant, which is load-bearing for the claimed speedup.

free parameters (1)
  • Visual token position assignment = 0 for all visual tokens (uniform)
    Chosen by hand to remove positional bias in T2V attention in Section 3.1; no search over alternative position assignments is reported.
assumptions (4)
  • standard math Softmax attention over a concatenated key/value sequence can be split into two attention distributions weighted by the total softmax masses of each partition (Eq. 4-8).
    The derivation in Section 3.3 is exact for the unmasked softmax and requires no approximation.
  • domain assumption Visual tokens precede textual tokens and causal masking is applied, so the decomposition into V2V, T2V, and T2T blocks is valid.
    The entire method assumes this token ordering, as stated at the start of Section 3.
  • domain assumption RoPE/relative positional encodings create a distance-based attention bias between text and image tokens that is undesirable; setting all visual positions to P0 removes it beneficially.
    This is the motivation for debiased positional encodings in Section 3.1; it is a modeling assumption about what the LLM should attend to.
  • domain assumption Visual tokens from a bidirectional vision transformer already encode contextual information, so V2V self-attention within the LLM is redundant and can be replaced by identity.
    Stated in Sections 3.2 and 4.3; the only support is a qualitative observation of diagonal concentration, and the ablation shows it is not uniformly true by itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of D-Attn: Decomposed Attention for Large Vision-and-Language Models." pith.science (2026). https://pith.science/paper/IOBSLV6W

@misc{pith2026250201906,
  author       = {Pith},
  title        = {Pith review of: D-Attn: Decomposed Attention for Large Vision-and-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOBSLV6W}},
  note         = {Machine review of arXiv:2502.01906}
}
abstract

Large vision-and-language models (LVLMs) have traditionally integrated visual and textual tokens by concatenating them into a single homogeneous input for large language models (LLMs), thereby maximally preserving the pre-trained language capabilities. However, this constrained architecture for visual and textual tokens restricts the design space for processing visual tokens, potentially leading to suboptimal performance and efficiency. In this paper, we propose Decomposed Attention (D-Attn), a more flexible attention architecture for LVLMs, which enables modification of visual token operations without affecting textual-to-textual attention. D-Attn decomposes the 1-D causal self-attention of LVLMs into visual-to-visual, textual-to-visual, and textual-to-textual attentions, and the visual and textual output tokens from the decomposed attentions are merged with a carefully derived weighting strategy, namely $\alpha$-weighting. Taking advantage of the flexibility, we are able to introduce two critical improvements in visual token processing while maintaining the capacity of pre-trained LLMs: 1) We rectify the biased positional encoding in textual-to-visual attention to boost visual understanding performance. 2) We diagonalize visual-to-visual attention to reduce computation complexity from $O(|V|^2)$ to $O(|V|)$ for $|V|$ visual tokens without compromising performance. Extensive experiments and analysis validate the effectiveness of D-Attn, demonstrating significant improvements on multiple image benchmarks while significantly reducing computational costs (\eg, $5\times$ faster). Code will be available at https://github.com/bytedance/DecomposedAttention.

Figures

Figures reproduced from arXiv: 2502.01906 by the authors.

Figure 1
Figure 1. Performance comparison between proposed D-Attn models and their self-attention (S-Attn) counterparts on popular image benchmarks. Detailed results are available in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Decomposition of causal self-attention within an LVLM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An illustration of positional encoding. In T2V XA, textual tokens interact with visual tokens to incorporate visual information, where textual tokens (queries) attend to visual tokens (keys and values) as shown in Figure 3a. In the figure, we can see that the rotary/relative positional encodings skew attention weights based on the positional distance between visual and textual tokens. For example, distant pairs such… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Module architecture and operations of (a) conventional attention in LVLM with visual and texual tokens concatenated as a [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons between D-Attn and its Self-Attn (S-Attn) counterpart. Erroneous outputs from the S-Attn model are highlighted in red, while the accurate and preferred responses from D-Attn are highlighted in blue. Model Existence Count Position Color Posters C…
Figure 6
Figure 6. Figure 6: Visualization of αV value on SQA-I, GQA, VQA-T benchmarks. Model Scene Understanding Instance Identity Instance Location Instance Attributes Instances Counting Spatial Relation Instance Interaction Visual Reasoning Text Understanding S-Attn 76.9 74.5 74.7 67.3 64.2 57.…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 16 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 2

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint arXiv:2308.01390 , 2023. 2

    arXiv 2023

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. 5

    arXiv 2022

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    arXiv 2023

  6. [6]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330,

    arXiv

  7. [7]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems, 2023. 2, 5

    work page 2023

  8. [8]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. 6

    2022

Show all 45 references
  1. [9]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 2

    2024 arXiv

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  3. [11]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models, 2024. 5, 7, 8

    2024

  4. [12]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  5. [13]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5

    2019

  6. [14]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 1, 2, 5

    arXiv

  7. [15]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Informa...

    2024

  8. [16]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 2

    2024 arXiv

  9. [17]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 5, 8

    2023 arXiv

  10. [18]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  11. [19]

    Cumo: Scaling multimodal llm with co-upcycled mixture- of-experts

    Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal llm with co-upcycled mixture- of-experts. arXiv preprint arXiv:2405.05949, 2024. 2, 5

    2024 arXiv

  12. [20]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  13. [21]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 2, 5

    2024

  14. [22]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 5

    2024

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 5

    2024

  16. [24]

    Mmbench: Is your multi-modal model an 9 all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an 9 all-around player? arXiv preprint arXiv:2307.06281, 2023. 5, 8

    2023 arXiv

  17. [25]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  18. [26]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 5

    2016

  19. [27]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011. 5

    2011

  20. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  21. [29]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 5

    2024

  22. [30]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parame- ters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parame- ters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pag...

    2020

  23. [31]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  24. [32]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of ACL, 2018. 5

    2018

  25. [33]

    Megatron- lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. 6

    1909 arXiv

  26. [34]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2

    2024 arXiv

  27. [35]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,

    arXiv

  28. [36]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2, 5

    2024 arXiv

  29. [37]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 5

    2024

  30. [38]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  31. [39]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2

    2017

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 5

    2024 arXiv

  33. [41]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 2, 5

    2024

  34. [42]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 5

    2023

  35. [43]

    Root mean square layer nor- malization

    Biao Zhang and Rico Sennrich. Root mean square layer nor- malization. Advances in Neural Information Processing Sys- tems, 32, 2019. 5

    2019

  36. [44]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 2, 5

    2024 arXiv

  37. [45]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 1 10

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.