Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In multimodal LLMs, fewer than 5% of attention heads carry visual understanding, and SparseMM skews KV-cache budgets toward those heads to accelerate generation without losing accuracy.

desk verdict A useful, reproducible KV-cache allocation method behind an under-validated 'visual head' story; the <5% claim lacks a threshold and FastV/ElasticCache are missing. read the letter →

arxiv 2506.05344 v2 pith:U2PFYIWF submitted 2025-06-05 cs.CV

Jiahui Wang , Zuyan Liu , Yongming Rao , Jiwen Lu This is my paper
classification cs.CV
keywords visualheadsheadsparsityKVcachecompressionmultimodallargelanguagemodelsattentionanalysisOCRgroundingasymmetricallocationinferenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multimodal large language models inherit a sparse visual faculty: fewer than 5% of attention heads carry essentially all of the model's visual understanding, across both multi-head and grouped-query attention architectures. The authors propose a training-free OCR-anchored procedure that scores each head by how often its strongest attention lands on the image patch corresponding to the text token being generated. They then use those scores to allocate asymmetric KV-cache budgets, keeping more cache for visual heads and compressing the rest, and report that this preserves benchmark accuracy while accelerating decoding and cutting memory. The significance is that visual semantics in MLLMs can be identified cheaply before inference and exploited for efficiency without retraining.

What carries the argument

The machinery is the visual-score matrix S, computed without training by matching each generated OCR token to its ground-truth bounding box, mapping that box to image tokens, and incrementing a head's score when its argmax attention lands inside those tokens. The second piece is the SparseMM cache allocator, which splits each head's budget into a local window of the newest tokens, a small uniform baseline share, and a score-proportional share of the remaining cache; the surviving keys in each head are then chosen by top-k attention within a 32-token observation window. Together these convert a static head-level sparsity measurement into a per-head, per-layer cache budget.

What would settle it

Compute the same visual scores on a text-only control: run the OCR task with the image replaced by a blank or scrambled patch grid while keeping the same output tokens, and check whether the top-scoring visual heads still receive frequent argmax hits on image-token positions; if they do, the score does not isolate visual function. Alternatively, shuffle the mapping between bounding boxes and image patches and show that the head ranking barely changes, which would indicate the score tracks attention concentration rather than visual content.

Watch

Extended reading notes

Core claim

The central claim is that visual understanding in MLLMs is concentrated in a sparse set of attention heads, termed visual heads, which emerge consistently across model families and attention designs. The authors demonstrate this by defining a visual score per head: over OCR images, for each generated token they locate the aligned image patch, and a head scores a hit if its argmax attention falls on one of those image tokens; scores are normalized and aggregated over 1,000 images. The claim is that heads in the top few percent of this score matrix are genuinely visual: masking them hurts OCR and VQA performance far more than masking the same number of random heads, and they generalize to non-OCR tasks. On this basis SparseMM gives visual heads a larger share of the KV-cache budget and reports parity with full-cache performance at a small fraction of the cache, with concrete efficiency gains.

Load-bearing premise

The argument assumes that repeatedly observing a head's argmax attention land on the OCR-aligned image patch measures that head's causal contribution to visual understanding; if argmax hits instead reflect general attention concentration, such as attention sinks or text priors, the sparsity phenomenon and the cache advantage would not be specifically visual.

Editorial extensions

If this is right

  • If the sparsity claim is right, MLLM inference can be accelerated without retraining by concentrating KV-cache budget on the small visual-head subset.
  • On DocVQA, LLaVA-NeXT-Vicuna-7B matches full-cache accuracy at 20% of the cache and Qwen2-VL-7B-Instruct at 5.3% of the cache.
  • With 32K input tokens, LLaVA-NeXT-Vicuna-7B keeps decoding latency nearly constant, achieving roughly a 1.87x speedup and cutting peak memory from 32.87 GB to 17.38 GB; overall the paper reports 1.38x real-time acceleration and a 52% memory reduction.
  • Visual heads identified on OCR transfer to object recognition and scene understanding, so the budget skew applies beyond document tasks.
  • Masking the top 5% of visual heads causes a larger performance drop than masking an additional 5%, consistent with a sparse but indispensable distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-argmax criterion could conflate visual heads with generic attention concentration, such as attention sinks or language priors; a natural control is to run the same scoring on text-only inputs and check whether the same heads still receive frequent argmax hits on image-token positions.
  • The paper's own ablation shows that removing the uniform baseline (rho = 0) collapses performance for the Mistral model, so the reported gains depend on the hybrid allocation rather than on visual-head scores alone.
  • The identification stage costs 1,000 OCR forward passes; whether SparseMM is an end-to-end win depends on amortizing that one-time profiling cost over many downstream inferences for the same model.
  • A testable extension is to replace OCR supervision with caption-based alignment signals to see whether the same sparse head set emerges, which would show whether the phenomenon is intrinsic to the base LLM or shaped by the instruction-tuning data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates attention-head specialization in multimodal large language models (MLLMs) and claims that fewer than 5% of decoder attention heads, termed visual heads, drive visual understanding. The authors introduce a training-free identification procedure based on OCR-aligned argmax attention hits (Eq. 3-4), report that these heads are sparse across Vicuna- and Qwen2-based backbones, and use the resulting head scores to allocate asymmetric KV-cache budgets in a method called SparseMM. Experiments across DocVQA, OCRBench, TextVQA, ChartQA, TextCaps, MMBench, GQA, and VQAv2 show that SparseMM generally outperforms SnapKV, PyramidKV, AdaKV, and a random-head baseline at low cache budgets, with reported latency and memory gains.

Significance. If the sparsity claim is correct, the paper offers a simple, training-free route to identify modality-specialized heads, which is of scientific interest for interpretability of MLLMs and of practical value for inference acceleration. Strengths include: the identification method is training-free and model-agnostic; the paper includes a random-head control that is consistently worse; the main results span three backbones and many benchmarks; code is open-sourced; and the appendix provides numerical tables. However, the central claim of '<5%' is not quantitatively defined, and the hit-based score may confound generic attention concentration with visual specialization; these issues are load-bearing for the paper's framing and need to be resolved.

major comments (3)
  1. [Sec. 3.2, Algorithm 1, Abstract] The claim that 'approximately less than 5%' of heads are visual is never tied to a threshold on the visual score. Eq. (3) produces a continuous score, but the paper does not state what score cutoff, percentile, or other rule separates visual from non-visual heads; Fig. 2 is a qualitative heatmap with no scale or cutoff marked. Without a stated decision rule, the sparsity claim is not falsifiable, and the connection to the budget allocation in Eq. (8) is loose. Please define the criterion explicitly (e.g., a score threshold calibrated against a null distribution, or a top-k selection) and report the resulting number of heads per layer/model.
  2. [Sec. 3.2, Eq. (4)] The hit criterion in Eq. (4) uses only the argmax position and ignores how much attention mass is placed on the aligned image token. A head that puts 30% of its mass on the target is counted identically to one that puts 100%. Moreover, because OCR images have text regions covering a large fraction of the image, a head with generic concentration on salient patches or attention sinks may accumulate hits by chance. To support the interpretation that these heads are specifically responsible for visual understanding, please add a control, for example a permutation test that randomly reassigns token-to-region correspondences, or a baseline of attention to random image patches, and show that observed scores exceed chance.
  3. [Sec. 4.2, Tab. 3, Fig. 4] The experimental comparisons are reported as point estimates without error bars, multiple seeds, or significance tests, and several differences at moderate budgets are small (e.g., Table 6 at 256 tokens: SparseMM 64.52 vs AdaKV 64.65 on GQA; at 512 tokens both achieve 81.52 on MMBench, with PyramidKV at 81.53). In addition, the hyperparameter rho is tuned on OCRBench (Table 3) and the benchmark suite is dominated by OCR-style tasks, raising a risk of selection bias. Please report variance across at least three seeds and either fix rho on a separate validation set or show sensitivity across benchmarks for all evaluated values.
minor comments (4)
  1. [Sec. 3.2] In Eq. (3) and the surrounding text, the normalization of the visual score matrix is not specified; 'These scores are then normalized' (end of Sec. 3.2) is vague. Please state the normalization formula and whether it is applied per layer, per head, or globally.
  2. [Algorithm 1] Algorithm 1 does not specify how output tokens that do not match any OCR text/bbox pair (e.g., function words, punctuation) are handled; this could bias the hit counts. Clarify the matching procedure and the treatment of non-matching tokens.
  3. [Abstract, Sec. 4.3, Tab. 2] The efficiency numbers in the abstract (1.38x acceleration, 52% memory reduction) are not directly traceable to a table: Table 2 reports 52.9/37.1 = 1.43x latency reduction, and the Sec. 4.3 memory example gives 32.87 to 17.38 GB (47%). Please state the exact configuration for the abstract numbers.
  4. [Throughout] Typos: 'sparity' in the abstract should be 'sparsity'; 'Mm asking' in the Fig. 7 caption should be 'Masking'. Additionally, the statement in Sec. 4.2 that the random head method is comparable to SnapKV for the MHA model is not fully supported by Table 5 at low budgets on DocVQA (e.g., 0.4977 vs 0.4868 at 64 tokens), and should be qualified.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: 'visual heads' are defined as high scorers on the same OCR score matrix used to assert the <5% sparsity, and SparseMM's headline gains are demonstrated on OCR-family benchmarks consistent with that score; non-OCR tests provide only partial independent support.

  1. self definitional [Sec. 3.2, Eq. (3)-(4), and Fig. 2 caption; Abstract]
    "Visual Score for Head h = 1/N Σ_i I_hit(y_i,A_h)/#image_tokens ... hit(y_i, A_h) = 1 if argmax(A_h) ∈ I_{y_i}. ... We use OCR tasks to obtain visual scores for all heads. Upon visualizing these scores, we discovered that high-scoring heads, which we refer to as visual heads, are quite sparse within the MLLM, comprising only about 5%."

    The class 'visual heads' is defined as the high-scoring heads in the score matrix S produced by Eq. (3)-(4). The central sparsity claim ('approximately less than 5%') is then read off that same matrix, with no independent threshold or falsification criterion stated; the Sec. 4.4 ablation operationally takes 'visual heads' to be the top 5% of scorers. Thus the statement that visual heads are <5% is an artifact of the chosen cutoff and score definition, not a property of the model measured independently of the scoring procedure. The score also measures only whether the argmax position lies in an OCR-aligned token set, ignoring attention mass and causal contribution, so 'visual relevance' is definitionally tied to the OCR hit test.

  2. fitted input called prediction [Sec. 3.2 (Synthdog score computation) and Sec. 4.2 (DocVQA, OCRBench, TextVQA, ChartQA, TextCaps)]
    "Finally, we aggregate the scores from all heads across 1,000 OCR images from the Synthdog dataset [18]. ... Specifically, we utilize DocVQA [35], OCRBench [29], TextVQA [44], ChartQA [34], and TextCaps [43]."

    SparseMM's budget allocation (Eq. 8) is proportional to the visual scores, and those scores are computed by measuring argmax hits onto image tokens corresponding to OCR text (Eq. 3-4). The principal benchmarks used to demonstrate accuracy parity and gains are OCR-family tasks (DocVQA, OCRBench, TextVQA). On those tasks, preserving heads that by construction attend to OCR-aligned patches is expected to preserve exactly the behavior used to define the scores, so the headline 'prediction' of maintained performance is a consistency check on the score definition rather than an out-of-sample test. The MMBench, GQA, and VQAv2 results provide partial independent grounding, which keeps the circularity partial rather than total.

full rationale

The paper's central discovery and its application are built on one operationalization: a head is 'visual' if its argmax attention position falls on an image token aligned with OCR output text (Eq. 3-4). From this single score matrix, the paper reads off both the identity of 'visual heads' and the 'less than 5%' sparsity figure, without stating the threshold that defines a high score; the Sec. 4.4 masking experiment then treats the top 5% as visual heads, making the sparsity statement partly true by construction of the cutoff. The SparseMM acceleration results are then demonstrated mainly on OCR-family benchmarks, where the allocation prior derived from OCR-aligned attention is expected to line up with the task; this is a fitted-input-called-prediction pattern in weakened form, since no labels are fitted, only the head-scoring prior is derived from the same data family. Importantly, the paper does include non-OCR evaluations (MMBench, GQA, VQAv2) and a random-head baseline, which give some independent evidence that the head scores carry useful signal beyond the OCR definition. There is no load-bearing self-citation: the cited ElasticCache work by the same authors appears only in related work and does not justify the method. The main correctness risk is the unvalidated claim that argmax OCR hits measure causal visual understanding rather than generic attention concentration, but that concern is about construct validity and threshold reproducibility, not about a derivation that reduces entirely to its inputs. Overall, the circularity is partial and localized to the definition-to-conclusion and OCR-to-OCR evaluation loop, hence a score of 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central method depends on several hand-set or fitted parameters (rho, window size, top-5% threshold) and on the assumption that OCR-derived argmax hits reveal a stable visual specialization. The causal masking experiments partially support the visual-head construct but do not validate the specific 5% cutoff or its use as a universal allocation signal.

free parameters (4)
  • visual_head_topk_threshold = approximately 5% (no explicit formula)
    The paper calls the top-scoring heads "visual heads" (Sec. 3.2, Fig. 2) and later masks the top 5% in Fig. 7, but no criterion or statistical test fixes this percentage.
  • uniform_ratio_rho = 0.1
    Chosen from the OCRBench ablation in Table 3; the best value is selected using an evaluation benchmark, so it is a fitted hyperparameter.
  • local_window_size_w = 32 (default)
    Fixed as a default (Sec. 3.3) and not ablated in the paper.
  • observation_window_size = 32
    Adopted from SnapKV for the local attention computation in Eq. 10-13 without re-validation on MLLMs.
assumptions (4)
  • domain assumption Argmax attention score is a valid proxy for a head's functional role
    Eq. 4 defines visual score as hits of the argmax attention on image tokens; no causal validation is provided at that stage.
  • domain assumption OCR gives exact token-to-image-patch correspondence
    Algorithm 1 matches generated tokens to bounding-box pairs from OCR; this is exact only for synthetic OCR ground truth and may be noisy for free-form outputs.
  • domain assumption Visual heads identified on OCR generalize to other visual tasks
    Figs. 8 and 9 test this and show partial transfer, but the method assumes this generality when applying OCR-derived scores to all benchmarks.
  • domain assumption A 32-token observation window is sufficient to estimate attention for cache selection
    Eq. 10-13 inherit this from SnapKV without a dedicated validation on multimodal inputs.
invented entities (1)
  • visual head
    purpose: A functional category of attention heads claimed to be the only ones contributing to visual understanding; used to set KV-cache budgets.
    The category is defined by the paper's own OCR score and threshold; no independent evaluation protocol outside the paper defines visual heads, so it remains an internal construct, though the masking experiments in Fig. 7 provide in-paper causal support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs." pith.science (2026). https://pith.science/paper/U2PFYIWF

@misc{pith2026250605344,
  author       = {Pith},
  title        = {Pith review of: SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2PFYIWF}},
  note         = {Machine review of arXiv:2506.05344}
}
read the original abstract

Multimodal Large Language Models (MLLMs) are commonly derived by extending pre-trained Large Language Models (LLMs) with visual capabilities. In this work, we investigate how MLLMs process visual inputs by analyzing their attention mechanisms. We reveal a surprising sparsity phenomenon: only a small subset (approximately less than 5%) of attention heads in LLMs actively contribute to visual understanding, termed visual heads. To identify these heads efficiently, we design a training-free framework that quantifies head-level visual relevance through targeted response analysis. Building on this discovery, we introduce SparseMM, a KV-Cache optimization strategy that allocates asymmetric computation budgets to heads in LLMs based on their visual scores, leveraging the sparity of visual heads for accelerating the inference of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the particularity of visual, SparseMM prioritizes stress and retaining visual semantics during decoding. Extensive evaluations across mainstream multimodal benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52% memory reduction during generation while maintaining performance parity on efficiency test. Our project is open sourced at https://github.com/CR400AF-A/SparseMM.

Figures

Figures reproduced from arXiv: 2506.05344 by the authors.

Figure 1
Figure 1. Head Sparsity Emerges from Visual Concept Re￾sponses. We observe the visual-relevant heads are sparse in various MLLMs. Based on this observation, we devise a KV-Cache opti￾mization strategy that allocates asymmetric budgets to LLM heads based on their importance for visual tokens, achieving better trade￾off under limited computational resources. lightweight adapters to project visual features into the lan￾guage mod… view at source ↗
Figure 2
Figure 2. Visual Heads are Sparse in MLLMs. We use OCR tasks to obtain visual scores for all heads. Upon visualizing these scores, we discovered that high-scoring heads, which we refer to as visual heads, are quite sparse within the MLLM, comprising only about 5%. The majority of heads have very low scores, indicating that most heads in LLMs do not focus on visual information. of the KV-Cache. For instance, StreamingLLM [50] … view at source ↗
Figure 3
Figure 3. SparseMM for MLLM Acceleration. The KV Cache budget for each head is composed of three parts: Local Window Cache, Uniform-Based Cache, and Score-Preferred Cache. The top-K KV caches are selected based on attention scores. detailed in Sec. 3.2. In an ideal setting, the cache allocation would be determined exclusively by the values in ScoreL×H. However, inspired by AdaKV [13], and to account for local￾ity and to ensur… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Main Results on Multi-Modal Benchmarks. We evaluate SparseMM and other baselines on several multimodal benchmarks, and conduct experiments on a series of backbones. Our SparseMM consistently outperforms the other baselines [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results on Multiple-choice Benchmarks. We evaluate SparseMM and other baselines on multiple-choice visual bench￾marks with Qwen2-VL-7B-Instruct as the backbone model. Our SparseMM consistently outperforms the other baselines. TextVQA [44] task using LLaVA-NeXT-Vicuna-7…
Figure 6
Figure 6. Figure 6: Efficiency Evaluation for SparseMM. Benefiting from the reduction in KV cache, SparseMM can maintain nearly constant decoding latency, achieving up to a 50% acceleration. Additionally, it effectively reduces peak memory usage. 4.4. Analysis Performance Influence of Vis…
Figure 7
Figure 7. Figure 7: Comparisons of Mmasking Visual Head and Random Head. The left figure is the result on OCRBench, and the right figure is the result on TextVQA [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 10
Figure 10. Figure 10: Visualizations of Visual Heads. We visualized the attention distribution of several heads. The visual heads are able to accurately capture text or objects within the images, whereas the non-visual heads provide random results. a budget of 256 KV Cache. With the suppor…
Figure 9
Figure 9. Figure 9: Results with Different Visual Head Identification Approaches and Datasets. We conduct an evaluation on visual heads identified on different datasets. The results on OCR datasets are similar and better than those on the detection dataset. datasets. In addition, we consi…
Figure 11
Figure 11. Figure 11: More Visualization Results. Visual heads are able to attend to the correct objects, whereas non-visual heads cannot. CIDEr assesses the quality of generated captions by comput￾ing a weighted n-gram similarity between the candidate and reference captions. B. More Visua…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

Reference graph

Works this paper leans on

56 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Bap- tiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073,

    arXiv

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebr ´on, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,

    arXiv

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35: 23716–23736, 2022. 1

    work page 2022

  4. [4]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024. 3, 5

    arXiv 2024

  5. [5]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024. 3

    work page 2024

  6. [6]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. 1

    arXiv 2024

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 1

    work page 2024

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

Show all 56 references
  1. [9]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wen- wei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. Advances in Neural Informa...

    2024

  2. [10]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Win- ston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 2024. 2

    2024 arXiv

  3. [11]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [12]

    Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language un- derstanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023, 2024. 2

    2024 arXiv

  5. [13]

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550, 2024. 3, 4, 5, 6

    2024 arXiv

  6. [14]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    GeminiTeam. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1

    2024 arXiv

  7. [15]

    Making the v in vqa matter: Elevating 10 Table 5

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating 10 Table 5. Numerical results of Fig. 4. Benchmark Method LLaV A-NeXT-Vicuna-7B LLaV A-NeXT-Mistral-7B Qwen2-VL-7B-Instruct 2048 1024 512 256 128 64 2048 1024 51...

    2017

  8. [16]

    3d-llm: Injecting the 3d world into large language models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. NeurIPS, 36:20482– 20494, 2023. 2

    2023

  9. [17]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 5

    2024 arXiv

  10. [18]

    Donut: Document understanding transformer without ocr

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7 (15):2, 2021. 4

    2021 arXiv

  11. [19]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  12. [20]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 2

    2024 arXiv

  13. [21]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 1

    2023

  14. [22]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Process- ing Systems, 37:22947–22970, 2024. 3, 5, 7

    2024

  15. [23]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 2

    2023 arXiv

  16. [24]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023. 2

    2023

  17. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence 11 Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceed...

    2014

  18. [26]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 1

    2024

  19. [27]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 2, 5

    2024

  20. [28]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 2, 5, 9

    2023 arXiv

  21. [29]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 2, 5, 6, 7, 9

    2023 arXiv

  22. [30]

    Oryx mllm: On-demand spatial- temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial- temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 2

    2024 arXiv

  23. [31]

    Effi- cient inference of vision instruction-following models with elastic cache

    Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Effi- cient inference of vision instruction-following models with elastic cache. In European Conference on Computer Vision, pages 54–69. Springer, 2024. 3

    2024

  24. [32]

    Ola: Pushing the frontiers of omni-modal language model

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Win- ston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328, 2025. 2

    2025 arXiv

  25. [33]

    Ovis: Structural embed- ding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 2

    2024 arXiv

  26. [34]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 5, 9

    2022 arXiv

  27. [35]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 2, 5, 6, 9

    2021

  28. [36]

    Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019

    Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowd- hury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al. Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In 201...

    2019

  29. [37]

    Openai gpt-3.5 api

    OpenAI. Openai gpt-3.5 api. OpenAI API, 2023. 1

    2023

  30. [38]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. OpenAI Blog, 2023. 2

    2023

  31. [39]

    Hello gpt-4o — openai

    OpenAI. Hello gpt-4o — openai. OpenAI Blog, 2024. 1

    2024

  32. [40]

    Qwen2 technical report

    QwenTeam. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 2

    2024 arXiv

  33. [41]

    Qwen2-vl: To see the world more clearly

    QwenTeam. Qwen2-vl: To see the world more clearly. Wwen Blog, 2024. 2, 5

    2024

  34. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 1, 2, 3

    2021

  35. [43]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Aman- preet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer,...

    2020

  36. [44]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xin- lei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 8317–8326, 2019. 2, 5, 6, 7, 9

    2019

  37. [45]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 1

    2024

  38. [46]

    Qwen2.5-vl, 2025

    Qwen Team. Qwen2.5-vl, 2025. 1

    2025

  39. [47]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  41. [49]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2, 5

    2017

  42. [50]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 3

    2023 arXiv

  43. [51]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2

    2024 arXiv

  44. [52]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 2

    2024 arXiv

  45. [53]

    mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, ...

    2024

  46. [54]

    A large chinese text dataset in the wild

    Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology, 34(3):509–521,

  47. [55]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 12 In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1, 2, 3

    2023

  48. [56]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R ´e, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Sys...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.