Pith. sign in

REVIEW 3 major objections 5 minor 71 references

LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding just six spatial tokens to a multimodal LLM's visual input beats LLaVA-1.5 on 10 of 11 benchmarks at the same speed.

desk verdict Solid modular projector paper whose headline gains likely don't exceed run-to-run noise, but the architecture and ablation honesty are real enough to referee. read the letter →

arxiv 2507.00505 v3 pith:ZK2BFSLO submitted 2025-07-01 cs.CV

Haoran Lou , Chunxiao Fan , Ziyan Liu , Yuexin Wu , Xinliang Wang This is my paper
classification cs.CV
keywords multimodallargelanguagemodelsvisualspatialtokensfeatureextractordetailintegratorgroundinghallucinationreductionLoRAfine-tuningCLIP-ViTlimitations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLaVA-SP claims that the main visual weakness of multimodal LLMs is recoverable at almost no cost. CLIP-ViT flattens an image into a 1D row of patches, losing the spatial relationship between neighboring patches; the paper's fix is to reshape those same patch features back into a 2D grid and extract just six extra visual spatial tokens that encode how the image is organized from the center outward or from abstract to specific. A cross-attention step then injects fine detail into these six tokens without adding tokens. Fine-tuned with LoRA, both variants (Cropping and Pooling) beat the fully trained LLaVA-1.5 on 10 of 11 multimodal benchmarks, with the largest gains on fine-grained tasks like visual grounding and hallucination, while adding six tokens to a 576-token sequence and keeping inference speed essentially unchanged. If the claim holds, it shows that fine-grained visual understanding can be bought with a better projector rather than with higher resolution, bigger encoders, or more tokens.

What carries the argument

The load-bearing object is the visual spatial token: each of the six tokens is produced by collapsing one multi-scale 2D feature map — sizes $4\times4$, $8\times8$, $12\times12$, $16\times16$, $20\times20$, $24\times24$ — with a convolutional kernel, so each token encodes a spatial scale and a position in a chosen ordering, central to global for cropping and abstract to specific for pooling, rather than a single patch. The Detail Feature Integrator is the second mechanism: a cross-attention layer that takes the six tokens as queries and a fine-grained $5\times5$ feature map (from a $16\times16$ kernel) as keys and values, letting detailed local features be folded into the six tokens without lengthening the sequence. The ablations attribute the gain to these choices: convolutional kernels beat transformer blocks for extraction, six tokens beat one or twelve, and the integrated spatial ordering beats a sliding-window token design, with the token order itself acting as an explicit spatial reading order for the LLM.

What would settle it

Re-run the LoRA fine-tuning for both LLaVA-SP variants and the LLaVA-1.5 baseline across three to five random seeds on the same eleven benchmarks. The reported margins are mostly below 1.5 points on the normalized average, so if the mean differences fall within the run-to-run spread, the claim that six spatial tokens materially improve visual representation would not be supported. A second check: evaluate the architecture chosen on the ablation suite (six tokens, kernels 4 to 24, $5\times5$ DFI map) on held-out fine-grained benchmarks that were not part of the eleven, to test whether the configuration transfers or is tuned to that suite.

Watch

Extended reading notes

Core claim

The paper's central claim is that the 2D spatial structure that CLIP-ViT destroys when it flattens patches is itself a recoverable source of visual signal, and that six tokens are enough to recover it. The Spatial Feature Extractor reshapes the $24\times24$ patch feature grid and produces six multi-scale feature maps — either by cropping inward from $24\times24$ down to $4\times4$, arranged from central region to global, or by adaptive pooling into the same sizes, arranged from abstract to specific — and collapses each map to one token with a convolutional kernel (sizes $k = 4, 8, 12, 16, 20, 24$). The Detail Feature Integrator then uses those six tokens as queries and a larger $5\times5$ feature map as keys and values in a cross-attention layer, so fine-grained detail flows into the six tokens without increasing their count. With the six spatial tokens concatenated ahead of the 576 patch tokens, LLaVA-SP-Cropping and LLaVA-SP-Pooling, fine-tuned with LoRA on the LLaVA-1.5 data, outperform the fully trained LLaVA-1.5 on 10 of 11 benchmarks, with the clearest margins on fine-grained tasks: VizWiz +5.9, TextVQA +2.5, SQA-IMG +2.1, RefCOCO val +5.6 points, and MMVP +6.6 points, at nearly identical latency (about 20.3 to 20.5 versus 20.8 tokens per second).

Load-bearing premise

The headline claim rests on single runs over a fixed set of eleven benchmarks, with the architecture's details (six tokens, kernel sizes, $5\times5$ feature map) chosen by ablating on that same set; if run-to-run noise or tuning to that suite explains the roughly one-point average gains, the improvements will not transfer, and the paper tests only 7B-scale models.

Editorial extensions

If this is right

  • Any LLaVA-1.5-style architecture can adopt the spatial tokens by swapping the MLP projector for the SFE+DFI projector, with no change to the LLM, training data, or visual token budget beyond six tokens.
  • Fine-grained perception tasks gain most: the RefCOCO/+/g grounding scores jump by roughly five points and MMVP hallucination accuracy rises from 24.7 to 31.3, so spatial tokens are a cheap way to reduce CLIP-blind confusions.
  • The gains transfer beyond the base setup: replacing CLIP-ViT-L/14 with SigLIP-L/16 and moving the module onto InternVL-2.0 both produce improvements, so the extractor is not tied to one encoder or one MLLM family.
  • Inference cost stays flat: at 582 total visual tokens versus 576, throughput is nearly identical (20.3 to 20.5 versus 20.8 tokens per second), so serving cost does not rise with the accuracy gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper leaves open is whether the six-token extractor compounds with higher-resolution or dynamic-resolution inputs, since the gains are concentrated in fine-grained tasks where resolution is the usual lever.
  • Because DFI consistently helped the Cropping variant but nudged several Pooling scores down in the paper's own analysis, a testable extension is to apply the integrator selectively per task or to train a router that picks cropping or pooling tokens per image.
  • The claim that token order encodes spatial semantics suggests a direct probe: permute or reverse the order of the six tokens and measure the benchmark drop; if order matters, the ordering is carrying information, and if not, the multi-scale content alone is what counts.
  • The mechanism is a general recipe for any ViT-based encoder: reshape, read scales with convolutions, fuse by cross-attention. Applying it to intermediate ViT layers rather than only the final $24\times24$ grid could extend the gains, since the paper's SFE uses only the final-layer features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LLaVA-SP, an enhancement to LLaVA-1.5's projector that adds six spatial visual tokens to the original 576 patch tokens. The Spatial Feature Extractor (SFE) uses either progressive cropping or adaptive pooling to obtain multi-scale feature maps, then applies full-size convolution kernels to produce one token per scale. The Detail Feature Integrator (DFI) uses cross-attention to inject finer-grained features into these tokens. The authors fine-tune Vicuna-7B with LoRA on LLaVA-1.5's training data and report that both variants outperform a LoRA-reproduced LLaVA-1.5 baseline on 10 of 11 benchmarks with nearly identical inference speed; they also test the module on SigLIP and InternVL-2.0.

Significance. If the reported gains replicate, the contribution is practically valuable: a six-token overhead that preserves inference latency and improves several fine-grained visual benchmarks, with a mechanism that is clearly described and reimplementable. The ablation set in Tables 3, 4, and 6 usefully isolates the main design degrees of freedom, and the generalization experiments in Table 7 are a welcome check. The authors also provide a public code/model link. The main caveat is that the quantitative claims currently rest on single runs and on design choices selected with the same benchmark suite used for the headline evaluation; several decisive deltas are small and may be within run-to-run variance. The significance is therefore conditional on additional statistical evidence.

major comments (3)
  1. [§4.2, Table 1] The headline claim that LLaVA-SP 'significantly' outperforms LLaVA-1.5 is supported only by single-run numbers, and several of the decisive deltas are small (e.g., VQAv2 +0.8, GQA +0.6, SEED-IMG +0.6, POPE +0.7), while MME-P decreases by 1.5 to 3.6 points relative to the LoRA baseline. The paper reports no error bars, no number of seeds, and no detailed evaluation protocol (e.g., exact inference settings for VQAv2 and TextVQA). For 7B instruction-tuned models with LoRA, these deltas are comparable to common run-to-run variance, so the 10-of-11 conclusion is not yet established. I request multi-seed runs or at least an uncertainty estimate for the main comparison.
  2. [§4.4, Tables 4 and 6] The design hyperparameters N=6 and Zs-big=5x5 are selected by comparing AvgN on the same 11 benchmarks used for the headline evaluation, and the decisive differences among options are small: in Table 4, N=6 gives AvgN 64.6 versus 64.1 for N=4 and ties N=12 at 64.6; in Table 6, Zs-big=5x5 gives AvgN 65.1 versus 64.7 for Zs-big=11x11 in the Pooling variant, with several individual benchmarks declining. Since every configuration is evaluated on the same suite and only one run is reported, the chosen hyperparameters may be overfit to this benchmark set. Please either use a held-out validation split for design choices or report multi-seed intervals for all configurations.
  3. [§4.4, Deep analysis] The paper explicitly concedes that DFI hurts the Pooling variant on most individual benchmarks. Concretely, relative to SFE-only Pooling, the chosen Zs-big=5x5 configuration improves VizWiz (+2.0), MME-P (+1.1), LLaVA-Bench (+0.8), and MM-Vet (+1.3), but declines on GQA (-0.2), SQA (-1.0), TextVQA (-0.2), POPE (-0.3), MMB (-0.6), and SEED (-0.4), with VQAv2 essentially flat (-0.1). On single runs, this pattern is equally consistent with noise and does not substantiate the claim that DFI 'further enhances visual representation.' The DFI contribution needs a more robust empirical basis, such as consistency across seeds or a focused evaluation on the fine-grained tasks it is designed for.
minor comments (5)
  1. [Fig. 1 and Abstract] The phrasing 'outperform the fully trained LLaVA-1.5' is stronger than Table 1 shows: against the full-training baseline LLaVA-1.5†, MME-P decreases by 34.8 to 36.9 points, and MM-Vet improves by only 0.1 for the Cropping variant. Please specify which baseline is used for each claim.
  2. [§4.1 and Table 1] The benchmark list in Section 4.1 mentions MMVP, but Table 1 does not include it; MMVP appears only in Table 10. Please make the benchmark list consistent.
  3. [Appendix A] Projector size is reported as '1536 MiB' and '836 MiB,' which mixes memory and parameter units; please report parameter counts and, ideally, FLOPs per image.
  4. [References] References [48] and [49] are the same paper (Singh et al., 'Towards VQA models that can read'); please deduplicate.
  5. [Figure 1] The bar chart in Figure 1 is visually cluttered, with overlapping value labels; please redraw it for readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: LLaVA-SP's central claim is an empirical benchmark comparison, not a derivation that reduces to its own inputs.

full rationale

The paper's central claim is that adding six visual spatial tokens via the SFE and DFI projector improves MLLM visual understanding, supported by benchmark evaluations in Tables 1, 3, 4, 6, and 7. No load-bearing step defines a predicted quantity in terms of a fitted constant or a self-cited theorem. The number of tokens and kernel sizes are chosen by ablations on the same benchmark suite, which is a selection-bias and overfitting concern rather than a circular reduction: the headline numbers are not mathematically forced by the chosen hyperparameters. The paper's only self-citation of note is [55] (Vit-CoMER, sharing author Xinliang Wang), used to support the general premise that ViT models local relationships poorly; this premise is not the paper's derived result and is not used to forbid alternatives. The evaluation is against external benchmarks, and the models are trained with standard instruction-tuning. The limitation section acknowledges scope constraints (7B models only and large projector parameters), but those are stated limitations, not circularity. Overall, the derivation chain is self-contained with respect to circularity; reported gains may be fragile, but that is an empirical robustness issue, not a circular one.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on benchmark evaluations, not on a mathematical derivation. The main free parameters are architectural choices (token count, kernel sizes, DFI granularity) tuned on the evaluation suite itself. No external physical or conceptual entities are postulated; the spatial tokens are trained components with ablation support. The key domain assumptions are that CLIP-ViT restricts local relationship modeling, that the chosen benchmarks measure the relevant abilities, that the training recipe matches LLaVA-1.5 apart from LoRA, and that reported single-run scores are stable.

free parameters (5)
  • Number of spatial tokens N = 6
    Selected by ablation on the benchmark suite in Table 4; N=12 reaches similar AvgN but doubles parameters, so 6 is chosen for efficiency.
  • Convolutional kernel sizes k = 4, 8, 12, 16, 20, 24
    Each kernel covers one of the six multi-scale feature maps (Eq. 2); the scale/kernel pairing is tuned in Tables 4 and 6.
  • DFI Zs-big kernel size = 16x16 (Zs-big 5x5)
    Ablation in Table 6 shows 5x5 gives the best average score, though per-benchmark effects are mixed.
  • Cropping stride = 2
    Determines the six feature-map scales (24,20,...4) in Figure 3a; chosen by design, not independently justified.
  • LoRA rank = 128
    Instruction-tuning hyperparameter in Table 11; part of the LoRA-fine-tuned setup but not the focus of the method.
assumptions (4)
  • domain assumption CLIP-ViT struggles to model local relationships between adjacent patches, harming detailed understanding.
    Introduction (Sec. 1) and cited [55]; if this premise is false, the motivation weakens, though the empirical gains could still hold.
  • domain assumption The 11 benchmark scores are a valid proxy for general and fine-grained visual understanding.
    All headline comparisons (Table 1) use these benchmarks; no external validation that gains transfer to other tasks.
  • domain assumption The training pipeline matches LLaVA-1.5 exactly except for LoRA fine-tuning.
    Sec. 4.1 states same components, datasets and two-stage strategy; if the recipe differs, the controlled comparison is compromised.
  • domain assumption Single-run evaluation scores are stable enough to support point-difference comparisons.
    All tables report one run per setting; no error bars or significance tests are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs." pith.science (2026). https://pith.science/paper/ZK2BFSLO

@misc{pith2026250700505,
  author       = {Pith},
  title        = {Pith review of: LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZK2BFSLO}},
  note         = {Machine review of arXiv:2507.00505}
}
read the original abstract

The architecture of multimodal large language models (MLLMs) commonly connects a vision encoder, often based on CLIP-ViT, to a large language model. While CLIP-ViT works well for capturing global image features, it struggles to model local relationships between adjacent patches, leading to weaker visual representation, which in turn affects the detailed understanding ability of MLLMs. To solve this, we propose LLaVA-SP, which only adds six spatial visual tokens to the original visual tokens to enhance the visual representation. Our approach offers three key advantages: 1) We propose a novel Projector, which uses convolutional kernels to derive visual spatial tokens from ViT patch features, simulating two visual spatial ordering approaches: "from central region to global" and "from abstract to specific". Then, a cross-attention mechanism is applied to fuse fine-grained visual information, enriching the overall visual representation. 2) We present two model variants: LLaVA-SP-Cropping, which focuses on detail features through progressive cropping, and LLaVA-SP-Pooling, which captures global semantics through adaptive pooling, enabling the model to handle diverse visual understanding tasks. 3) Extensive experiments show that LLaVA-SP, fine-tuned with LoRA, achieves significant performance improvements across various multimodal benchmarks, outperforming the state-of-the-art LLaVA-1.5 model in multiple tasks with nearly identical inference latency. The code and models are available at https://github.com/CnFaker/LLaVA-SP.

Figures

Figures reproduced from arXiv: 2507.00505 by the authors.

Figure 1
Figure 1. Our models, fine-tuned with LoRA, outperform the fully trained LLaVA-1.5 in 10 out of 11 multimodal bench￾marks. We name the model that employs the cropping operation as LLaVA-SP-Cropping and the one that uses the pooling operation as LLaVA-SP-Pooling. representation in MLLMs has primarily focused on the vi￾sual aspect. To reduce hallucinations in MLLMs caused by visual content, various strategies have been employed… view at source ↗
Figure 2
Figure 2. The architecture of LLaVA-SP is based on the structure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. SFE Structure. (a) illustrates the process of obtaining precise multi-scale features using the cropping operation, simulating the arrangement of visual spatial features as “from central region to global”, emphasizing details in image regions. (b) demonstrates the method of obtaining abstract feature maps at multi-scale using adaptive pooling, simulating the arrangement of visual spatial features “from abstract to sp… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: DFI architecture. Integrating Zs-big details and inject￾ing them into Zs-small. kernels with sizes k = 4, 8, 12, 16, 20, 24 can fully cover (Zp1, Zp2, Zp3, Zp4, Zp5, Zp6) and compute visual spatial features Zs-small through concatenation in sequence dimen￾sion: Zsi = c…
Figure 5
Figure 5. Figure 5: , the vertical axis of the attention map represents the queries, while the horizontal axis represents the keys. In LLaVA-SP-Cropping, attention is distributed more uni￾formly, whereas in LLaVA-SP-Pooling, it is more concen￾Semantic Image LLaVA-SP-Cropping Query Query 0…
Figure 6
Figure 6. Figure 6: Examples of LLaVA-SP-Cropping and LLaVA-SP [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Examples of LLaVA-SP-Cropping and LLaVA-SP-Pooling on MME dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Deep Analysis between LLaVA-SP-Cropping, LLaVA-SP-Pooling and LLaVA-1.5. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Compared to LLaVA-1.5, LLaVA-SP-Cropping captured the Mercedes logo on Faker’s clothing, while LLaVA-SP-Pooling [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 33 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, pages 23716–23736, 2022. 2

    work page 2022

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 1, 2, 5

    arXiv 2023

  3. [3]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024. 3, 5, 8

    work page 2024

  4. [4]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In CVPR, pages 3558–3568, 2021. 5

    work page 2021

  5. [5]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 5

    arXiv 2023

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2

    arXiv 2023

  7. [7]

    A sin- gle transformer for scalable vision-language modeling.arXiv preprint arXiv:2407.06438, 2024

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A sin- gle transformer for scalable vision-language modeling.arXiv preprint arXiv:2407.06438, 2024. 2

    arXiv 2024

  8. [8]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pages 104–120. Springer, 2020. 5

    work page 2020

Show all 71 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 3, 8

    2024 arXiv

  2. [10]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 3

    2024

  3. [11]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , ...

    2023

  4. [12]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, pages 49250–49267, 2023. 2, 5

    2023

  5. [13]

    Unveiling encoder-free vision-language models

    Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. arXiv preprint arXiv:2406.11832,

    arXiv

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1, 2

    2010 arXiv

  7. [15]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 5, 7, 13

    2023 arXiv

  8. [16]

    Mmgpt4lf: Leverag- ing an optimized pre-trained gpt-2 model with multi-modal cross-attention for load forecasting

    Mingyang Gao, Suyang Zhou, Wei Gu, Zhi Wu, Haiquan Liu, Aihua Zhou, and Xinliang Wang. Mmgpt4lf: Leverag- ing an optimized pre-trained gpt-2 model with multi-modal cross-attention for load forecasting. Applied Energy, 392: 125965, 2025. 1

    2025

  9. [17]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In CVPR, pages 6904–6913, 2017. 5

    2017

  10. [18]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018. 5

    2018

  11. [19]

    Efficient multi- modal learning from data-centric perspective

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multi- modal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024. 2

    2024 arXiv

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 5

    2021 arXiv

  13. [21]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019. 5

    2019

  14. [22]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024. 2

    2024 arXiv

  15. [23]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In CVPR, pages 5648–5656, 2018. 2

    2018

  16. [24]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 5, 7

    2023 arXiv

  17. [25]

    Otterhd: A high-resolution multi- modality model

    Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi- modality model. arXiv preprint arXiv:2311.04219, 2023. 2

    2023 arXiv

  18. [26]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888–12900, 2022. 2

    2022

  19. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 2, 5 9

    2023

  20. [28]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 5, 8

    2023 arXiv

  21. [29]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403.18814, 2023. 1

    2023 arXiv

  22. [30]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024. 1

    2024

  23. [31]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pages 26689–26699, 2024. 2

    2024

  24. [32]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 1, 3

    2023 arXiv

  25. [33]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023. 1, 2, 3, 5, 8, 12

    2023 arXiv

  26. [34]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, pages 34892–34916,

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 3

    2024

  28. [36]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233. Springer, 2025. 5, 7

    2025

  29. [37]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11976–11986, 2022. 3

    2022

  30. [38]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, pages 2507–2521, 2022. 5

    2022

  31. [39]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models

    G Luo, Y Zhou, Y Zhang, X Zheng, and X Sun. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv.2403.03003, 2024. 3

    2024 arXiv

  32. [40]

    Unirgb-ir: A unified frame- work for rgb-infrared semantic tasks via adapter tuning

    Yuan Maoxun, Cui Bo, Zhao Tianyi, Wang Jiayi, Fu Shan, Yang Xue, and Wei Xingxing. Unirgb-ir: A unified frame- work for rgb-infrared semantic tasks via adapter tuning. arXiv preprint arXiv:2404.17360, 2024. 1

    2024

  33. [41]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. 2

    2023

  34. [42]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Ni...

    2023

  35. [43]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. In NeurIPS, 2011. 5

    2011

  36. [44]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763, 2021. 1, 3, 5

    2021

  37. [45]

    Introducing our multimodal models, 2023

    Bavishi Rohan, Elsen Erich, Hawthorne Curtis, Nye Maxwell, Odena Augustus, Somani Arushi, and Ta¸sırlar Sagnak. Introducing our multimodal models, 2023. 2

    2023

  38. [46]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 5

    2021 arXiv

  39. [47]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 3

    2024 arXiv

  40. [48]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 5

    2019

  41. [49]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 2

    2019

  42. [50]

    Adapool: Expo- nential adaptive pooling for information-retaining downsam- pling

    Alexandros Stergiou and Ronald Poppe. Adapool: Expo- nential adaptive pooling for information-retaining downsam- pling. IEEE Transactions on Image Processing, 32:251–266,

  43. [51]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  44. [52]

    Visual autoregressive modeling: Scalable im- age generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction. arXiv preprint arXiv:2404.02905, 2024. 2, 15

    2024 arXiv

  45. [53]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pages 9568–9578, 2024. 5, 8, 12

    2024

  46. [54]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  47. [55]

    Vit-comer: Vision transformer with convolu- tional multi-scale feature interaction for dense predictions

    Chunlong Xia, Xinliang Wang, Feng Lv, Xin Hao, and Yifeng Shi. Vit-comer: Vision transformer with convolu- tional multi-scale feature interaction for dense predictions. In CVPR, pages 5493–5502, 2024. 2

    2024

  48. [56]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 3

    2024 arXiv

  49. [57]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 2

    2024

  50. [58]

    Dense connector for mllms

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. arXiv preprint arXiv:2405.13800, 2024. 3

    2024 arXiv

  51. [59]

    Image difference cap- tioning with pre-training and contrastive learning

    Linli Yao, Weiying Wang, and Qin Jin. Image difference cap- tioning with pre-training and contrastive learning. In AAAI, pages 3108–3116, 2022. 2

    2022

  52. [60]

    Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models

    Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compres- sion from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20985, 2024. 3, 5, 7, 8

    2024 arXiv

  53. [61]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, pages 69–85. Springer, 2016. 5, 8

    2016

  54. [62]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 5

    2023 arXiv

  55. [63]

    Improving rgb-infrared object detection with cascade alignment-guided transformer

    Maoxun Yuan, Xiaorong Shi, Nan Wang, Yinyan Wang, and Xingxing Wei. Improving rgb-infrared object detection with cascade alignment-guided transformer. Information Fusion, 105:102246, 2024. 1

    2024

  56. [64]

    Lane detection transformer based on multi- frame horizontal and vertical attention and visual trans- former module

    Han Zhang, Yunchao Gu, Xinliang Wang, Junjun Pan, and Minghui Wang. Lane detection transformer based on multi- frame horizontal and vertical attention and visual trans- former module. In ECCV, pages 1–16. Springer, 2022. 1

    2022

  57. [65]

    Mmicl: Empowering vision-language model with multi-modal in-context learning

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915, 2023. 2

    2023 arXiv

  58. [66]

    Removal and selection: Improving rgb- infrared object detection via coarse-to-fine fusion

    Tianyi Zhao, Maoxun Yuan, Feng Jiang, Nan Wang, and Xingxing Wei. Removal and selection: Improving rgb- infrared object detection via coarse-to-fine fusion. arXiv preprint arXiv:2401.10731, 2024. 1

    2024

  59. [67]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2

    2023 arXiv

  60. [68]

    Deformable detr: Deformable trans- formers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 3 11 A. Implementation details Hyperparameters. The experimental setup follows LLaV A-1.5 ...

    2010 arXiv

  61. [70]

    Future work will involve experiments on various LLMs such as Qwen2.5, Mistral, and LLaMA3

    Did not utilize larger-scale LLMs: The experiments were conducted only on LLMs with 7B parameters, and the effec- tiveness of the method has not been validated on larger-scale LLMs. Future work will involve experiments on various LLMs such as Qwen2.5, Mistral, and LLaMA3

  62. [71]

    The input and output channels of the convolutional kernels are 1024 (equal to the visual feature dimension output by ViT) and 512, respectively

    Large Parameters: The SFE employs convolutional ker- nels to extract spatial information from visual features. The input and output channels of the convolutional kernels are 1024 (equal to the visual feature dimension output by ViT) and 512, respectively. Additionally, large k...

  63. [512]

    CLIP-bind pairs

    The total parameters of the projector are 1536 MiB. We also experimented with transformer blocks, a 4 layers encoder-decoder structure, where the input and output chan- nels are both 1024. The total parameters of the projector are 836 MiB. As mentioned in the main paper, the c...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.