Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models

Aarti Ghatkesar; Ganesh Venkatesh

REVIEW 4 major objections 6 minor 26 references

Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that blanking input tokens, adding a visual loss, using separate vision/text weights, and training on synthetic grids can make an 8B multimodal model surpass an 11B model on SpatialEval at lower input resolution.

desk verdict A plausible MLLM training recipe for visual grounding, but the headline SpatialEval gains are confounded by swapping the input encoder to AIM-v2 without an ablation; the central causal claim is not yet supported. read the letter →

arxiv 2505.05626 v3 pith:YYKGEA33 submitted 2025-05-08 cs.CV cs.AI

Aarti Ghatkesar , Ganesh Venkatesh This is my paper · ORCID

classification cs.CVcs.AI

keywords multimodallargelanguagemodelsvisualgroundingpriorsspatialreasoningauxiliarylosstokenmaskingdisentangledvision-textarchitecturesyntheticdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the visual grounding weakness of multimodal LLMs is fixable through training rather than only through larger models or stronger encoders. It argues that MLLMs already develop an unlabeled, patch-level sense of image content, and that this latent ability can be amplified by four techniques: an auxiliary visual loss that makes the language backbone predict features from a strong image encoder, blanking leading text tokens so responses must start from the image, separate attention weights for image and text tokens, and synthetic grid images that force spatial reading. If the paper is right, the payoff is concrete: an 8B model at 224x224 resolution beats the Llama 3.2 11B model on SpatialEval, with more than 10 percentage points of gain on the grid task and consistent gains on other spatial subsets.

What carries the argument

The argument is carried by four training-time mechanisms acting on the LLM backbone, plus a diagnostic. VISUALLOSS adds an MSE term that forces the backbone's visual-token representations to match the representations produced by a pretrained I-JEPA encoder, so the language model must encode the whole image rather than only text-relevant regions. BLANKTOKENS replaces the first five input tokens and about 20% of later tokens with a designated blank token, preventing the model from autocompleting from language cues and forcing it to start from the image. The disentangled architecture gives image and text tokens separate self-attention weights, resolving the conflict between next-token prediction and the visual representation objective; this duplicates parameters but adds no inference-time compute because only the pathway for the current token's modality is active. Synthetic grid images built from Open Images objects with programmed questions supply training examples whose answers cannot be guessed from language priors. The upstream probe, reading the LM head's top token for each visual patch, shows the model already attaches semantic labels to image patches, motivating the interventions. The final PERCEPTLLM replaces the input encoder with AIM-v2, which the paper credits for the strongest results.

What would settle it

Run the same baseline with only the input encoder changed from CLIP to AIM-v2 and none of the proposed training modifications, then evaluate on Grid and MazeNav; if that model approaches 50.8 and 35.0 respectively, the training innovations are not the cause of the headline gains.

Watch

Extended reading notes

Core claim

The central claim is that MLLMs can be trained to rely on what they actually see instead of on language priors, and the paper supports this with a proposed model, PERCEPTLLM. The authors first probe internal visual tokens by passing them through the LM head and find that the model already attaches meaningful semantic labels to image patches without explicit supervision. Building on that, they add an auxiliary visual loss that forces the LLM backbone's visual-token representations to match I-JEPA's representations, blank out initial and random input tokens to block language-based autocompletion, give image and text tokens independent attention weights, and augment training with synthetic grid images whose answers cannot be guessed from text. The final configuration also swaps the input encoder to AIM-v2. The paper reports that this combination lowers next-token loss on the visually dependent SpatialMM dataset and raises SpatialEval accuracy across all subsets, with the largest gains on Grid and MazeNav, so that PERCEPTLLM outperforms the larger Llama 3.2 11B model while being more efficient in size, architecture, and input resolution.

Load-bearing premise

The load-bearing premise is that the reported gains come from the proposed training modifications, yet the final model also switches the input encoder to AIM-v2 and the paper never ablates that encoder swap alone under the same training recipe.

Editorial extensions

If this is right

If PERCEPTLLM is right, visual grounding in MLLMs can be improved during training without extra visual annotations and without additional encoders at inference time.
A smaller model at lower input resolution can beat a larger, higher-resolution model on spatial reasoning benchmarks, which matters for deployment efficiency.
The consistent per-subset gains on SpatialEval suggest the techniques transfer across different kinds of spatial questions, not just grid layouts.
The upstream analysis predicts that visually dependent tokens, such as objects, directions, colors, and shapes, will be predicted with lower loss after training, giving a finer-grained diagnostic than overall accuracy.
Because the auxiliary encoder used for the visual loss can be swapped, the framework offers a path for future encoder advances to flow into the language backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the final PERCEPTLLM row swaps the input encoder to AIM-v2 under the same headline as the other techniques, the Grid and MazeNav gains are not cleanly attributable to the training modifications; an ablation with AIM-v2 alone under the same recipe would settle it.
Editorial inference: BLANKTOKENS is a form of input corruption during training, so a natural control is standard dropout or random token masking without a dedicated blank token; if that control matched the gains, the specific blank-token mechanism would not be the cause.
Editorial inference: the paper's Future Work passage acknowledges that the larger capacity of PERCEPTLLM may make the baseline data mixture and hyperparameters sub-optimal and could increase overfitting on synthetic data, a caveat that qualifies the generality of the reported configuration.
Editorial inference: the synthetic grid data could be varied in object count, grid size, and question type to test whether the technique generalizes from spatial layout to counting, matching, and relation tasks.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you work on multimodal grounding, but budget your time. The paper puts together a clean, practical recipe: an auxiliary MSE loss that makes the LLM's visual tokens predict features from a self-supervised encoder (I-JEPA, later AM-Radio), blanking the first N text tokens plus random masking to weaken language priors, disentangled vision/text transformer weights, and synthetic grid data from Open Images. None of that is individually revolutionary, but the combination is new, and the upstream analysis on SpatialMM (NTP loss) is a nice diagnostic. The writing is readable and the recipe is concrete enough to reproduce.

The soft spots matter more. The core claim is causal: the proposed techniques, not the stronger input encoder, drive the gains. Table 2 doesn't support that. Rows 1-5 keep the CLIP input encoder; the final PERCEPTLLM row simultaneously introduces AIM-v2 as the input encoder. So the jumps on Grid and MazeNav, and the drops on Map and Real, could be entirely an encoder effect. No ablation fixes the training recipe and varies only the input encoder. On top of that, the 'strongest results' statement is contradicted by the paper's own table on two of four subsets: PERCEPTLLM's Map is 55.13 vs 64.1 for the row above, and Real is 52.6 vs 53.3. Also, the benchmark prompt/parsing modifications are not re-run on Llama 3.2 under the same protocol, so that comparison is weakened, and there are no error bars or statistical tests across runs.

I don't think this is a broken idea; the confound is addressable. The paper gives enough detail that an ablation would settle it, and the auxiliary-visual-loss idea is worth testing even if the current evidence doesn't isolate it. But as written, the headline claim is under-supported, and the overclaim about strongest results should be fixed.

Who's this for? People designing training objectives for MLLMs, especially those interested in using self-supervised encoders as supervision targets. It deserves a serious referee: it's concrete, falsifiable, and the central recipe is plausible. But I would not cite it in its current form, and I'd want the ablation before relying on the numbers.

Referee Report

4 major / 6 minor

Summary. The paper proposes PERCEPTLLM, a multimodal LLM built on the LLaVA recipe with a Llama 3.1 8B backbone, and attributes its accuracy gains to four training interventions: VISUALLOSS (an auxiliary MSE loss matching the LLM's visual-token features to features from an independently pretrained vision encoder), BLANKTOKENS (masking the first N text tokens and a random 20% of later tokens during training), independent weights for image and text pathways within the transformer, and synthetic grid data generated from OpenImages objects. The evaluation has two parts: an upstream next-token-prediction loss on SpatialMM and subset accuracies on SpatialEval. The headline claim is that PERCEPTLLM obtains a more than 10 percentage point boost over the baseline and outperforms Llama 3.2 11B while being smaller and lower-resolution. The causal claim that the proposed techniques, rather than the simultaneous switch to the AIM-v2 input encoder, drive the improvement is the central point that needs scrutiny.

Significance. If established, the paper would be a useful empirical contribution to visually grounded MLLM training: VISUALLOSS is self-supervised in the sense of not requiring region annotations, BLANKTOKENS is a simple regularization, and the released prompting and parsing code aids reproducibility. The upstream token-level analysis is a nice microscopic check. However, as submitted the evidence does not yet support the causal attribution, because the final model confounds the training recipe with an input encoder swap and the modified SpatialEval protocol is not re-baselined. The evaluation is not circular: the auxiliary representations come from independently pretrained encoders and the benchmark is not used for training. The contribution therefore hinges on additional controlled experiments rather than on a logical flaw.

major comments (4)

[Section 4.2, Table 2] The row labeled PERCEPTLLM simultaneously introduces the AIM-v2 input encoder, while all earlier rows use the CLIP input encoder; there is no ablation that changes only the input encoder under a fixed training recipe. Consequently, the gains on Grid (43.5 to 50.8) and MazeNav (26.7 to 35.0) cannot be attributed to VISUALLOSS, BLANKTOKENS, independent weights, or synthetic data; they may be due entirely to the stronger encoder. The same confound makes the drops on Map (64.1 to 55.13) and Real (53.3 to 52.6) between VISUALLOSSADV and PERCEPTLLM uninterpretable. Please add an ablation that applies the proposed training recipe with the CLIP encoder and with AIM-v2, and report all four subsets per configuration.
[Section 4.2, footnote 2] The paper states that the SpatialEval prompting and parsing were modified for this work and that the code is released, but it does not re-run the baseline or Llama 3.2 under the modified protocol. Without such a re-baselining, the comparison against Llama 3.2 11B and even the baseline is not controlled, because changes to prompt templates and answer parsing can change measured accuracy independently of model capability. Please re-evaluate the baseline and at least Llama 3.2 with the released protocol, or quantify the effect of the prompt and parser changes.
[Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported for either the next-token-prediction losses in Table 1 or the accuracies in Table 2. Given that several adjacent rows in Table 2 differ by only a few points (for example, the Real subset values 52.6 versus 53.3) and some baseline values are in the 15 to 50 range, the claimed improvements and the 1.52 versus 1.365 average comparison rest on point estimates whose variability is unknown. Please report variance across at least three evaluation runs or a statistical test.
[Section 4.2 and Section 5] The statement that PERCEPTLLM 'yields our strongest results' is not supported by Table 2 on two subsets: PERCEPTLLM scores 55.13 on Map, below the 64.1 of +VISUALLOSSADV, and 52.6 on Real, below the 53.3 of +Synthetic and +VISUALLOSSADV. The aggregate Average does not override these subset-level regressions; the text should either report the per-subset trade-offs honestly or provide a stated criterion under which the aggregate is the primary metric.

minor comments (6)

[Section 6.2, Eq. (3)] The semantics of the mask M appear inverted: the text says M determines whether to blank a token, but the formula keeps Tin when M is True and substitutes bId otherwise. Please align the formula with the intended behavior or reword the definition.
[Sections 3.2, 3.4, and Table 2] The relationships among the auxiliary encoders I-JEPA and AM-Radio and the final input encoder AIM-v2 should be stated in one place; as written, the reader cannot tell which encoder is used in which row of Table 2 without cross-referencing the appendix.
[Table 2 caption] The Average column is labeled 'mean of normalized accuracy improvement'; please define the normalization in the caption and note whether the average is over the four subset ratios or a different quantity.
[Section 3.4] The claim that the disentangled architecture incurs 'no additional computational or memory access overhead during inference' should be quantified or softened; doubling the weight matrices increases memory residency even if per-token FLOPs are unchanged.
[Section 6.4] The training setup lists beta, N, and the synthetic-data fraction, but not batch size, learning rate, number of steps or epochs, pretraining mixture size, or compute; please add these details for reproducibility.
[Title and Abstract] The title and abstract promise an analysis of visual attention, but the paper never measures attention weights; the evidence is per-token prediction loss and accuracy. Please add an attention analysis or adjust the framing.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is against an external benchmark and no test quantity is built from a fitted parameter or self-citation.

full rationale

I walked the claimed derivation chain: VISUALLOSS (Eq. 1) trains the LLM visual tokens to match representations from an independently pretrained encoder (I-JEPA); BLANKTOKENS (Eq. 3) masks input tokens; disentangled weights come from cited external architectures; synthetic data is generated programmatically from OpenImages. The SpatialEval benchmark and SpatialMM NTP-loss analysis are external evaluation sets, not training objectives whose values are fed back into the model. No equation in the paper reduces a reported accuracy to a fitted quantity, and no load-bearing premise rests on a self-citation or an author-imported uniqueness theorem. The substantive weakness is experimental attribution: the PERCEPTLLM row in Table 2 simultaneously introduces the AIM-v2 input encoder, so the headline gains over the additive rows could be caused by the encoder swap rather than by the proposed training techniques; the paper itself lists ablations as future work and notes modified prompting/parsing. This is a confound and a comparison-fairness concern, not a circular derivation, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The free parameters are hand-set constants and design choices that are not swept, and the axioms are inherited or asserted assumptions that the evaluation does not independently verify. No new physical or conceptual entities are introduced.

free parameters (6)

beta (visual loss weight) = 0.5
Set in Appendix 6.4; no sensitivity analysis reported.
N (initial blanked tokens) = 5
Blanked first N=5 text tokens in BLANKTOKENS; chosen by hand.
random blank fraction = 20%
Randomly blank 20% of subsequent input tokens; chosen by hand.
synthetic data mixture = 25%
Stage 3 data mixture is 75% standard instruction data and 25% synthetic grid data; no ablation.
auxiliary vision encoder choice = I-JEPA, then AM-Radio, then AIM-v2
The auxiliary encoder is swapped across rows in Table 2; the final model uses AIM-v2 as both input and auxiliary, conflating encoder gains with training gains.
synthetic grid configuration = 4x4 or 8x8 grids
Grid size and object counts are design choices; exact values are not fully specified.

assumptions (6)

domain assumption The standard CLIP vision encoder provides an adequate input representation for the LLM.
Used throughout as the input encoder for the baseline and ablation rows; inherited from LLaVA.
domain assumption Predicting I-JEPA/AIM-v2 features from LLM visual tokens improves visual understanding.
VISUALLOSS (Eq. 1) relies on the auxiliary encoder's features being a good supervisory target; no evidence that this target is optimal.
domain assumption MLLMs develop localized semantic understanding of image patches, as shown qualitatively in Section 3.1.
The motivation for VISUALLOSS rests on this qualitative observation; it is not quantified.
domain assumption Synthetic grid data transfers to SpatialEval benchmark tasks.
The improvements from +Synthetic in Table 2 assume transfer from grid layouts to maze, navigation, map, and real-image tasks.
ad hoc to paper The modified SpatialEval prompting and parsing does not change relative model rankings.
The footnote in Section 4.2 states the benchmark was enhanced, but the baseline is not re-measured under the modified protocol.
ad hoc to paper Independent weights for image and text pathways do not require rebalancing the data mixture.
The Future Work paragraph admits the baseline data mixture may be suboptimal for the new architecture.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models." pith.science (2026). https://pith.science/paper/YYKGEA33

@misc{pith2026250505626,
  author       = {Pith},
  title        = {Pith review of: Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYKGEA33}},
  note         = {Machine review of arXiv:2505.05626}
}

read the original abstract

Achieving deep alignment between vision and language remains a central challenge for Multimodal Large Language Models (MLLMs). These models often fail to fully leverage visual input, defaulting to strong language priors. Our approach first provides insights into how MLLMs internally build visual understanding of image regions and then introduces techniques to amplify this capability. Specifically, we explore techniques designed both to deepen the model's understanding of visual content and to ensure that these visual insights actively guide language generation. We demonstrate the superior multimodal understanding of our resultant model through a detailed upstream analysis quantifying its ability to predict visually-dependent tokens as well as 10 pt boost on visually challenging tasks.

Figures

Figures reproduced from arXiv: 2505.05626 by the authors.

**Figure 2.** Proposed modifications to MLLM training that en [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Examples of synthetic grid data for spatial awareness. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Examples of synthetic visual samples with objects placed on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Examples of spatial queries of type Describe, Directional, Distance, Location generated for a visual sample. both the vision encoder and the LLM backbone frozen. We do not apply VISUALLOSS and BLANKTOKENS in this stage. Stage 2 (End-to-End Continued Pretraining): Subs…

**Figure 6.** Figure 6: This figure visualizes the next-token prediction loss on a per-token basis for our proposed approaches. The [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

[1]

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timoth ´ee Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Mar...

work page 2024
[2]

Self-supervised learning from images with a joint-embedding predictive architecture

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023. 3, 5, 1

work page 2023
[3]

Hallucination of multimodal large language models: A survey

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1, 2

arXiv 2024
[4]

From colouring-in to pointillism: revisiting semantic segmentation supervision

Rodrigo Benenson and Vittorio Ferrari. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142, 2022. 4

arXiv 2022
[5]

Mul- timodal autoregressive pre-training of large vision encoders

Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guil- herme Turrisi da Costa, Louis B´ethune, Zhe Gan, et al. Mul- timodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024. 5

arXiv 2024
[6]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1, 4, 5

arXiv 2024
[7]

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 2020. 4

work page 2020
[8]

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956–1981,

work page 1956

Show all 26 references

[9]

Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models

Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.04996, 2024. 3, 5

2024 arXiv
[10]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 4

2023
[11]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

2021
[12]

Vision language models are blind

Pooyan Rahmanzadehgervi, Logan Bolton, Moham- mad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 18–34, 2024. 1

2024
[13]

Am-radio: Agglomerative vision foundation model reduce all domains into one

Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024. 5

2024
[14]

Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

2024
[15]

Eagle: Exploring the design space for multimodal llms with mixture of encoders, 2025

Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catan- zaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal llms wi...

2025
[16]

An empirical analysis on spatial reason- ing capabilities of large multimodal models

Fatemeh Shiri, Xiao-Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reason- ing capabilities of large multimodal models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21440–21455, 2024. 4

2024
[17]

Sparkle: Mastering basic spatial capabilities in vision language models elicits gen- eralization to composite spatial reasoning

Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, and Jinhua Zhao. Sparkle: Mastering basic spatial capabilities in vision language models elicits gen- eralization to composite spatial reasoning. arXiv preprint arXiv:2...
[18]

Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms

Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms. Advances in Neural Information Proces...

2024
[19]

Eyes wide shut? exploring the visual shortcomings of multimodal llms

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 2

2024
[20]

Is a picture worth a thousand words? delving into spatial reasoning for vi- sion language models

Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vi- sion language models. In Advances in Neural Information Processing Systems, pages 75392–75421. Curran Asso- ciat...

2024
[21]

Cogvlm: Visual expert for pretrained lan- guage models

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained lan- guage models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024. 3

2024
[22]

Mm-llms: Recent ad- vances in multimodal large language models

Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent ad- vances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024. 1

2024 arXiv
[23]

Llava-grounding: Grounded visual chat with large multimodal models

Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19–35. Springer, 2024. 2, 3

2024
[24]

Ferret- v2: An improved baseline for referring and grounding with large language models

Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret- v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973,

arXiv
[26]

Appendix 6.1. VISUALLOSS Formulation Formally, we introduce an auxiliary vision encoder (say, A(·;θA)) in addition the the standard vision encoder (de- fined asG(·;θG) representing CLIP [11] in our implemen- tation connected to LLM backbone using MLP connector M). We choose th...
[2024]

3 Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models Supplementary Material

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timoth ´ee Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Mar...

work page 2024

[2] [2]

Self-supervised learning from images with a joint-embedding predictive architecture

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023. 3, 5, 1

work page 2023

[3] [3]

Hallucination of multimodal large language models: A survey

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1, 2

arXiv 2024

[4] [4]

From colouring-in to pointillism: revisiting semantic segmentation supervision

Rodrigo Benenson and Vittorio Ferrari. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142, 2022. 4

arXiv 2022

[5] [5]

Mul- timodal autoregressive pre-training of large vision encoders

Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guil- herme Turrisi da Costa, Louis B´ethune, Zhe Gan, et al. Mul- timodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024. 5

arXiv 2024

[6] [6]

The llama 3 herd of models

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1, 4, 5

arXiv 2024

[7] [7]

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 2020. 4

work page 2020

[8] [8]

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956–1981,

work page 1956