Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Vision-Language Models Can't See the Obvious

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that state-of-the-art vision-language models largely fail to detect visually obvious odd-one-out features, with the strongest model scoring only 47.6% F1 on natural images.

desk verdict SalBench is a useful odd-one-out benchmark with a solid negative result, but the 'obvious to humans' claim is undersupported without a human baseline. read the letter →

arxiv 2507.04741 v1 pith:HKFJMD53 submitted 2025-07-07 cs.CV

Yasser Dahou , Ngoc Dung Huynh , Phuc H. Le-Khac , Wamiq Reyaz Para , Ankit Singh , Sanath Narayan This is my paper
classification cs.CV
keywords saliencybenchmarklargevision-languagemodelsodd-one-outdetectionvisualsearchlow-levelfeaturesfeatureintegrationtheoryperceptualalignmentLVLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether large vision-language models can detect the low-level visual features that make an object conspicuously different from its surroundings, such as a large circle among small ones or a red object among green ones. It introduces SalBench, a benchmark built from synthetic 7x7 search arrays and natural odd-one-out photos, with three tasks: naming the odd feature, naming it given a text bounding box, and naming it given a red-box highlight. Across fourteen models, the paper finds that even the strongest evaluated model reaches only 47.6% F1 on natural-image detection, and no model substantially exceeds half on natural images. The authors argue this reveals a structural gap between LVLMs and human visual attention, and that the bottleneck lies more in the vision encoder than in the language model's knowledge of saliency.

What carries the argument

The central object is SalBench, a benchmark that repurposes the P3 synthetic visual-search arrays (2,514 seven-by-seven grids varying in color, orientation, or size) and the O3 natural odd-one-out photos (2,001 images with distractors and a singleton differing in color, shape, size, focus, orientation, location, or pattern). The three tasks — Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out — are designed to separate pure perception from language-guided attention: detection requires spotting the odd feature without local cues, referring gives a text bounding box, and visual referring highlights the target with a red box. Multi-label classification with exact-match accuracy and per-category F1 is the scoring mechanism, and a feature-integration-theory prompt supplies the closed set of feature categories. The key diagnostic machinery is a two-part ablation: an LLM knowledge test shows language models know feature integration theory well, while a zero-shot retrieval test on vision embeddings shows the visual representations are not discriminative enough, locating the failure in perception rather than reasoning.

What would settle it

Run the same three SalBench tasks with human participants on a sample of the O3 natural images and compare their feature-naming accuracy to the 47.6% F1 reported for the strongest model; if human accuracy on hard focus and size categories is far below near-perfect, the obviousness premise fails. Conversely, a model that reaches human-level accuracy on the natural split would refute the claim that current LVLMs cannot see the obvious.

Watch

Extended reading notes

Core claim

The central discovery is that current large vision-language models largely fail at odd-one-out perception when the target differs from distractors in low-level attributes such as size, orientation, focus, or pattern, and that the difficulty is not a matter of language understanding. On natural images, the strongest model scores 47.6% F1 on the detection task and most models score between 20% and 45%; on synthetic grids the same models reach 70–90%, showing that simplified arrays are far easier. Models are strongly biased toward color, which is near-ceiling on easy synthetic items, while size and focus degrade sharply at hard difficulty; adding more distractors monotonically lowers accuracy. A training experiment with one million synthetic saliency image-caption pairs does not fix the problem, and a vision-backbone retrieval probe shows the evaluated vision embeddings carry little discriminative information for the natural-image categories. The paper concludes that the perceptual alignment of LVLMs with human attention is still missing.

Load-bearing premise

The benchmark's central premise is that the odd-one-out targets are visually obvious to humans; no human observers were tested on the natural images, so if humans also find some of these images ambiguous, the claim that models cannot see the obvious would be overstated.

Editorial extensions

If this is right

  • SalBench can serve as a perceptual alignment probe for any LVLM, complementing benchmarks that measure high-level reasoning.
  • Current LVLMs deployed as robotic or GUI agents should not be assumed to notice salient scene changes, such as a person being swapped in a conversation.
  • Providing bounding boxes or red-box highlights only marginally helps on natural images, so the limitation is not primarily in following referring instructions.
  • Few-shot examples are not a reliable remedy and sometimes degrade performance, meaning simple in-context prompting will not close the gap.
  • The near-ceiling color performance alongside poor size and focus performance indicates that models rely on low-level pixel color rather than on relational geometry, pointing to a specific architectural weakness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a human-baseline study on the O3 natural images would test the central premise of obviousness; if humans are far from perfect on the hard focus and size categories, the headline gap shrinks.
  • The same benchmark could be used with a different training objective, such as contrastive learning on odd-one-out pairs, to test whether the vision-encoder bottleneck can be fixed without new architectures.
  • The distractor-count and difficulty-level trends suggest a scaling law for perceptual search that might predict when an LVLM will fail in the wild, which would be useful for safety cases in agents.
  • The low retrieval accuracy of the evaluated vision encoders on natural images implies that current vision-language pretraining objectives under-weight precisely the low-level features human attention uses, a hypothesis worth testing on newer backbones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SalBench, a benchmark built from the P3 and O3 datasets, with three tasks (Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out) that ask LVLMs to identify low-level visual attributes (color, orientation, size, focus, etc.) in which a singleton differs from distractors. The authors evaluate a wide range of open and closed LVLMs under zero-shot and few-shot settings, report F1 and matching-accuracy results, analyze difficulty levels and distractor counts, test vision-backbone retrieval, and train custom LVLMs on synthetic saliency data. The central claim is that current LVLMs, including GPT-4o, fail at detecting visually obvious odd-one-out targets, with GPT-4o achieving only 47.6 F1 on the natural image split of the Detection task.

Significance. If the central claim holds, SalBench fills a real gap in vision-centric LVLM evaluation: existing benchmarks emphasize high-level reasoning, whereas this benchmark targets low-level perceptual features. The paper's strengths are its broad model coverage, the three-task design that separates detection from grounding, the per-attribute and per-difficulty breakdowns, and the honest training experiment showing that in-distribution synthetic saliency data does not produce high scores. The benchmark itself is likely to be a useful resource for future work. However, the headline claim that the targets are 'obvious' to humans is currently an assumption rather than a measured fact, and the difference between F1 and matching accuracy is conflated in the abstract and conclusion. These issues affect the interpretation of the results, not the raw measurements themselves.

major comments (4)
  1. [Abstract; Section 7; Table 1 caption; Supplementary Table A.1] The paper's headline number is mislabeled. The abstract says GPT-4o achieves '47.6% accuracy', and the conclusion repeats '46% accuracy', but Table 1 explicitly reports F1 scores, and the matching accuracy for GPT-4o on the natural Detection split at zero-shot is 45.2% (Supplementary Table A.1). Since the abstract and conclusion make the paper's central quantitative claim using the wrong metric, the text should be corrected to state F1, and matching accuracy should be reported in the abstract if the authors intend accuracy as the headline metric.
  2. [Section 1; Section 3; Supplementary Section C] The premise that SalBench targets are 'readily apparent' and 'visually obvious' to humans is load-bearing but unmeasured. No human participants were tested on SalBench, and the supplementary explicitly states that the focus attribute is 'particularly challenging not only for models but also for humans' (Supplementary Section C). Because the evaluation uses exact-match multi-label scoring, a model is marked wrong unless it outputs exactly the annotated attribute set; if human annotators disagree on which attributes are salient (especially for focus), the observed model scores may be closer to human-level agreement than the 'can't see the obvious' framing suggests. The authors should run a human study on a representative sample, report human-human exact-match agreement and per-attribute agreement, and compare model scores against that baseline.
  3. [Section 4.1; Table 4] The natural-split F1 scores are not calibrated against chance or majority baselines. The natural split is heavily dominated by the color attribute (over 80% of images), so a model that always predicts color would obtain high color F1 and a nontrivial overall F1; the reported GPT-4o overall F1 of 47.6% needs to be interpreted against such a baseline, especially because the paper's central comparison is between model performance and human performance. Adding random-predictor and majority-predictor baselines, computed with the same exact-match and F1 protocols, would make the headline numbers much easier to assess.
  4. [Section 4.1, 'Better Performance on Synthetic Data'] There is a direct inconsistency between the prose and Table 1 in the same section. The text states that GPT-4o obtains 'an F1 score of 70.9% in the Detection task at 5-shot on synthetic split' and '47.6% F1 in the same task and shot setting' on natural images, but Table 1 lists 86.0% and 41.9% for GPT-4o at 5-shot, and 47.6% is the zero-shot natural Detection value. The narrative numbers should match the table.
minor comments (5)
  1. [Throughout] The phrase 'F1 accuracy' is used in several places (e.g., Section 4.1); F1 and accuracy are distinct metrics and should not be combined into one term.
  2. [Section 2; Section 4; Supplementary tables] There are multiple typos in model and attribute names: 'orinetation' (Section 2), 'V´ısion' (Supplementary Table A.2 caption), 'Idefic2' and 'Idefics3' (Table 1), and 'LLama3.1' (Section 5). These should be corrected.
  3. [Table A.1 and A.2] The supplementary tables are extremely dense and difficult to read; splitting them by task or using a per-attribute figure would improve usability.
  4. [Section 5] The training experiment is a useful negative result, but the text says the models 'struggle to capture the salient features' without reporting the number of training steps or final validation loss; adding these details would make the experiment more reproducible.
  5. [References] Reference [24] is cited for both P3 and O3, but the paper also cites [6] and [52] for the 7x7 grid design; please verify that the dataset source is unambiguous and that all dataset attributions are consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark with no circular derivation; model scores are direct external measurements.

full rationale

No circularity found. SalBench is an evaluation benchmark, not a derived prediction: the central numbers are direct zero-shot and few-shot measurements of LVLMs on the external P3/O3 datasets, and the paper's own training experiment is a transfer test on generated synthetic data rather than a fitted parameter renamed as a result. The benchmark construction is explicitly an augmentation of the existing P3/O3 datasets with language instructions and three task formats, so there is no equation-level reduction in which an output is defined in terms of the quantity it claims to predict. The only evaluative premise is that the odd-one-out targets are 'readily apparent' and 'obvious' to humans; this premise is assumed rather than established by a human-subject baseline, and Supplementary Section C even concedes that the focus attribute is 'particularly challenging not only for models but also for humans.' That is a benchmark-validity or difficulty-calibration concern, not circularity, because the model scores do not depend on that premise for their computation. The repeated abstract/conclusion phrasing of 47.6% as 'accuracy' while Table 1 reports F1 scores is a reporting inconsistency, not a circular step. There is no load-bearing self-citation: the P3/O3 source is an external dataset paper, and the training recipe cites standard external works. The derivation chain is self-contained against the external benchmark data, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the P3/O3 ground-truth labels, the assumption that the chosen feature categories capture human attention, and the assumption that free-form model outputs can be reliably mapped to those categories. No fitted parameters are central to the main result; the hand-chosen difficulty thresholds only affect secondary analyses.

free parameters (4)
  • Color distance thresholds = 50, 100
    Hand-chosen to split color difficulty into easy/medium/hard in Section 6.1; they affect the difficulty analysis but not the headline model comparison.
  • Size ratio thresholds = 0.3, 1.5, 3.0
    Hand-chosen to define size difficulty levels in Section 6.1.
  • Orientation angle thresholds = 30, 60, 90 degrees
    Hand-chosen to define orientation difficulty in Section 6.1.
  • Distractor count bins = 0-7, 7-15, 15-25, >25
    Hand-chosen bins in Section 6.2 for analyzing impact of distractor quantity.
assumptions (4)
  • domain assumption The P3/O3 ground-truth labels correctly identify the odd-one-out target and its distinguishing features.
    The benchmark relies on these labels without independent human verification; introduced in Section 3.
  • domain assumption The seven feature categories (orientation, color, focus, shape, size, location, pattern) are the relevant low-level features for human visual attention.
    Used in prompts and scoring; based on Feature Integration Theory but not validated for these images.
  • domain assumption Model outputs can be unambiguously parsed into the provided categories.
    The evaluation assumes exact-match or F1 classification of free-form text; described in Section 4.
  • ad hoc to paper The two datasets P3 and O3 represent the population of human odd-one-out search tasks.
    The paper selects these two datasets without justification that they cover the intended low-level saliency space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision-Language Models Can't See the Obvious." pith.science (2026). https://pith.science/paper/HKFJMD53

@misc{pith2026250704741,
  author       = {Pith},
  title        = {Pith review of: Vision-Language Models Can't See the Obvious},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKFJMD53}},
  note         = {Machine review of arXiv:2507.04741}
}
read the original abstract

We present Saliency Benchmark (SalBench), a novel benchmark designed to assess the capability of Large Vision-Language Models (LVLM) in detecting visually salient features that are readily apparent to humans, such as a large circle amidst a grid of smaller ones. This benchmark focuses on low-level features including color, intensity, and orientation, which are fundamental to human visual processing. Our SalBench consists of images that highlight rare, unusual, or unexpected elements within scenes, and naturally draw human attention. It comprises three novel tasks for evaluating the perceptual capabilities of LVLM: Odd-One-Out Detection, Referring Odd-One-Out, and Visual Referring Odd-One-Out. We perform a comprehensive evaluation of state-of-the-art LVLM using SalBench and our findings reveal a surprising limitation: LVLM struggle to identify seemingly obvious visual anomalies, with even the advanced GPT-4o achieving only 47.6\% accuracy on such a simple task. SalBench will be an important step in measuring the capabilities of LVLM that align with the subtle definition of human attention.

Figures

Figures reproduced from arXiv: 2507.04741 by the authors.

Figure 1
Figure 1. Our SalBench evaluates the perceptual capabilities of vision-language models. It comprises natural and synthetic images, which consist of a single salient target among multiple distractors. In natural images, the target can vary from the rest in orientation, color, focus, shape, size, location and pattern, while it differs only in orientation, color or size in synthetic images. Sal￾Bench evaluates the odd-one-out un… view at source ↗
Figure 2
Figure 2. Example responses from state-of-the-art vision￾language models. The model responses are in green, enclosed in boxes, while failure reasons are in red. These results highlight that the models often fail to recognize the prominent and salient features, which would naturally capture the attention of an aver￾age human observer. Best viewed zoomed-in. ages (810 color, 864 orientation, and 840 size search ar￾rays) arrange… view at source ↗
Figure 3
Figure 3. Performance comparison of trained LVLM on saliency generated data, using two LLMs (Qwen2.5-7B and LLama3.1-8B) paired with two vision encoders (Siglip and Clip) across three tasks: Detection, Referring, and Visual Referring. The F1 scores suggest that training on such data does not help improve the performance on SalBench. Model Category Level Accuracy Detection Referring Visual Referring Qwen2-VL-72B Orientation Ea… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Perception change test. GPT-4o fails to identify that the individuals talking to the old man were switched. robots with LVLM as engines not to miss salient events. In￾corporating the saliency detection capabilities could enable robots and agents to prioritize important…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 39 canonical work pages

  1. [1]

    Phi-3 technical re- port: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ah- mad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 3, 4

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 4

    arXiv 2023

  3. [3]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019. 2

    work page 2019

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 1

    work page 2022

  5. [5]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https : / / www.anthropic.com/news/claude- 3- 5- sonnet, 2024. 4, 14

    work page 2024

  6. [6]

    Turning visual search time on its head

    SP Arun. Turning visual search time on its head. Vi- sion research, 74:86–92, 2012. 3

    work page 2012

  7. [7]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. 3, 4

    arXiv 2024

  8. [8]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, 2019. 2

    work page 2019

Show all 61 references
  1. [9]

    π 0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π 0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 8

    arXiv

  2. [10]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 13154–13164, 2023. 2

    2023

  3. [11]

    Are we on the right way for evaluating large vision-language mod- els? arXiv preprint arXiv:2403.20330 , 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language mod- els? arXiv preprint arXiv:2403.20330 , 2024. 1, 2, 8

    2024 arXiv

  4. [12]

    Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer V...

    2024

  5. [13]

    Nvlm: Open frontier-class multimodal llms

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mo- hammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024. 4

    2024 arXiv

  6. [14]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 4

    2024 arXiv

  7. [15]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 4, 6

    2024 arXiv

  8. [16]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2

    2023 arXiv

  9. [17]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6904–6913, 2...

    2017

  10. [18]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jef- frey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–...

    2018

  11. [19]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14281–...

    2024

  12. [20]

    Minicpm: Un- veiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxi- ang Huang, Weilin Zhao, et al. Minicpm: Un- veiling the potential of small language models with scalable training strategies. 2024. URL https://doi. org/10.48550/arXiv, 2404. 4

    2024 doi

  13. [21]

    Gqa: A new dataset for real-world visual reasoning and com- positional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and com- positional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 2

    2019

  14. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min- joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 2

    2016

  15. [23]

    Openvla: An open-source vision-language- action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246, 2024. 8

    2024 arXiv

  16. [24]

    Do saliency models detect odd-one- out targets? new datasets and evaluations

    Iuliia Kotseruba, Calden Wloka, Amir Rasouli, and John K Tsotsos. Do saliency models detect odd-one- out targets? new datasets and evaluations. arXiv preprint arXiv:2005.06583, 2020. 2, 3, 12

    2005 arXiv

  17. [25]

    Building and better understanding vision-language models: insights and future direc- tions

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L´eo Tronchon. Building and better understanding vision-language models: insights and future direc- tions. arXiv preprint arXiv:2408.12637, 2024. 3, 4

    2024 arXiv

  18. [26]

    What matters when building vision- language models? arXiv preprint arXiv:2405.02246,

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? arXiv preprint arXiv:2405.02246,

    arXiv

  19. [27]

    Seed- bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1, 8

    2024

  20. [28]

    Evaluating ob- ject hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating ob- ject hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natu- ral Language Processing, 2023. 2

    2023

  21. [29]

    Vila: On pre- training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mo- hammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 4

    2024

  22. [30]

    Microsoft coco: Com- mon objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proce...

    2014

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024. 1, 3, 4, 6

    2024

  24. [32]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1, 2

    2023 arXiv

  25. [33]

    Learn to explain: Multi- modal reasoning via thought chains for science ques- tion answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multi- modal reasoning via thought chains for science ques- tion answering. In The 36th Conference on Neural Information Processing Sys...

    2022

  26. [34]

    Mathvista: Evaluating math reasoning in visual contexts with gpt- 4v, bard, and other large multimodal models

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt- 4v, bard, and other large multimodal models. arXiv e-prints, 2023. 1, 2, 8

    2023

  27. [35]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 ,

    arXiv

  28. [36]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021. 2

    2021

  29. [37]

    Info- graphicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthe- nis Karatzas, Ernest Valveny, and CV Jawahar. Info- graphicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022. 2

    2022

  30. [38]

    Llama 3.2: Revolutionizing edge ai and vi- sion with open, customizable modelsy

    Meta. Llama 3.2: Revolutionizing edge ai and vi- sion with open, customizable modelsy. https:// ai.meta.com/blog/llama-3-2-connect- 2024 - vision - edge - mobile - devices/ ,

    2024

  31. [39]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In IC- DAR, 2019. 2

    2019

  32. [40]

    Mind children: The future of robot and human intelligence

    Hans Moravec. Mind children: The future of robot and human intelligence. Harvard University Press, 1988. 1

    1988

  33. [41]

    Screenagent: A vision language model-driven com- puter control agent

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven com- puter control agent. arXiv preprint arXiv:2402.07945,

    arXiv

  34. [42]

    OpenAI. Gpt-4o. https://platform.openai. com/docs/models#gpt-4o, 2024. 2

    2024

  35. [43]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, ...

    2021

  36. [44]

    A- okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer, 2022. 1, 2

    2022

  37. [45]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 2

    2019

  38. [46]

    Internvl2: Better than the best — ex- panding performance boundaries of open-source mul- timodal models with the progressive scaling strat- egy

    InternVL team. Internvl2: Better than the best — ex- panding performance boundaries of open-source mul- timodal models with the progressive scaling strat- egy. https://internvl.github.io/blog/ 2024-07-02-InternVL-2.0 , 2024. 4

    2024

  39. [47]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2, 6

    2024 arXiv

  40. [48]

    Eyes wide shut? ex- ploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? ex- ploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 9568– 9578, 2024. 2

    2024

  41. [49]

    A feature- integration theory of attention

    Anne M Treisman and Garry Gelade. A feature- integration theory of attention. Cognitive psychology, 12(1):97–136, 1980. 2, 6

    1980

  42. [50]

    Measuring mul- timodal mathematical reasoning with math-vision dataset, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring mul- timodal mathematical reasoning with math-vision dataset, 2024. 2

    2024

  43. [51]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 ,

    arXiv

  44. [52]

    Five factors that guide attention in visual search

    Jeremy M Wolfe and Todd S Horowitz. Five factors that guide attention in visual search. Nature human behaviour, 1(3):0058, 2017. 3

    2017

  45. [53]

    Grok-1.5v

    xAI. Grok-1.5v. https://x.ai/blog/grok- 1.5v, 2024. 2

    2024

  46. [54]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 6

    2024 arXiv

  47. [55]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 1, 8

    2023 arXiv

  48. [56]

    Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  49. [57]

    Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  50. [58]

    Sigmoid loss for language im- age pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language im- age pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975–11986, 2023. 6

    2023

  51. [59]

    Se- mantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Se- mantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302–321, 2019. 2

    2019

  52. [60]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1 Vision-Language Models Can’t See the Obvious Supplementary Material In this su...

    2023 arXiv

  53. [600]

    The next most common range is > 25 distractors, with a similar count to the lowest range, reflecting the dataset’s coverage of highly complex scenarios

    with 0–7 distractors, indicating that simpler scenarios with fewer distractors are well-represented. The next most common range is > 25 distractors, with a similar count to the lowest range, reflecting the dataset’s coverage of highly complex scenarios. Images with 7–15 distra...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.