Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current vision-language models fail at simple fine-grained questions in densely packed scenes, with the best of 37 models scoring 19.6% on the hardest split.

desk verdict A genuinely useful dense-scene VQA benchmark whose headline 'hard split' number is manufactured by the split's own definition; the paper deserves refereeing, with a required human baseline. read the letter →

arxiv 2509.25339 v3 pith:YYU3WYYX submitted 2025-09-29 cs.CV cs.AIcs.LGeess.IV

Paul Gavrikov , Wei Lin , M. Jehanzeb Mirza , Soumya Jahagirdar , Muhammad Huzaifa , Sivan Doveh , Serena Yeung-Levy , James Glass
show 1 more author
Hilde Kuehne
This is my paper · ORCID
classification cs.CVcs.AIcs.LGeess.IV
keywords VisualOverloadquestionansweringdensescenesfine-grainedperceptioncountingaccuracyOCRinimageslogicalconsistencyvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current vision-language models have not solved basic fine-grained visual understanding, and that existing benchmarks overstate their ability because they mostly test global scene-level reasoning. To expose this gap, the authors introduce VisualOverload, a benchmark of 2,720 hand-written question-answer pairs about 150 high-resolution public-domain paintings filled with many figures, actions, and small details. Across 37 tested models, the strongest model scores only 19.6 percent on the hardest difficulty split and 69.5 percent overall. Error analysis shows systematic failures in counting, reading text, and giving logically consistent answers to opposite paired questions. The paper concludes that dense, detail-rich scenes remain a critical bottleneck for current vision encoders.

What carries the argument

The central object is the VisualOverload benchmark itself: 150 high-resolution scans of densely populated public-domain paintings, annotated with 2,720 manually curated questions in six categories—activity, attribute, counting, OCR, reasoning, and scene classification. Its load-bearing design choices are the private ground truth (answers withheld, scoring via an evaluation server), the pairing of every binary yes/no question with its logical opposite, and a three-level difficulty split calibrated by the average accuracy of 37 tested models. These machinery pieces let the authors measure not just accuracy but logical consistency and shortcut reliance, and they convert the benchmark's difficulty from an assertion into a measured quantity.

What would settle it

Check whether humans can reliably answer the hard split: give the same 430 hard questions to a few dozen human annotators without training on the images; if human accuracy is also near 20% or if annotators disagree strongly, the difficulty is in the questions themselves, not in the models' vision. Alternatively, evaluate a model with a much larger visual token budget (e.g., 40 image patches instead of 12) on the same split; if its hard-split accuracy stays below 20%, the fixed-token-budget explanation is weakened.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that state-of-the-art VLMs, though strong at global scene classification, collapse on simple, knowledge-free visual questions once the scene is visually overloaded. The authors define three difficulty levels by the models' own accuracy: questions scored correct by under 20% of models are 'hard', 20-90% are 'medium', and over 90% are 'easy'. Under that split, the best proprietary model achieves just 19.6% on the hard split and 69.5% overall, while the strongest open-weight model reaches 7.2% and 67.6%. The benchmark's paired logical-opposite questions further reveal that models frequently contradict themselves—answering 'yes' to 'Is it day?' and 'yes' to 'Is it night?'—with consistency dropping from 83.3% on scene questions to 60.6% on reasoning questions. The paper therefore claims that the vision encoder, which compresses images into a fixed token budget, is the limiting factor for fine-grained perception.

Load-bearing premise

The hardest split is defined by the models' own low accuracy (under 20%), so the headline '19.6% on hard questions' depends on the unstated premise that these particular models' difficulties reflect intrinsic properties of the scenes rather than quirks of this model cohort.

Editorial extensions

If this is right

  • If the results hold, published VQA accuracy numbers substantially overstate real-world fine-grained perception, since models that look strong on global questions fail on detail-level questions in dense scenes.
  • Counting and OCR, not just reasoning, are the weakest skills: even the best counting model reaches 41.7% and the best OCR model 62.7%, so applications relying on inventory counts or reading signs in clutter are not yet safe.
  • Logical-consistency scoring gives a cheap extra signal beyond accuracy: a model that answers opposite paired questions inconsistently is likely exploiting shortcuts, and this measure can be added to existing benchmarks without new ground-truth annotation.
  • Difficulty calibrated by model performance means the hard split is a moving target: as models improve, the same split becomes easier, so the benchmark will need periodic re-splitting or an intrinsic difficulty measure to stay meaningful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to fix difficulty by human annotation time or by intrinsic scene properties (number of objects, text length, figure count) rather than model accuracy; if those intrinsic splits reproduce the large model gap, the headline result would become independent of the particular model cohort.
  • The encoder-bottleneck hypothesis predicts that giving a model more visual tokens (e.g., more image patches) should improve fine-grained performance more than scaling the language model; this is directly measurable with the released benchmark by patching InternVL3 or similar models at higher token budgets.
  • The pairing of logical opposites could be reused as a self-supervised probe: a model that is accurate but logically inconsistent on a pair reveals that it is likely answering from language priors or spurious correlations rather than from a coherent scene representation.
  • If the hard-split questions are re-run with a future model that uses adaptive computation or search over image regions, the 19.6% ceiling may rise sharply, which would localize the current failure specifically to fixed-budget vision encoders rather than to the questions themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces VisualOverload, a VQA benchmark consisting of 2,720 manually curated question–answer pairs over 150 high-resolution public-domain paintings, designed to stress fine-grained visual understanding in dense scenes. Questions span six categories (activity recognition, attribute recognition, counting, OCR, reasoning, scene classification), with multiple-choice and freeform answer formats, held-out ground truths, and an evaluation server. The authors evaluate 37 VLMs, report task- and difficulty-wise accuracies, and analyze error modes including counting underestimation, OCR edit-distance errors, and logical inconsistency on paired opposite binary questions. The headline finding is that the best tested model, o3, reaches only 19.6% accuracy on the hardest split and 69.5% overall, which the authors interpret as evidence of a critical gap in current VLM visual understanding.

Significance. If the results hold, the benchmark would provide a valuable stress test for dense-scene perception: it uses fresh, high-resolution, publicly licensed images; manual annotation; private ground truths; a broad 37-model comparison; and informative error analyses (counting tolerance curves, normalized Levenshtein distances, logical-consistency ratios). The resolution ablation and chain-of-thought experiments are useful additions. However, the significance of the headline numbers is reduced by two issues: the difficulty split is defined by the same models it is meant to evaluate, and no human baseline is provided. These issues do not invalidate the benchmark as a resource, but they do mean that the paper's central claim about a 'critical gap' requires additional support.

major comments (3)
  1. [2.1 (Difficulty splits), 3.2, Abstract] The hard split is defined in Section 2.1 by the average accuracy of the 37 evaluated models, with thresholds [0,20] for hard. The abstract and Section 3.2 then report that o3 achieves only 19.6% on this split. Because the split is constructed from the same model scores that are subsequently reported, the hard-split number is not an independent measure of question difficulty; it is a consequence of selecting questions on which this particular model set scored below 20%. The claim that these questions are 'hard' in a vision-specific sense requires either an intrinsic definition of difficulty or a human baseline. Please report the distribution of per-question accuracies, clarify how many questions fall into each split, and show whether o3's 19.6% differs meaningfully from the split threshold, or redefine the splits without using the evaluated models.
  2. [2.1 (Quality control), 3.2] No human baseline is reported, so the paper's central assertion that low VLM accuracy reveals a 'critical gap' in visual understanding is not fully supported. The quality-control checks — blind evaluation of three models, Gemini 2.5 Pro language-bias detection, and manual verification of ground truths only for questions solved by few models — are all model-based and do not establish that a competent human can answer the hard questions from the image at the distributed resolution. Moreover, several reasoning questions, e.g., 'Does capital punishment appear to be legal in this scene?' and 'I am allergic to seafood, is all of the food on the table safe for me?', require legal, cultural, or domain knowledge beyond the stated 'basic level of everyday world knowledge', which contradicts the 'knowledge-free' design goal. I recommend adding a human-subject evaluation on a stratified random sample of questions (including freeform answers, with the same extraction heuristics) and, at a minimum, softening the claims to 'current VLMs cannot solve these questions' rather than 'these questions should be solvable by basic visual understanding.'
  3. [Table 2, Abstract] The overall accuracy (69.5% for o3) is dominated by the scene-classification category, which accounts for 1388 of 2720 questions (51%) and is explicitly described in Section 2.1 as not requiring fine-grained understanding. Presenting 'overall 69.5%' in the abstract alongside '19.6% on the hardest split' is misleading because the former largely reflects performance on the easiest, most global category. Please report a macro-average over the six task categories (or over the five non-scene categories) as the headline overall accuracy, and make the category composition clear in the abstract.
minor comments (5)
  1. [2.2 (Answer extraction)] The OCR normalization step replaces 'V' with 'U' and 'J' with 'I', which is a lenient equivalence that directly affects the reported OCR accuracy; please justify this choice with examples and report OCR accuracy without this normalization as a sensitivity check.
  2. [4 (Counting)] Treating refusals and blank responses as 0 in Figure 2a conflates non-response with underestimation; please also present the count distribution with refusals removed or as a separate category.
  3. [Table 2] The numerical entries for several rows are garbled in the submitted version (e.g., the InternVL3-78B row reads '78.078.0 80.534.7'); please ensure the final table is correctly typeset.
  4. [2.3] The claim that the human-centered annotation process ensures 'unbiased evaluation' is too strong given that the quality-control pipeline itself relies on model outputs; please soften this wording.
  5. [Abstract] The abstract describes the benchmark as 'slightly different', which is unnecessarily informal for a research paper; consider removing the qualifier.

Circularity Check

1 steps flagged · score 6.0 of 10

The hard-split headline is partly definitional: the 'hardest' bucket is defined by the evaluated models' own low scores, so the 19.6% result is not independent evidence of a vision-specific gap.

  1. self definitional [Sec. 2.1 (Difficulty splits); headline in Abstract and Sec. 3.2]
    "We divide our questions into three difficulty levels—easy, medium, and hard—based on model performance in Sec. 3. The thresholds are defined by the percentage of correct responses: [0,20] for hard, (20,90) for medium, and [90,100] for easy. ... Indeed, we observe that even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on all questions."

    The 'hardest' split is not defined by any intrinsic property of the questions, such as human error rates, resolution needs, or ambiguity, but by the measured performance of the same 37 models whose results are then reported. A question is hard exactly when the model panel scores in [0,20]% correct responses, so the statement that VLMs perform poorly on the hard split is enforced by the split's construction rather than independently demonstrated. The headline 19.6% is the best model's accuracy on a subset deliberately selected to be near the floor of the panel's performance. No human baseline or independent answerability measure is provided; the only answerability checks use models (37-VLM screening, image ablations, and Gemini 2.5 Pro bias detection in Sec. 2.1).

full rationale

The central derivation chain is otherwise self-contained: VisualOverload introduces a newly collected set of 150 public-domain images and 2,720 manually annotated question-answer pairs with private ground truths, evaluates 37 VLMs, and reports per-task accuracies, resolution ablations, and error analyses that do not reduce to any fitted input. The main circular element is the difficulty split: it is calibrated from the same 37-model evaluation that produces the headline '19.6% on the hardest split.' Because 'hard' is defined as questions on which the model panel scores 0-20% correct, the claim that VLMs perform badly on the hard split is, at the panel level, guaranteed by the definition of the split. No human baseline or independent intrinsic-difficulty measure is supplied, so the 19.6% result is not by itself evidence of a vision-specific limitation. The overall 69.5% accuracy and the per-task numbers are empirical and not circular, and no load-bearing self-citation chain is present. The circularity is therefore real but partial, warranting a score of 6 rather than higher.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central numbers rest on unverified assumptions about ground-truth correctness, question knowledge-freeness, transfer from paintings to real scenes, and scoring normalization. The difficulty-split thresholds are fitted to the evaluated models themselves, which is the main circular element.

free parameters (2)
  • Difficulty split thresholds = hard: [0,20], medium: (20,90), easy: [90,100] average model accuracy
    Chosen after evaluating the 37 models; the hard split is literally the set of questions where models scored below 20%, so the headline hard-split accuracy is partly definitional.
  • OCR answer normalization = remove diacritics, punctuation, and spacing; lowercase; replace V with U and J with I
    Hand-chosen normalization rules applied before scoring OCR answers; this affects measured OCR accuracy and encodes an assumption about Latin text conventions.
assumptions (4)
  • domain assumption The human-annotated ground-truth answers are correct and unambiguous.
    All accuracy numbers rest on annotation quality; only questions that few models solved were manually re-verified (Sec. 2.1, Quality control), and no inter-annotator agreement is reported.
  • domain assumption Every question is answerable from the image with only everyday world knowledge.
    Section 2.1 restricts questions to be knowledge-free, but several examples (e.g., 'Does capital punishment appear to be legal in this scene?', 'I am allergic to seafood...') require nontrivial external knowledge, so score differences may partly reflect knowledge rather than pure vision.
  • domain assumption Performance on dense public-domain paintings transfers to real-world dense scenes.
    The introduction motivates the benchmark with real-world applications, but the images are artworks; artistic style, composition, and historical context may affect VLM performance independently of visual density.
  • domain assumption The answer-extraction heuristics are fair across models.
    Free-form counting and OCR responses are post-processed with hand-written rules (Sec. 2.2); if the heuristics favor certain output phrasings, the reported accuracies may not reflect true visual ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes." pith.science (2026). https://pith.science/paper/YYU3WYYX

@misc{pith2026250925339,
  author       = {Pith},
  title        = {Pith review of: VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYU3WYYX}},
  note         = {Machine review of arXiv:2509.25339}
}
read the original abstract

Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answering (VQA) benchmark comprising 2,720 question-answer pairs, with privately held ground-truth responses. Unlike prior VQA datasets that typically focus on near global image understanding, VisualOverload challenges models to perform simple, knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our dataset consists of high-resolution scans of public-domain paintings that are populated with multiple figures, actions, and unfolding subplots set against elaborately detailed backdrops. We manually annotated these images with questions across six task categories to probe for a thorough understanding of the scene. We hypothesize that current benchmarks overestimate the performance of VLMs, and encoding and reasoning over details is still a challenging task for them, especially if they are confronted with densely populated scenes. Indeed, we observe that even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on all questions. Beyond a thorough evaluation, we complement our benchmark with an error analysis that reveals multiple failure modes, including a lack of counting skills, failure in OCR, and striking logical inconsistencies under complex tasks. Altogether, VisualOverload exposes a critical gap in current vision models and offers a crucial resource for the community to develop better models. Benchmark: http://paulgavrikov.github.io/visualoverload

Figures

Figures reproduced from arXiv: 2509.25339 by the authors.

Figure 1
Figure 1. Example questions from VisualOverload. Our benchmark consists of images displaying densely populated scenes paired with handcrafted questions (multiple-choice and free-form) covering six core vision tasks. All yes/no questions are paired with questions asking for a logical opposite question to decrease the random chance and to provide an additional signal for measuring logical consistency. 1 arXiv:2509.25339v2 [cs.C… view at source ↗
Figure 2
Figure 2. Insights into counting errors. All analyses display distributions over all model predictions exclusively for the counting task. To quantify the magnitude of these errors, we measured accuracy under varying tolerance levels, shown in Fig. 2b. Prediction errors are typically severe: even with a 10% tolerance, average accuracy improves by only 1.6%. Larger tolerances, such as 50% or 100%, yield more substantial improve… view at source ↗
Figure 3
Figure 3. OCR prediction er￾ror distance. OCR. Similar to counting, we aim to quantify the magnitude of errors in OCR predictions. To do this, we measure the Levenshtein edit distance (Levenshtein, 1965) between preprocessed predictions (as described in Sec. 2.2) and ground truths for incorrect answers. We normalize the distance by the maximum sequence length and visualize the distribution in [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Logical consistency. Logical Consistency. As described in Sec. 2.1, our dataset contains binary questions, where each such question is paired with a logically opposite. A strong model should argue logi￾cally consistently, even if the answer is wrong. For instance, if a…
Figure 5
Figure 5. Figure 5: Resolution ablation. We distribute VisualOverload at a resolution that matches the pixels of 4K (with a few outliers). Additionally, we downsampled images to match the number of pixels of VGA (640×480 pixels), HD (1280×720 pixels), FHD (1920×1080 pixels), QHD (2560×144…
Figure 6
Figure 6. Figure 6: Question Embeddings. A.6 DATASHEET In the following, we provide a datasheet (Gebru et al., 2021). We have anonymized some entries for the review process and will update these upon release. Motivation For what purpose was the dataset created? Was there a specific task i…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    ReKey introduces a live benchmark protocol that regenerates visual keys in images to produce contamination-resilient VQA evaluations, showing 9.5-18.8 point higher scores on original items across eight VLMs.

Reference graph

Works this paper leans on

62 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Analyzing the behavior of visual question answering models, 2016

    Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models, 2016. URL https://arxiv.org/abs/1606.07356

    arXiv 2016

  2. [2]

    Don't just assume; look and answer: Overcoming priors for visual question answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    work page 2018

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA : V isual Q uestion A nswering. In International Conference on Computer Vision (ICCV), 2015

    work page 2015

  4. [4]

    Touchstone: Evaluating vision-language models by language models

    Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890, 2023

    arXiv 2023

  5. [5]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Visit-bench: A benchmark for vision-language instruction following inspired by real-world use

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595, 2023

    arXiv 2023

  7. [7]

    An introduction to vision-language modeling

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Ma \ n as, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247, 2024

    arXiv 2024

  8. [8]

    Rubi: Reducing unimodal biases for visual question answering

    Remi Cadene, Corentin Dancette, Hedi Ben younes, Matthieu Cord, and Devi Parikh. Rubi: Reducing unimodal biases for visual question answering. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings....

    work page 2019

Show all 62 references
  1. [9]

    Frankland, Thomas L

    Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicol\` o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M. Frankland, Thomas L. Griffiths, Jonathan D. Cohen, and Taylor Webb. Understanding the limits of vision language models through the lens of the binding problem...

    2024

  2. [10]

    Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 0 27056--27087, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37: 0 27056--27087, 2024

    2024

  3. [11]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. Advances in Neural Informati...

    2024

  4. [12]

    Dense and aligned captions (dac) promote compositional reasoning in vl models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processin...

    2023

  5. [13]

    Teaching structured vision & language concepts to vision & language models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Comput...

    2023

  6. [14]

    Datasheets for datasets

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum\' e III, and Kate Crawford. Datasheets for datasets. Commun. ACM, 64 0 (12): 0 86–92, November 2021. ISSN 0001-0782. doi:10.1145/3458723. URL https://doi.org/10.1145/3458723

    2021 doi

  7. [15]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team . Gemini: A family of highly capable multimodal models, 2024

    2024

  8. [16]

    Gemini 2.0 Flash Model Card , April 2025

    Gemini Team . Gemini 2.0 Flash Model Card , April 2025. URL https://storage.googleapis.com/model-cards/documents/gemini-2-flash.pdf. [Online; accessed 28. Aug. 2025]

    2025

  9. [17]

    Gemma 3 technical report, 2025

    Gemma Team . Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  10. [18]

    Announcing Gemma 3n preview: powerful, efficient, mobile-first AI , May 2025

    Lucas Gonzalez and Rakesh Shivanna. Announcing Gemma 3n preview: powerful, efficient, mobile-first AI , May 2025. URL https://developers.googleblog.com/en/introducing-gemma-3n. [Online; accessed 28. Aug. 2025]

    2025

  11. [19]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  12. [20]

    Horizon Alpha - Advanced AI Language Model , August 2025

    Horizon Alpha Team . Horizon Alpha - Advanced AI Language Model , August 2025. URL https://horizonalpha.ai. [Online; accessed 28. Aug. 2025]

    2025

  13. [21]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36: 0 31096--31116, 2023

    2023

  14. [22]

    Conme: Rethinking evaluation of compositional reasoning for modern vlms

    Irene Huang, Wei Lin, Muhammad Jehanzeb Mirza, Jacob Hansen, Sivan Doveh, Victor Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuehne, Trevor Darrell, et al. Conme: Rethinking evaluation of compositional reasoning for modern vlms. Advances in Neural Information Processing Systems, ...

    2024

  15. [23]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 2...

    2022

  16. [24]

    Dvoichnye kody s ispravleniem vypadenii, vstavok i zameshchenii simvolov

    Vladimir Iosifovich Levenshtein. Dvoichnye kody s ispravleniem vypadenii, vstavok i zameshchenii simvolov. Doklady Akademii Nauk SSSR, 163 0 (4): 0 845--848, 1965

    1965

  17. [25]

    Otterhd: A high-resolution multi-modality model

    Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint arXiv:2311.04219, 2023 a

    2023 arXiv

  18. [26]

    LLaVA-OneVision: Easy Visual Task Transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy Visual Task Transfer . arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  19. [27]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 b

    2023 arXiv

  20. [28]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 c

    2023

  21. [29]

    Omnibench: Towards the future of universal omni-language models

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272, 2024 b

    2024

  22. [30]

    Introducing LFM2: The Fastest On-Device Foundation Models on the Market Liquid AI , August 2025

    Liquid AI . Introducing LFM2: The Fastest On-Device Foundation Models on the Market Liquid AI , August 2025. URL https://www.liquid.ai/blog/liquid-foundation-models-v2-our-second-series-of-generative-ai-models. [Online; accessed 28. Aug. 2025]

    2025

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023 a

    2023 arXiv

  24. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023 b

    2023

  25. [33]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  26. [34]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233. Springer, 2024 b

    2024

  27. [35]

    Smolvlm: Redefining small and efficient multimodal models, 2025

    Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvlm: Redefi...

    2025 arXiv

  28. [36]

    Umap: Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3 0 (29): 0 861, 2018. doi:10.21105/joss.00861. URL https://doi.org/10.21105/joss.00861

    2018 doi

  29. [37]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , August 2025

    Meta AI . The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation , August 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence. [Online; accessed 28. Aug. 2025]

    2025

  30. [38]

    Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections

    Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. Advances in Neural Information Processing Systems, 36: 0 5...

    2023

  31. [39]

    Muhammad Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, Yuki Mitsufuji, Horst Possegger, Rogerio Feris, Leonid Karlinsky, and James R. Glass. GLOV : Guided large language models as imp...

    2025

  32. [40]

    Gpt-4 technical report, 2024

    OpenAI . Gpt-4 technical report, 2024

    2024

  33. [41]

    OpenAI o3 and o4-mini System Card , August 2025

    OpenAI . OpenAI o3 and o4-mini System Card , August 2025. URL https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf. [Online; accessed 28. Aug. 2025]

    2025

  34. [42]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 3170--3180, October 2023

    2023

  35. [43]

    Humanity's last exam, 2025

    Long Phan et al. Humanity's last exam, 2025. URL https://arxiv.org/abs/2501.14249

    2025 arXiv

  36. [44]

    Scaling vision pre-training to 4k resolution, 2025

    Baifeng Shi, Boyi Li, Han Cai, Yao Lu, Sifei Liu, Marco Pavone, Jan Kautz, Song Han, Trevor Darrell, Pavlo Molchanov, and Hongxu Yin. Scaling vision pre-training to 4k resolution, 2025. URL https://arxiv.org/abs/2503.19903

    2025 arXiv

  37. [45]

    Paligemma 2: A family of versatile vlms for transfer, 2024

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lu...

    2024 arXiv

  38. [46]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  39. [47]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in N...

    2022

  40. [48]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023

    2023 arXiv

  41. [49]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13084--13094, 2024

    2024

  42. [50]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 arXiv

  43. [51]

    Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765, 2024

    2024 arXiv

  44. [52]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  45. [53]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output, 2024 a

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  46. [54]

    Yin and yang: Balancing and answering binary visual questions

    Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  47. [55]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  48. [56]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506.05176

    2025 arXiv

  49. [57]

    Why are visually-grounded language models bad at image classification? arXiv preprint arXiv:2405.18415, 2024 b

    Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually-grounded language models bad at image classification? arXiv preprint arXiv:2405.18415, 2024 b

    2024 arXiv

  50. [58]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  51. [59]

    InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models , 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025 arXiv

  52. [60]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [61]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [62]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.