Pith. sign in

REVIEW 4 major objections 5 minor 93 references

Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal LLMs fail a new safety-detail benchmark, best score 0.46

desk verdict Useful safety-oriented visual commonsense benchmark; the headline 0.46 is conditional on an unvalidated GPT-4o judge. read the letter →

arxiv 2506.14805 v2 pith:RTTHMIUY submitted 2025-06-03 cs.CV cs.AIcs.CLcs.LGcs.MM

Yang Yao , Lingyu Li , Jiaxin Song , Chiyu Chen , Zhenqi He , Yixu Wang , Xin Wang , Tianle Gu
show 3 more authors
Jie Li Yan Teng Yingchun Wang
This is my paper
classification cs.CVcs.AIcs.CLcs.LGcs.MM
keywords multimodallargelanguagemodelsfine-grainedvisualperceptioncommonsensereasoningcausalinferencebenchmarkevaluationsigmoidscoringfunctionembodiedintelligencesafety-criticalvision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether today's multimodal large language models can do two things at once: see a small, easily missed detail in an image and use that detail to make a commonsense safety decision. To force the test, the authors built Argus Inspection, a benchmark of 1,430 image-plus-text scenarios across ten commonsense domains, with the decisive 'trap' element present only in the image and never named in the text. A harder 'deceptive' version adds morally or emotionally loaded wording meant to pull the model's attention away from the visual clue. The authors evaluate 26 current models with a scoring rule that separately rewards noticing the trap and choosing the correct action; the strongest model reaches only 0.46 on the basic level. The paper's conclusion is that fine-grained visual perception and commonsense causal inference remain unsolved, which matters if models are to act in physical, human-centered settings.

What carries the argument

The load-bearing object is the 'trap': a commonsense-relevant visual element deliberately inserted into an otherwise benign image while being absolutely isolated from the accompanying text, so the image is the only source of the critical information. Around this, the Eye of Panoptes evaluation applies a binary parametric Sigmoid, $f(x,y)=1/(1+e^{-p(x+qy-r)})$ with $(p,q,r)=(3,0.8,6)$, to the Scorer-assigned trap-detection score $x$ and action-alignment score $y$ (each in $\{1,2,3,4\}$), then multiplies by an indicator $d$-score set to 1 only when the model names the trap in a separate image-description pass. The function maps intermediate cases—mentioning the trap without acting on it, or acting correctly without noticing the trap—to nonzero rewards instead of the usual binary 0/1; this metric is what turns the raw responses into the 0.46 ceiling and model ordering.

What would settle it

Take a random subset of Argus items, have several human raters apply the Appendix B scoring rules to the same model responses, and compare their scores to the GPT-4o Scorer's; if agreement is low, or if human-based rankings differ substantially, the reported 0.46 ceiling and model ordering are not a valid measure of multimodal capability. A simpler check is to verify the traps themselves: an image-only probe should show human raters can identify the trap from the image alone at high agreement, otherwise the 'isolation from text' premise fails.

Watch

Extended reading notes

Core claim

The central discovery, as the authors state it, is that current MLLMs lack the fine-grained capture and commonsense causal reasoning required for applications such as embodied intelligence. Across 26 mainstream open-source and proprietary models, the highest overall score on Argus-basic is 0.46, achieved by GPT-4.1, with the next two Gemini models above 0.40; scores fall to as low as 0.05 for smaller open models. In the deceptive version, most models drop further, and paired t-tests across the ten domains show the deceptive level is significantly harder overall, although the Gemini 2.5 family appears resistant to the added textual distraction. The authors take this as evidence that the ability to notice a visually embedded trap and turn it into a correct, safe action is not yet present in state-of-the-art systems.

Load-bearing premise

The measurement assumes the GPT-4o-based Scorer's judgments of whether a model noticed the trap and matches the standard answer are accurate enough to stand in for human judgment, and no human-agreement baseline is reported.

Editorial extensions

If this is right

  • If the benchmark's claim holds, state-of-the-art multimodal models cannot yet be trusted to read safety-critical details in images before acting.
  • A model can score well only by combining two abilities at once—finding the trap and drawing the right causal conclusion—so the low ceiling points to both perception and reasoning as bottlenecks.
  • The deceptive version being significantly harder across domains implies that emotional or moral wording can push models off the correct visual inference, a risk for human-facing assistants.
  • Closed-source models dominate the upper ranks, but even the best leave most scenarios unsolved, so current capabilities are far from the 'all-seeing' level the benchmark name invokes.
  • The metric's graded rewards for partial success make it usable as a training objective for post-tuning fine-grained perception and causal reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the same model family generated, verified, and scored the data, a human-rater replication on a random subset is the natural check; without it, part of the 0.46 gap could be scorer leniency or bias rather than model failure.
  • Inference: the rare cases where deceptive scores exceed basic scores suggest text length or framing effects deserve their own controlled study; the paper proposes this direction but does not establish it.
  • Inference: the trap-isolation recipe could be adapted to video or interactive agent evaluations, where the 'trap' is an event or object that appears only at a certain moment; such extensions are not in the paper.
  • Inference: if the benchmark is right, fine-grained perception is not merely an engineering detail but a safety property; benchmarks like this could serve as red-team tests for embodied agents before deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Argus Inspection, a multimodal benchmark of 1,430 entries across ten commonsense domains, with basic and deceptive levels, and the Eye of Panoptes evaluation framework, which combines a parametric Sigmoid over x- and y-scores with an indicator function d-score. The authors evaluate 26 open- and closed-source MLLMs and report that the best model (GPT-4.1-2025-04-14) reaches only 0.46 on the basic level, concluding that current MLLMs lack fine-grained visual perception and commonsense causal reasoning. The paper also compares basic versus deceptive difficulty using paired t-tests.

Significance. If the benchmark construction and scoring are valid, the contribution is genuinely useful: it targets a realistic gap (small visual details that determine safe versus dangerous commonsense actions), provides a graded metric that distinguishes trap detection from action alignment, and ships public data and evaluation code. The design of two difficulty levels and the inclusion of 26 models are strengths, as are the explicit temperature settings and per-domain statistical tests. The main significance is conditional, however, because the headline 0.46 ceiling and the model rankings all pass through a single unvalidated LLM judge that belongs to the same model family being evaluated. The paper's conclusion would be strengthened materially by human validation of the judge and of a sample of the generated data.

major comments (4)
  1. [Section 4.1, Eq. (2), Appendix B] The central claim, including the 0.46 ceiling in Table 2, depends on GPT-4o-2024-11-20 as the Scorer for the d-, x-, and y-scores, while GPT-4o-family models are themselves among the evaluated systems. No inter-annotator agreement, human-baseline comparison, scorer error analysis, or repeated-run variance is reported. The d-score in Eq. (2) is particularly load-bearing: the indicator I multiplies the entire score, so a false-negative trap mention costs a model its whole score, while the acceptance criteria in Appendix B.1 ('appropriately vague' and 'similar visual image') can grant d=1 without the model having identified the causal trap. The paper's only justification, the citation to Liu et al. [46] in Section 4.1, is to a jailbreak paper and does not address grading reliability. The authors should provide a human-annotated subset, quantify scorer agreement, and report error rates for all three scoring dimensions before the headline result can be accepted.
  2. [Section 4.1, Eq. (1), Figure 4] The metric parameters p=3, q=0.8, r=6 are chosen through 'preliminary experiments,' and the paper explicitly states that suitable values are not unique and that no ablation studies are reported. Since Eq. (1) is a nonlinear sigmoid, the absolute value 0.46 and the relative gaps between models could be sensitive to these choices. The paper should report a sensitivity analysis over a plausible parameter range and show that the main conclusions (low overall ceiling, closed-source above open-source) are invariant, or provide a principled argument for parameter invariance.
  3. [Section 3.3] The data construction pipeline uses GPT-4o-2024-11-20 to generate topics, traps, scenarios, questions, and standard answers, and GPT-4o also performs the final multimodal logical verification; images are generated by gpt-image-1. The paper mentions 'human expert filtering' but reports no inter-annotator agreement, no rejection rates, and no human-verification statistics for whether the generated images actually display the trap in a human-visible way. This bears directly on the benchmark's representativeness and on the 'absolute isolation' claim, and should be supported with a human validation sub-study.
  4. [Section 5, Appendix C, Tables 6 and 7] The text states that 'the global paired t-test results indicate that across ten domains, the difficulty of deceptive-level is significantly higher than that of basic-level.' This is not supported for every model: Table 7 reports negative mean gaps for Gemini-2.5-Flash-Preview-05-20 (-0.0216), Gemini-2.5-Pro-Preview-05-06 (-0.0091), and Seed-1.5-VL-250428 (-0.0092), with p-values of 0.0067, 0.423, and 0.322, respectively. The finding should be restated as a model-average effect with explicitly identified exceptions; the current wording overgeneralizes, and the 'True' significance label for a negative gap in Table 7 is misleading.
minor comments (5)
  1. [Appendix C] Typographical errors in captions and table headings, including 'Learderboard' and 'Arugs', and numerous misspellings in Table 3 ('harzards', 'manruver', 'choing', 'misues', 'tempaterature', 'ligntning'), should be corrected.
  2. [Section 4.1, Reference [46]] The citation to Liu et al. [46] is a jailbreak study and does not support the claim that 'GPT's scoring is closest to human judgment'; either replace it with a relevant reliability study or remove the claim.
  3. [Section 5] The paper states that temperature was set to 0 for evaluated models, but it does not report the decoding settings (temperature, top-p, or number of runs) for the LLM-based Scorer itself; this should be specified for reproducibility, particularly because the Scorer's outputs are treated as ground truth.
  4. [References] Reference [43] and Reference [44] are the same paper; the bibliography should be deduplicated.
  5. [Section 5] The explanation that longer deceptive text may guide models to focus on the image is presented as a hypothesis; it should be labeled as speculative or supported with the traced logs.

Circularity Check

0 steps flagged · score 0.0 of 10

No formal circularity: the sigmoid metric is an explicitly hand-tuned scoring function and the leaderboard is an empirical measurement; the GPT-4o scorer issue is a validity risk, not a derivation from its own inputs.

full rationale

The paper's central claim ('current MLLMs lack multimodal fine-grained capture and causal reasoning capabilities') is an empirical inference from Table 2, not a consequence of a definition or a self-citation. The standard answers are generated with GPT-4o-2024-11-20 but are 'subjected to logical filtering by human experts' and pass 'a final round of human expert filtering' (Section 3.3), so the ground truth is not GPT-4o's output by construction. The Eye of Panoptes metric f(x,y) is an openly chosen logistic function: the authors state 'Through preliminary experiments to adjust hyperparameters, we chose the combination (p=3, q=0.8, r=6)' and explicitly say 'suitable values are not unique.' Thus the 0.46 ceiling is a measurement made with that metric, not a prediction derived from the metric itself. The d-score gate in Eq. (2) is an operational definition of 'visually recognized the trap' applied to actual model descriptions; it could be wrong (scorer false negatives), but a wrong measurement is not circularity. The use of GPT-4o-2024-11-20 as the LLM Scorer (Section 4.1) is a reliability/external-validity concern because no inter-annotator agreement or human baseline is reported, but replacing the scorer with a validated judge would leave the framework's logic unchanged, so the concern does not make the result true by definition. The only overlapping-author citation, [14] (MLLMGuard), appears in Related Work and is not load-bearing. No equation-level reduction, fitted-value-renamed-as-prediction, or author-imported uniqueness claim exists.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The reported deficiency score rests on three hand-tuned metric parameters and on unverified assumptions about GPT-4o as generator, verifier, and judge. The benchmark's validity as a measure of MLLM capability depends on these assumptions; no independent human reliability study is provided.

free parameters (3)
  • Sigmoid slope p = 3
    Eq. (1); chosen by preliminary experiments to control steepness of the score transition. No ablation or sensitivity analysis is reported.
  • Sigmoid weight q = 0.8
    Eq. (1); set below 1 to amplify the x-score (trap detection) relative to the y-score (action alignment). Tuned by hand.
  • Sigmoid offset r = 6
    Eq. (1); selected so that high (x,y) scores map near 0.97 and low scores near 3.37e-06. Chosen through preliminary experiments.
assumptions (4)
  • domain assumption LLM-based scoring by GPT-4o approximates human judgment closely enough for benchmark scoring.
    Section 4.1 invokes Liu et al. [46] for 'GPT's scoring is closest to human judgment', but no human agreement study, scorer reliability, or human baseline is provided in this paper.
  • domain assumption Trap elements are visually present in the gpt-image-1 generated images and are absent from the textual scenario and question.
    Section 3.3 states that trap is isolated from scenario and question, but verification relies on GPT-4o plus human filtering; no quantitative leakage check or image-annotation audit is reported.
  • domain assumption The GPT-4o-generated and human-filtered standard answers are correct ground truth for the commonsense safety decisions.
    The benchmark answers are generated by GPT-4o and filtered by human experts, but the filtering process is not documented with agreement statistics or error rates.
  • ad hoc to paper The ten commonsense domains and 200 GPT-4o-generated topics adequately represent the target space of real-world commonsense scenarios.
    The taxonomy in Section 3.1 and Appendix A is a design choice; no argument is given that the selection is representative or unbiased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?." pith.science (2026). https://pith.science/paper/RTTHMIUY

@misc{pith2026250614805,
  author       = {Pith},
  title        = {Pith review of: Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RTTHMIUY}},
  note         = {Machine review of arXiv:2506.14805}
}
read the original abstract

As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark with two levels of difficulty, emphasizing detailed visual recognition while incorporating real-world commonsense understanding to evaluate causal reasoning abilities. Expanding on it, we present the Eye of Panoptes framework, which integrates a binary parametric Sigmoid metric with an indicator function, enabling a more holistic evaluation of MLLMs' responses in opinion-based reasoning tasks. Experiments conducted on 26 mainstream MLLMs reveal that the highest performance in visual fine-grained reasoning reaches only 0.46, highlighting considerable potential for enhancement. Our research offers valuable perspectives for the continued refinement of MLLMs.

Figures

Figures reproduced from arXiv: 2506.14805 by the authors.

Figure 1
Figure 1. Illustration of the Argus. This figure illustrates deceptive-level cases, with portions of text appropriately omitted. including inquiries about an object’s name, color, category, physical attributes, and spatial location; and (3) relational inference, such as questions regarding temporal relationships, spatial positioning, and comparative attributes of objects in an image [32, 35]. Despite their effectiveness in ev… view at source ↗
Figure 2
Figure 2. Pattern of the Argus Inspection. To simulate more complex factual scenarios and contextual de￾pendencies, our benchmark is divided into two versions: the basic and the deceptive levels. In the deceptive level, textual information is infused with moral or emotional interference, guiding MLLMs to overlook trap visual elements, thereby “losing” their all-seeing capability and generating incorrect and dangerous response… view at source ↗
Figure 3
Figure 3. Pipeline of Argus Inspection and framework of Eye of Panoptes. the range {1,2,3,4}. The scoring rules are detailed in Appendix B. Intuitively, if the model identifies the trap elements and responds correctly based on them, it receives the highest score (4,4); if the model fails to notice the trap elements and provides an adverse re￾sponse, it receives the lowest score (1,1). Considering the varying rewards for diffe… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: 3D function plot and 2D contour map. Considering that some MLLMs may have deficiencies in visual recognition capabilities, we introduce an indicator function I, which takes values of 0 or 1. First, we ask MLLMs to provide a detailed description of the image, and the Sc…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 20 canonical work pages

  1. [46]

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2023. Query-relevant images jailbreak large multi-modal models. arXiv preprint arXiv:2311.17600 (2023)

    arXiv 2023

  2. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query trans- former models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  3. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  4. [3]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 39–48

    2016

  5. [4]

    Anthropic. 2025. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3- 7-sonnet

    2025

  6. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  7. [6]

    Lei Chen, Feng Yan, Yujie Zhong, Shaoxiang Chen, Zequn Jie, and Lin Ma. 2024. Mindbench: A comprehensive benchmark for mind map structure recognition and analysis. arXiv preprint arXiv:2407.02842 (2024)

    work page Pith review arXiv 2024

  8. [7]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 24185–24198

    2024

Show all 93 references
  1. [8]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV]

    2023 arXiv

  2. [9]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen

  3. [10]

    Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, and Leonid Sigal

  4. [11]

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. 2024. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.03230 (2024)

    2024 arXiv

  5. [12]

    Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li. 2024. Interleaved-modal chain- of-thought. arXiv preprint arXiv:2411.19488 (2024)

    2024 arXiv

  6. [13]

    arXiv preprint arXiv:2407.14506 (2024)

    On pre-training of multimodal language models customized for chart understanding. arXiv preprint arXiv:2407.14506 (2024)

    2024 arXiv

  7. [14]

    Tianle Gu, Zeyang Zhou, Kexin Huang, Liang Dandan, Yixu Wang, Haiquan Zhao, Yuanqi Yao, Yujiu Yang, Yan Teng, Yu Qiao, et al . 2024. Mllmguard: A multi-dimensional safety evaluation suite for multimodal large language models. Advances in Neural Information Processing Systems 3...

    2024

  8. [15]

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. [n. d.]. MMWorld: Towards Multi- discipline Multi-faceted World Model Evaluation in Videos. In The Thirteenth International Conference on Learning ...

  9. [16]

    Google. 2025. Gemini 2.5 pro model card. https://storage.googleapis.com/model- cards/documents/gemini-2.5-pro-preview.pdf

    2025

  10. [17]

    Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. 2021. AI2D-RST: a multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation 55 (2021...

    2021

  11. [18]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  12. [19]

    Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. 2024. Cmmu: A benchmark for chinese multi-modal multi- type question understanding and reasoning. arXiv preprint arXiv:2401.14011 (2024)

    2024 arXiv

  13. [20]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916

    2021

  14. [21]

    Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. 2024. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405.10739 (2024)

    2024

  15. [22]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720 (2024)

    2024 arXiv

  16. [23]

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5648–5656

    2018

  17. [24]

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s" up" with vision- language models? Investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785 (2023)

    2023 arXiv

  18. [25]

    Kushal Kafle and Christopher Kanan. 2017. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding 163 (2017), 3–20

    2017

  19. [26]

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. 2024. Tablevqa-bench: A vi- sual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205 (2024)

    2024 arXiv

  20. [27]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  21. [28]

    Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acik- goz, Letitia Parcalabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, et al. 2023. Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models. arXiv prepri...

    2023 arXiv

  22. [29]

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. 2025. Imagine while Reasoning in Space: Multimodal Visualization-of-Thought. arXiv preprint arXiv:2501.07542 (2025)

    2025 arXiv

  23. [30]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  24. [31]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al

  25. [32]

    Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen. 2024. A Survey on Multimodal Benchmarks: In the Era of Large AI Models. arXiv preprint arXiv:2409.18142 (2024)

    2024 arXiv

  26. [33]

    Ming Li, Keyu Chen, Ziqian Bi, Ming Liu, Benji Peng, Qian Niu, Junyu Liu, Jinlang Wang, Sen Zhang, Xuanhe Pan, et al. 2024. Surveying the mllm landscape: A meta-review of current surveys. arXiv preprint arXiv:2409.18991 (2024)

    2024

  27. [34]

    Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349 (2023)

    2023 arXiv

  28. [35]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

    2022

  29. [36]

    Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, Tianwen Qian, Bin Zhu, Na Zhao, and Yu-Gang Jiang. 2024. Look before you decide: Prompting active deduction of mllms for assumptive reasoning. arXiv preprint arXiv:2404.12966 (2024)

    2024 arXiv

  30. [37]

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al . 2024. Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding. arXiv preprint arXiv:2407.04903 (2024)

    2024 arXiv

  31. [38]

    Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. 2024. Scemqa: A scientific college entrance level multimodal question answering benchmark. arXiv preprint arXiv:2402.05138 (2024)

    2024 arXiv

  32. [39]

    Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, a...

    2025 arXiv

  33. [40]

    Yujie Lin, Ante Wang, Moye Chen, Jingyao Liu, Hao Liu, Jinsong Su, and Xinyan Xiao. 2025. Investigating inference-time scaling for chain of multi-modal thought: A preliminary study. arXiv preprint arXiv:2502.11514 (2025)

    2025 arXiv

  34. [41]

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. 2023. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning.arXiv preprint arXiv:2311.10774 (2023)

    2023 arXiv

  35. [42]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

    2024

  36. [43]

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 (2024)

    2024 arXiv

  37. [44]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  38. [45]

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955 (2024)

    2024 arXiv

  39. [47]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. 7

    2023

  40. [48]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision . 2200–2209

    2021

  41. [49]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty

  42. [50]

    Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multimodal question answering on scientific papers. arXiv preprint arXiv:2407.09413 (2024)

    2024 arXiv

  43. [51]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 (2023)

    2023 arXiv

  44. [52]

    Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, and Samuel Albanie

  45. [53]

    Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. 2023. Visual chain of thought: bridging logical gaps with multimodal infillings. arXiv preprint arXiv:2305.02317 (2023)

    2023 arXiv

  46. [54]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural i...

    2022

  47. [55]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki

  48. [56]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  49. [57]

    Wenqi Shao, Meng Lei, Yutao Hu, Peng Gao, Peng Xu, Kaipeng Zhang, Fanqing Meng, Siyuan Huang, Hongsheng Li, and Yu Qiao. 2025. Tinylvlm-ehub: Towards comprehensive and efficient evaluation for large vision-language models. IEEE Transactions on Big Data (2025)

    2025

  50. [58]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Charting new territories: Exploring the geographic and geospatial capabili- ties of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 554–563

  51. [59]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8317–8326

    2019

  52. [60]

    Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. Yfcc100m: The new data in multimedia research. Commun. ACM 59, 2 (2016), 64–73

    2016

  53. [61]

    Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. 2023. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF Internat...

    2023

  54. [62]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems 37 (2024), 95095–95169

    2024

  55. [63]

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought rea- soning. Advances in Neural Information Processing S...

    2024

  56. [64]

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. 2024. Needle in a multimodal haystack. Advances in Neural Information Processing Systems 37 (2024), 20540–20565

    2024

  57. [65]

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-llava: Bootstrapping mathematical rea- soning for multimodal large language models. arXiv preprint arXiv:2406.17294 (2024)

    2024 arXiv

  58. [66]

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. 2024. Logicvista: Mul- timodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973 (2024)

    2024 arXiv

  59. [67]

    Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory networks for visual and textual question answering. In International conference on machine learning. PMLR, 2397–2406

    2016

  60. [68]

    Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2023. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915 (2023)

    2023 arXiv

  61. [69]

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 21–29

    2016

  62. [70]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. 2024. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems 37 (2024), 121475–121499

    2024

  63. [71]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiao- dan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 (2021)

    2021 arXiv

  64. [72]

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems 37 (2024), 11...

    2024

  65. [73]

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. 2024. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006 (2024)

    2024 arXiv

  66. [74]

    Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. 2024. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944 (2024)

    2024 arXiv

  67. [75]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 (2023)

    2023 arXiv

  68. [76]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. In European Conference on Computer Vision . Springer, 169–186

    2024

  69. [77]

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1 (2023), 1

    2023 arXiv

  70. [78]

    Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-o1: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405 (2024)

    2024 arXiv

  71. [79]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review 11, 12 (2024)

    2024

  72. [80]

    Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. 2025. Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models. arXiv preprint arXiv:2504.21277 (2025)

    2025 arXiv

  73. [81]

    Pengfei Zhou, Fanrui Zhang, Xiaopeng Peng, Zhaopan Xu, Jiaxin Ai, Yansheng Qiu, Chuanhao Li, Zhen Li, Ming Li, Yukang Feng, et al. 2025. MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models. arXiv preprint arXiv:2504.05782 (2025)

    2025 arXiv

  74. [82]

    Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang

  75. [83]

    Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. 2024. Is your model really a good math reasoner? evaluating mathematical reasoning with checklist. arXiv preprint arXiv:2407.08733 (2024)

    2024 arXiv

  76. [84]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 (2023)

    2023 arXiv

  77. [85]

    Please list the elements in the image

    Zichen Zhu, Yang Xu, Lu Chen, Jingkai Yang, Yichuan Ma, Yiming Sun, Hailin Wen, Jiaqi Liu, Jinyu Cai, Yingzi Ma, et al. 2024. Multi: Multimodal understanding leaderboard with text and images. arXiv preprint arXiv:2402.03173 (2024). 8 A Taxonomy of Commonsense The comprehensive...

    2024

  78. [86]

    Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal table understanding. arXiv preprint arXiv:2406.08100 (2024)

    2024 arXiv

  79. [90]

    arXiv preprint arXiv:2405.13872 (2024)

    Image-of-thought prompting for visual reasoning refinement in multimodal large language models. arXiv preprint arXiv:2405.13872 (2024)

    2024 arXiv

  80. [92]

    Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. 2024. Scan- reason: Empowering 3d visual grounding with reasoning capabilities. InEuropean Conference on Computer Vision . Springer, 151–168

    2024

  81. [2017]

    International journal of computer vision 123 (2017), 32–73

    Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123 (2017), 32–73

    2017

  82. [2019]

    In 2019 international conference on document analysis and recognition (ICDAR)

    Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR) . IEEE, 947–952

    2019

  83. [2021]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114 (2021)

    2021 arXiv

  84. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  85. [2024]

    arXiv preprint arXiv:2407.11691 (2024)

    VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models. arXiv preprint arXiv:2407.11691 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.