Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligence

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MIRAGE shows that state-of-the-art vision-language models lose about 20 accuracy points when they must count objects under spatial constraints.

desk verdict A useful but under-polished benchmark paper whose headline number is misreported by more than a factor of two; the main qualitative finding survives, but the paper needs factual corrections and statistical rigor. read the letter →

arxiv 2505.10604 v2 pith:NO4OJCVD submitted 2025-05-15 cs.CV cs.AI

Chonghan Liu , Haoran Wang , Felix Henry , Pu Miao , Yajie Zhang , Yu Zhao , Peiran Wu This is my paper · ORCID
classification cs.CVcs.AI
keywords MIRAGEvision-languagemodelsspatialreasoningobjectcountingcompositionalbenchmarkvisualgroundingrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MIRAGE is a benchmark built to test whether vision-language models can count objects, understand spatial relations such as 'left of' and 'above', and combine the two in one question. The paper's central claim is that state-of-the-art models—including 72B and 78B parameter systems—handle isolated counting or relation tasks reasonably well but lose roughly 20 accuracy points when a question requires both, as in 'How many objects are to the left of the kettle and above the red container?' If that claim holds, current models lack compositional spatial reasoning rather than simply lacking perception or language skill. The benchmark also exposes occlusion, dense scenes, and ambiguous spatial referents as systematic causes of failure, and it offers a 50-question tiny subset designed to reproduce the full ranking cheaply.

What carries the argument

The load-bearing object is MIRAGE itself: 1,710 questions with three task types—Counting, Relation, and Counting with Relation—each paired with a JSON label and assigned to a difficulty tier. Difficulty is assigned by rule-based consensus using pass@64 on two small models, InternVL-2.5-4B and Qwen2.5VL-3B: a sample is Hard when both models succeed fewer than 2 times, Medium when they succeed 2 to 16 times, and Easy when either succeeds more than 16 times. That stratification is what lets the benchmark separate task complexity from model scale, and it turns the Relation-to-Combination gap into a single comparable number across models. The 50-question Tiny subset is designed as a fast proxy that preserves the performance ordering of the full benchmark.

What would settle it

Construct a held-out set of 500 freshly photographed scenes using the same counting and relation templates, have multiple independent annotators label them, and test the same models; if human disagreement is high, or if the Relation-to-Combination gap drops to a few points on fresh images, the reported limitation would be partly an artifact of label noise or data contamination.

Watch

Extended reading notes

Core claim

The paper introduces MIRAGE, a manually annotated set of 1,710 image-question pairs split into 680 Counting questions, 754 Relation questions, and 276 Counting-with-Relation questions, with images gathered from egocentric video, web search, stock photography, and original photos. On the full benchmark, Qwen2.5VL-72B scores 56.62% on Counting, 85.31% on Relation, and 36.94% on Combination, while InternVL3-78B scores 55.15%, 82.60%, and 36.10%, a roughly 20-point drop that the paper calls evidence of fundamental limitations in compositional spatial reasoning. Diagnostic experiments show that adding one exemplar or rewritten instructions helps only modestly, that simple horizontal or vertical flips lower Combination accuracy by about 6 points, and that reasoning-style prompts can introduce fluent but visually unsupported hallucinations. The authors conclude that the bottleneck is visual grounding and spatial invariance, not instruction comprehension alone.

Load-bearing premise

The benchmark's conclusions rest on the assumption that the 1,710 manual labels are correct and unambiguous, that the difficulty tiers set by two small models are fair, and that images scraped from the web are not already memorized by the tested models during pretraining.

Editorial extensions

If this is right

  • Applications that assume spatial competence, such as robot manipulation, navigation, or augmented reality, should not rely on current vision-language models for count-within-spatial-constraint queries.
  • Prompt engineering yields small gains, but flips and noise still break performance, so fixes are more likely to come from spatial representations than from instruction tuning.
  • Reasoning-style prompting is a double-edged tool: it improves grounding in easy cases and increases hallucination risk in ambiguous ones.
  • The 50-question Tiny subset is asserted to track full-benchmark ordering, making it usable for fast iteration during model development.
  • The 13-point gap between the 3B and 72B Qwen models on Counting shows scale helps, but not enough to close the compositional gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: the benchmark could be converted into a training set, and the decisive test of compositionality would be whether fine-tuning on Combination questions closes the gap on held-out scenes.
  • Because the image pool includes web and social-media sources, some images may already appear in pretraining corpora; the reported gap might shrink on a strictly out-of-distribution set of freshly captured photos.
  • Difficulty tiers are calibrated to two small models, so as models improve the tiers will need re-normalization; otherwise 'Hard' will silently become 'Medium'.
  • The Relation-to-Combination drop is a cheap, model-agnostic diagnostic that could be tracked during architecture development as a proxy for compositional grounding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. Liu et al. introduce MIRAGE, a 1,710-question visual question-answering benchmark with three task types: Counting, Relation, and Counting with Relation (Combination). The dataset draws on egocentric and web-sourced images with manual annotation, and each item is assigned a difficulty tier (Easy/Medium/Hard) using pass@64 scores from two small VLMs. The authors evaluate a set of open and proprietary VLMs, reporting that models perform best on Relation, worse on Counting, and worst on Combination, and interpret the large drop as evidence of limited compositional spatial reasoning. Additional experiments study prompt modifications, image augmentation robustness, and qualitative failure modes. The central claim in Section 4.1 is that even Qwen2.5VL-72B and InternVL3-78B 'show a ~20-point drop' moving from Relation to Combination, revealing fundamental limitations in compositional spatial reasoning.

Significance. The proposed MIRAGE benchmark addresses a real and under-served aspect of VLM evaluation: compositional combination of counting and spatial relations. The task decomposition is conceptually clean, and the paper states that code and data are released, which would make it a reusable diagnostic instrument. If the numerical reporting and statistical support are corrected, the qualitative finding that combination tasks are markedly harder than either component alone across a range of open models is plausible and consistent with the pattern in Table 1. The diagnostic studies (prompting, augmentation, failure modes) are also potentially useful. However, the present version contains direct contradictions between prose and tables that affect the headline claim, and the validity evidence for the tiny subset and for label quality is not yet sufficient.

major comments (5)
  1. [Section 4.1 / Table 1] Section 4.1 states that Qwen2.5VL-72B and InternVL3-78B show a '~20-point drop in accuracy when moving from Relation to Combination.' Table 1 reports for Qwen2.5VL-72B Relation=85.31 and Combination=36.94, a 48.37-point drop, and for InternVL3-78B Relation=82.60 and Combination=36.10, a 46.50-point drop. The only ~20-point drops in the table are from Counting to Combination: 56.62 to 36.94 (19.68) and 55.15 to 36.10 (19.05). This misreport of the paper's own central result by more than a factor of two must be corrected; if the intended comparison is Counting to Combination, the prose and any related framing must be adjusted consistently.
  2. [Section 4.1 / Table 1 caption] The caption of Table 1 and the text in Section 4.1 claim that 'performance trends on the tiny subset are consistent with those on the full benchmark, making it a reliable proxy.' The reported numbers contradict this. For Qwen2.5VL-3B, full-set accuracy is Counting 38.33 vs Combination 23.83, while tiny accuracy is Counting 30.00 vs Combination 40.00, reversing the ordering. For InternVL-3-8B, the full set gives Counting 44.24 > Combination 29.24, but the tiny set gives 38.00 = 38.00. Since all proprietary models are evaluated only on the tiny subset, the conclusions drawn from those models rest on an unvalidated proxy; a quantitative validation (e.g., rank correlation, per-task confidence intervals) or a reassignment of the tiny subset is required.
  3. [Section 4.2.1 / Table 2] Section 4.2.1 says 'both prompt modifications lead to consistent gains over the zero-shot baseline.' Table 2 shows the opposite for the Combination task: baseline 29.24, one-shot 28.52, prompt engineering 30.69. The following sentence adds that prompt rewriting is 'particularly on the more challenging combination task,' but its gain over baseline is only 1.45 points while one-shot loses 0.72 points. The text should either be revised to reflect the task-specific direction of the effects or the analysis should be restricted to tasks where the gains are consistent.
  4. [Section 4 / NeurIPS Checklist Item 7] The NeurIPS checklist states 'We report the confidence interval in Section 4,' but Section 4 and Table 1 contain no confidence intervals, error bars, or significance tests. The main comparisons, including the claimed ~20/48-point drops and the tiny-vs-full consistency, require some measure of sampling variability, especially given the small combination subset (276 questions) and the 50-item tiny subset. Add bootstrap confidence intervals or an equivalent, and either remove or correct the checklist statement.
  5. [Section 3.2] Section 3.2 says all 1,710 questions were 'manually annotated and underwent a rigorous review process,' but no annotation protocol, number of annotators, or inter-annotator agreement is reported. Since the benchmark's value depends on ground-truth labels being unambiguous and correct, and since systematic label noise would directly affect the reported model accuracies, the paper should report at least a sample-based inter-annotator agreement (e.g., Cohen's kappa or percentage agreement) and describe how disagreements were resolved.
minor comments (6)
  1. [Section 3.2.2] Difficulty tiers are defined by pass@64 of InternVL-2.5-4B and Qwen2.5VL-3B; this makes statements such as 'state-of-the-art models fail on Hard questions' partly definitional for those model families. Consider framing difficulty labels as 'model-relative' rather than absolute.
  2. [Section 4.2.2 / Table 3] The introductory sentence refers to 'the same counting tasks,' but Table 3 reports 'Counting with Relation accuracy' (30.94, 24.82, 28.42). The text and table should use the same task name.
  3. [Table 1] Several proprietary entries are hard to parse (e.g., '-48.00- 58.00 - 50.00' for Claude-3.5-sonnet); use separate columns or explicit null markers.
  4. [NeurIPS Checklist] The checklist gives conflicting answers about asset release: item 4/5 state code and data are released, while item 13 says 'This paper does not release new assets.' Resolve this inconsistency and, if assets are released, include a license.
  5. [Appendix B.1] Appendix B.1 lists a 'Two-Stage Prompting' strategy but Table 2 and the text refer to 'One-shot' and 'Prompt Engineering'; clarify how these map to the two-stage and direct-sampling conditions, and provide the exact one-shot prompt used.
  6. [Appendix A] Appendix A lists images from Weibo, Taobao, Baidu, Xiaohongshu, 500px, and Google Images without license or terms-of-service information, despite the checklist's claim that licenses are mentioned. Please add this information or state explicitly that it is not available.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline Counting/Relation/Combination comparison is measured independently, but the benchmark's difficulty tiers and Tiny subset are defined from the same Qwen/InternVL model families later discussed, making Hard/Medium/Easy and tiny-proxy statements partly self-definitional. A separate Section 4.1 vs. Table 1 numerical mismatch is a correctness issue, not circularity.

  1. self definitional [Section 3.2.2 (Tiny Subset and Difficulty Tiers), Figure 3 caption; used in Figure 1 right and Section 4.1 discussion of difficulty]
    "Using a pass@64 metric evaluated over InternVL-2.5-4B and Qwen2.5VL-3B, we label a sample as Hard if both models succeed fewer than 2 times, Medium if between 2 and 16 completions are correct, and Easy if either model succeeds more than 16 times."

    The tier labels are constructed from pass@64 of two Qwen/InternVL models, and the Tiny subset is likewise selected by Qwen2.5VL-3B vs. 72B success. Statements that models in these same families perform worse on Hard than Easy items, or that the Tiny subset is a 'trend-preserving proxy,' partly restate the selection criterion rather than an independently measured property of the questions. The paper's headline Counting/Relation/Combination comparison, however, is not derived from these labels: Table 1 reports measured accuracies per task type, so the central conclusion does not reduce by construction. This is a partial, secondary circularity in the difficulty-tier framing.

full rationale

MIRAGE's principal claim—that VLMs drop sharply from Counting/Relation to Counting-with-Relation—is supported by independent measurements in Table 1 and is not an algebraic consequence of the benchmark definitions. No parameter is fitted and then renamed as a prediction in the main task-type comparison, and no load-bearing uniqueness theorem or self-citation chain forces the conclusion. The one genuine circular element is the difficulty taxonomy: Hard/Medium/Easy are operationalized through pass@64 of Qwen2.5VL-3B and InternVL-2.5-4B, and the Tiny subset is selected by Qwen2.5VL-3B vs. 72B success, so any inference that 'models fail the Hard items' for these same model families is partly definitional. This does not undermine the task-type result, which is why the score is 4 rather than higher. Separately, but outside the circularity rubric, Section 4.1's claim of a '~20-point drop ... from Relation to Combination' is not supported by the paper's own Table 1: the Relation-to-Combination drops are about 48 points for Qwen2.5VL-72B and 46.5 points for InternVL3-78B, while the ~20-point drops occur from Counting to Combination. That is an internal numerical inconsistency to correct, not a circular deduction. The self-citation [21] by co-author Peiran Wu is confined to related work and is not load-bearing.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark's central results rest on dataset-construction choices, such as thresholds and augmentation settings, and on data-quality assumptions, such as label correctness, no leakage, and model-relative difficulty, rather than on mathematical axioms. The quantitative conclusions inherit these choices, so they should be read as conditional on the benchmark rather than as unconditional measurements.

free parameters (2)
  • Difficulty thresholds (2 and 16 correct passes) = Hard: fewer than 2, Medium: 2 to 16, Easy: more than 16 correct out of 64
    Section 3.2.2: thresholds were chosen by hand to split pass@64 counts from InternVL-2.5-4B and Qwen2.5VL-3B into tiers; every use of the Easy, Medium, and Hard labels inherits this choice.
  • Image augmentation parameters = Gaussian noise sigma=15; salt-and-pepper 2%; Gaussian blur 5x5, sigma=1.5; contrast alpha~U(0.8,1.2), beta~U(-20,20)
    Appendix B.2: these values are chosen by hand for the perturbation study; the magnitude of the reported accuracy drops in Table 3 depends on them.
assumptions (3)
  • domain assumption The 1,710 manually annotated questions have correct, unambiguous ground-truth answers.
    Section 3.2 states that all questions were manually annotated and reviewed, but no inter-annotator agreement, adjudication, or error-rate statistics are reported.
  • domain assumption Difficulty tiers based on pass@64 of InternVL-2.5-4B and Qwen2.5VL-3B are a meaningful measure of intrinsic task difficulty.
    Section 3.2.2 defines Hard, Medium, and Easy from these model scores; this is a model-relative measure, not an independent standard such as human performance.
  • domain assumption Web-scraped images are representative and are not already memorized by the evaluated models.
    Appendix A lists Google Images, Baidu, Weibo, Taobao, and other web sources, but the paper does not test for training-data leakage or verify whether the evaluated VLMs have seen these images.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligence." pith.science (2026). https://pith.science/paper/NO4OJCVD

@misc{pith2026250510604,
  author       = {Pith},
  title        = {Pith review of: MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NO4OJCVD}},
  note         = {Machine review of arXiv:2505.10604}
}
read the original abstract

Spatial perception and reasoning are core components of human cognition, encompassing object recognition, spatial relational understanding, and dynamic reasoning. Despite progress in computer vision, existing benchmarks reveal significant gaps in models' abilities to accurately recognize object attributes and reason about spatial relationships, both essential for dynamic reasoning. To address these limitations, we propose MIRAGE, a multi-modal benchmark designed to evaluate models' capabilities in Counting (object attribute recognition), Relation (spatial relational reasoning), and Counting with Relation. Through diverse and complex scenarios requiring fine-grained recognition and reasoning, MIRAGE highlights critical limitations in state-of-the-art models, underscoring the need for improved representations and reasoning frameworks. By targeting these foundational abilities, MIRAGE provides a pathway toward spatiotemporal reasoning in future research.

Figures

Figures reproduced from arXiv: 2505.10604 by the authors.

Figure 1
Figure 1. Left: Examples of our three task types—Counting, Relation, and Counting with Relation. Tasks increase in complexity as they require understanding object attributes, spatial relationships, and their composition. Right: Model performance across difficulty tiers (Easy, Medium, Hard) and task types. State-of-the-art models show consistent drops in the Combination with Relation setting, revealing weaknesses in compositio… view at source ↗
Figure 2
Figure 2. Dataset composition and difficulty breakdown in MIRAGE. Left: Distribution of the three task types: Counting (39.8%), Relation (44.1%), and Counting & Relation (16.1%). Middle: Difficulty stratification (Easy, Medium, Hard) within each task type. Right: Overall difficulty distribution across the entire dataset: 40.9% Easy, 39.6% Medium, and 19.5% Hard. 3.1 Task Definition MIRAGE evaluates two core aspects of visual … view at source ↗
Figure 3
Figure 3. We collect images from both curated sources (e.g., EPIC-KITCHENS) and web-scale [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: We illustrate the three core task types in MIRAGE: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Failure cases for leading visual language models. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: QwenVL-max correctly counts ten chopsticks. In contrast, QvQ-72B, encouraged to "think [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Augmentation Case 1. Left: original kitchen scene. Centre: horizontally flipped. Right: Gaussian-blurred and contrast-shifted. The query targets the broom count right of the hanging plants; flipping reverses the reference frame and breaks the model’s grounding. Questio…
Figure 8
Figure 8. Figure 8: Augmentation Case 2. Street-side café scene with reflective window. Flipping disrupts left–right reflection cues, leading to under-counting of cars, while salt-and-pepper noise adds spurious edges yet leaves spatial layout intact. 2. Noise Injection. For each sample we…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models

    cs.CV 2026-03 conditional novelty 6.0 of 10

    TickTockVQA (12k real-world clocks) plus Swap-DPO lifts Llama-3.2-11B full-time analog clock accuracy from 1.41% to 46.22%, far above synthetic-data baselines.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkow...

    work page 2022

  2. [2]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    2023

  3. [3]

    Spatialbot: Precise spatial understanding with vision language models, 2025

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models, 2025

    2025

  4. [4]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu...

    2024

  5. [5]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    2024

  6. [6]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. InEuropean Conference on Computer Vision (ECCV), 2018

    2018

  7. [7]

    Geobench-vlm: Benchmarking vision-language models for geospatial tasks, 2025

    Muhammad Sohail Danish, Muhammad Akhtar Munir, Syed Roshaan Ali Shah, Kartik Kuckreja, Fahad Shahbaz Khan, Paolo Fraccaro, Alexandre Lacoste, and Salman Khan. Geobench-vlm: Benchmarking vision-language models for geospatial tasks, 2025

    work page 2025

  8. [8]

    Mm-spatial: Exploring 3d spatial understanding in multimodal llms, 2025

    Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang, Afshin Dehghan, and Peter Grasch. Mm-spatial: Exploring 3d spatial understanding in multimodal llms, 2025

    2025

Show all 34 references
  1. [9]

    Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Luˇci´c, and Neil Houlsby. Patch n’ pack: Navit, ...

    2023

  2. [10]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  4. [12]

    Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024

  5. [13]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    2022

  6. [14]

    Sti- bench: Are mllms ready for precise spatial-temporal world understanding?, 2025

    Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti- bench: Are mllms ready for precise spatial-temporal world understanding?, 2025. 10

    2025

  7. [15]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  8. [16]

    ivispar – an interactive visual-spatial reasoning benchmark for vlms, 2025

    Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, and Elia Bruni. ivispar – an interactive visual-spatial reasoning benchmark for vlms, 2025

    2025

  9. [17]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  10. [18]

    Gsr-bench: A benchmark for grounded spatial reasoning evaluation via multimodal llms, 2024

    Navid Rajabi and Jana Kosecka. Gsr-bench: A benchmark for grounded spatial reasoning evaluation via multimodal llms, 2024

    2024

  11. [19]

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yan...

    2025

  12. [20]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024

  13. [21]

    St-think: How multimodal large language models reason about 4d worlds from ego-centric videos.arXiv preprint arXiv:2503.12542, 2025

    Peiran Wu, Yunze Liu, Miao Liu, and Junxiao Shen. St-think: How multimodal large language models reason about 4d worlds from ego-centric videos.arXiv preprint arXiv:2503.12542, 2025

    2025 arXiv

  14. [22]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models, 2023

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models, 2023

    2023

  15. [23]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2024

    2024

  16. [24]

    Minicpm-v: A gpt-4v level mllm on your phone, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. M...

    2024

  17. [25]

    Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data, 2024

    Chenhui Zhang and Sherrie Wang. Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data, 2024

    2024

  18. [26]

    image_caption

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025

  19. [27]

    Main Course (Foreground) •Donburi-style bowl with tonkatsu rice •Topped with shredded egg (kinshi tamago) •Garnished with finely chopped green onions

  20. [28]

    Side Dishes •Sushi plate: Three pieces on decorative seaweed base •Harumaki (spring rolls): Vegetable-filled, sesame-garnished •Tempura or katsu: Breaded and fried, with drizzled sauce

  21. [29]

    Tea Service •Traditional ceramic teapot: Blue and white pattern •Two matching teacups with geometric designs

  22. [30]

    Additional Items •Background bowl: Secondary rice dish or soup •Side plate: Stir-fried vegetables with meat

  23. [31]

    } Stage 2: Task-Specific Questions a. Spatial Relation Task RELATION BASE PROMPT You should output a json string with format {

    Condiments •Small ceramic pitcher containing soy sauce Overall Ambiance: The presentation suggests an authentic Japanese restaurant setting, with attention to traditional serving methods and aesthetic arrangement." } Stage 2: Task-Specific Questions a. Spatial Relation Task RE...

  24. [32]

    3Implemented with PIL.Image.transpose

    Geometric Flip.We applyhorizontal(‘left–right”) andvertical(‘top–bottom”) flips 3 to ex- amine whether models properly internalise spatial relations rather than memorising canonical arrangements. 3Implemented with PIL.Image.transpose. The operation leaves low-level statistics ...

  25. [33]

    Limitations

    Noise Injection.For each sample werandomly pick oneof the following four photometric corruptions: • Gaussian Noise— additive noise drawn fromN(0,σ 2) withσ= 15 (RGB range [0,255] ), simulating sensor noise; • Salt-and-Pepper Noise— 2% of pixels are randomly set to either 0 or ...

  26. [34]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.