Pith. sign in

REVIEW 2 major objections 4 minor 100 references

A new 482-pair benchmark of real photos shows most vision-language models at near-chance levels on spatial-relation questions that humans answer correctly 95 percent of the time, while reasoning models approach human performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new contrastive real-image benchmark shows most vision-language models fail spatial relation tasks, while chain-of-thought reasoning models approach human-level accuracy.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid new benchmark with a real empirical gap, but the chance baseline is mis-set and the reasoning-vs-localization claim outruns its probe. the 2 major comments →

arxiv 2509.02175 v2 pith:F2IM7JQH submitted 2025-09-02 cs.CV cs.AIcs.CLcs.LG

Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks

Nils Hoehing , Mayug Maniparambil , Ellen Rushe , Noel E. O'Connor , Anthony Ventresque This is my paper
classification cs.CV cs.AIcs.CLcs.LG
keywords spatial relationsvision-language modelscontrastive benchmarkchain-of-thought reasoningspatial reasoning vs localizationmultimodal evaluationreal-world image-text pairs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RocketScience is a new benchmark of 482 contrastive image-text pairs built from fresh real-world photos, designed so that each question is trivial for a person (human accuracy 0.95) while forcing a model to compare two captions and two images that differ only in the position of objects. The paper's central claim is that nearly all current vision-language models, including frontier closed models, perform at or near chance on this benchmark, while models trained to reason step-by-step approach human performance (o4-mini 0.89, Gemini 2.5 Pro 0.83). It also claims to separate two skill components—finding objects and inferring their spatial relation—and concludes that the failure is not about perception but about reasoning: gpt-4o localizes objects nearly as well as o4-mini on a horizontal-position probe, yet cannot turn that into the correct relation. If this is right, the benchmark is a clean diagnostic that redirects effort from better object detection toward relational reasoning, and it provides a release-time difficulty score for comparing benchmarks fairly.

Core claim

On its own terms, the paper establishes that spatial-relation understanding in real images is largely missing from ordinary VLMs and present only in explicit reasoning models. Every non-reasoning model tested—CLIP-style dual encoders, open VLMs, and commercial VLMs such as GPT-4o and Claude Sonnet—scores near the 0.17 random baseline (0.01 to 0.24 group score), despite the same tasks being solved by humans at 0.985 with 0.008 standard deviation. Prompting the same gpt-4o to reason before answering lifts it to 0.51, and models with internal chain-of-thought—Gemini 2.5 Pro at 0.83 and o4-mini at 0.89—approach humans. The paper's disentanglement experiment asks gpt-4o and o4-mini to output boun

What carries the argument

The contrastive pair is the load-bearing object: a tuple of two new real photographs and two captions that differ only by the position of the same two objects (e.g., word order or swapped preposition), so neither language priors nor object presence can give away the answer. Evaluation splits each pair into four forced-choice questions (image-with-two-captions and caption-with-two-images) and a group score that requires both directions to be correct, following Winoground's scoring adapted to VLMs. The disentanglement probe is a bounding-box configuration check: models name boxes for both objects and the test verifies that the box order matches the caption's left/right relation, deliberately n

Load-bearing premise

The claim that failures are bottlenecked by reasoning rather than localization rests on a probe that checks left/right box order on only the horizontal-position subset for just two models; if depth, proximity, order, or vertical items demand harder localization, the conclusion overstates the evidence.

What would settle it

Run the localization probe on all five categories, not just the horizontal subset, for gpt-4o and o4-mini: if gpt-4o's bounding-box accuracy falls well below o4-mini's on depth, proximity, or order items, the paper's conclusion that object localization is not the bottleneck is falsified; if the two models stay matched across all categories, the conclusion holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RocketScience gives a human-trivial, machine-hard test where chance is 0.17; any model that cannot exceed it lacks usable spatial-relation understanding of real scenes.
  • Because gpt-4o jumps from 0.19 to 0.51 merely by being asked to reason aloud, evaluation reports should state whether chain-of-thought was used; a model's default single-pass score understates what the same weights can do.
  • The bottleneck claim implies that improving detectors or adding more spatial training data to encoders will not fix the failure; the lever is relational inference, e.g., CoT prompting or reasoning-trained models.
  • Current non-contrastive or recycled benchmarks likely overstate spatial ability; the paper's release-time difficulty metric offers a way to compare benchmarks fairly as models improve.
  • The dataset itself, released openly with a permissive license, is a reusable diagnostic for tracking whether future VLMs genuinely improve on spatial relations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A cheap two-stage system—off-the-shelf object detector plus a symbolic rule that compares bounding-box relations—would probably pass much of RocketScience, which would test whether the benchmark is truly beyond perception; the paper does not run this baseline.
  • The mirrored-image pairs in the horizontal category could introduce a non-spatial cue (model left/right bias or mirrored text), so an ablation using only physically transported objects would strengthen the claim; the paper acknowledges mirroring but does not quantify its effect.
  • Reasoning gains are shown only on top-tier commercial models; whether smaller open reasoning models get the same boost is untested, so the conclusion that 'reasoning is the bottleneck' may not transfer to open-weight models.
  • Because all images come from Europe and the US, the spatial-relation failure may be intertwined with object familiarity; testing with objects from other regions would separate purely spatial competence from cultural familiarity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces RocketScience, a new contrastive vision-language benchmark of 482 manually curated, real-world image-text pairs designed to measure relative spatial relation understanding (horizontal/vertical position, depth, proximity, order, and a small absolute-position subset). The benchmark is intended to be trivial for humans but difficult for current VLMs, and the authors report a human accuracy of 0.985 versus chance-level performance for most non-reasoning models and near-human performance for reasoning models (o4-mini 0.89, Gemini 2.5 Pro 0.83). The paper also presents a disentanglement analysis claiming that performance is bottlenecked by spatial reasoning rather than object localization, and releases the dataset and evaluation code.

Significance. If the claims hold, RocketScience would be a useful diagnostic: it is contrastive, uses new real-world images, includes hard negatives, controls for object presence, and is easy for humans but hard for existing VLMs. The strengths of the paper include the manual curation and diversity of the dataset, a human baseline, the explicit contrastive-pair design, a stability check over random subsets (Table 6), and the public release of data and code. The disentanglement analysis, if made rigorous, would provide a concrete separation of perception and reasoning failures. However, two load-bearing issues need addressing before the paper's central claims can be accepted at face value: the reported random-chance baseline appears to be mis-set for the VLM evaluation protocol, and the localization probe used to support the 'bottlenecked by reasoning' claim covers only a small subset of the benchmark with a very coarse spatial-configuration check.

major comments (2)
  1. [§4.3, Eq. (4); Table 1; Table 3] The random-chance baseline is mis-set for the VLM group score. Under the four-question forced-choice protocol in §4.2, each of the two decisions in f and each of the two decisions in g has probability 1/2 under uniform guessing, so the group score h has chance 1/16 ≈ 0.0625, not the reported 0.17. Table 3 itself reports random text score 0.25 and image score 0.25, which also implies a group chance of 0.0625. This is not a cosmetic issue: the abstract and Section 5 state that non-reasoning models 'perform at chance levels,' but with n=482, gpt-4o (0.19), Llama-4 (0.20), and Claude-3.7 (0.24) are many standard errors above 0.0625, whereas they are only about one standard error above 0.17. The authors should correct the baseline, recompute the 'above chance' conclusions, and temper the corresponding wording.
  2. [§5.1, Fig. 4a] The claim that 'performance on the benchmark is bottlenecked by spatial reasoning and not object localization' (Abstract, §5.1, §6.3) is supported by a localization probe that is substantially narrower than the benchmark itself. The probe runs only on the horizontal-position subset, only on gpt-4o and o4-mini, and only checks whether two requested bounding boxes are in the correct left/right configuration, with no stated tolerance and no test of depth, proximity, order, attribute-based localization, or cluttered scenes from the other categories. A model can pass this coarse check while still failing to localize objects in the harder cases present in the full 482-pair benchmark. The bottleneck conclusion should either be supported by a broader probe or explicitly qualified as applying only to the horizontal-position subset.
minor comments (4)
  1. [§4.4, Appendix G] The human evaluation uses n=4 participants and one question per contrastive pair. The result 0.985 ± 0.008 is suggestive of low ambiguity, but the wording 'prove that examples are not ambiguous' (Section 3.4) is stronger than the evidence supports. A larger or per-item human agreement study, or a softer claim, would be more appropriate.
  2. [§3.2.1] For horizontally mirrored images, the claim that 'mirrored text may be visible, but never in a way that would simplify the task' is not supported by examples or a systematic check. Since mirrored text is a potential non-spatial cue, the authors should either provide evidence that such cases are not exploitable or exclude them from the analysis.
  3. [Table 1] Results for API models are single-run, as acknowledged in §6.1, but the table itself gives no indication of this. Adding a footnote or error bars would help readers interpret the differences between models.
  4. [Eq. (1), Figure 3a] The normalized benchmark-quality score uses a 'Random' denominator that depends on the number of choices in each benchmark. Since benchmarks differ in their random baselines, the cross-benchmark comparison in Figure 3a may be partly driven by this choice. Clarify or normalize in a way that accounts for different chance levels.

Circularity Check

0 steps flagged

No significant circularity: claims are anchored by external human baselines and independent model evaluations.

full rationale

RocketScience is a new benchmark with externally anchored validation: human performance is measured (Section 4.4, mean 0.985) and model scores are reported for independent commercial and open models. The central 'hard for VLMs, easy for humans' claim is empirical, not derived from the benchmark's construction. The disentanglement analysis (Section 5.1) is an experimental decomposition: it measures two models' bounding-box localization accuracy on the horizontal-position subset and compares CoT vs. non-CoT group scores; the conclusion that spatial reasoning is the bottleneck is an inductive inference and does not follow by construction from Eqs. 2-4. The benchmark quality score (Eq. 1) is an explicitly defined normalization, not a prediction fitted to the data. The paper contains minor self-citations ([25], [18]) but they support general motivation and are corroborated by non-self references; they are not load-bearing. Two limitations are noted but are not circularity: the reported random-chance baseline 0.17 (Tables 1 and 3) is inconsistent with the four-question group-score protocol (uniform guessing gives 0.0625), and the localization probe (Section 5.1, Figure 4a) covers only the horizontal subset with two models. These affect the strength of the 'at chance' and 'bottleneck' claims but do not make any derivation reduce to its own inputs. No circular step can be exhibited, so score 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No data-fitted parameters exist; RocketScience is an evaluation artifact, not a fitted model. The listed free parameters are hand-selected anchors or tolerances that affect reported secondary numbers (Eq 1 difficulty score, Figure 4a localization accuracies). The axioms are the benchmark's methodological bets: contrastive construction blocks shortcuts, mirroring is clean, the human pilot proves unambiguity, single API runs are stable, and box-drawing is a valid localization proxy. The last two directly govern the strength of the headline bottleneck claim.

free parameters (2)
  • SOTA_at_release anchor in benchmark quality score (Eq 1)
    The difficulty metric Score = 1 - SOTAAtRelease/(1 - Random) uses the authors' selection of the best non-CoT model at release; choosing among Claude-3-7 (0.24), GPT-4o (0.19) or Llama-4 (0.20) changes the Figure 3a values. It does not affect any model's score.
  • Bounding-box 'correct spatial configuration' tolerance in the localization probe = unspecified
    Section 5.1 verifies only that both requested boxes are in the correct left/right order; the rule and tolerance are not stated, so the reported 90.6-96.7% localization accuracies cannot be exactly reproduced or audited.
axioms (5)
  • domain assumption Contrastive design with both target objects present in both images blocks object-presence and linguistic-co-occurrence shortcuts.
    Sections 2.2, 3.1, 3.5. Inherited from Winoground methodology; if a residual shortcut existed, the benchmark would measure detection or text likelihood rather than spatial relations.
  • domain assumption Horizontal mirroring preserves the intended spatial-relation signal and adds no exploitable non-spatial cue.
    Section 3.2.1: mirrored text is claimed to never simplify the task, but lighting gradients, shadows, background asymmetry and duplicated objects are not audited. This is load-bearing for the largest category (horizontal position, Figure 8).
  • domain assumption Human accuracy of 0.985 from n=4 non-expert participants establishes that the dataset is unambiguous.
    Section 4.4: one question per contrastive pair per participant, scores pooled across testers. A small convenience sample underpins the claim that model failures reflect model limits, not ambiguous items.
  • domain assumption Single temperature-0 API runs are stable enough to rank models.
    Section 4.2 and Limitations 6.1: the authors note API outputs can still vary at temperature 0; Table 6 documents stability for gpt-4o and Gemini 2.5 Pro only, so stability for o4-mini and GLM-4.1v is assumed.
  • ad hoc to paper The bounding-box localization test is a faithful proxy for the localization required by the contrastive task.
    Section 5.1: models receive one image and produce boxes; the check verifies only the left/right order of two boxes. The paper uses this to conclude that perception is not the bottleneck, generalizing to all categories.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks." pith.science (2026). https://pith.science/paper/F2IM7JQH

@misc{pith2026250902175,
  author       = {Pith},
  title        = {Pith review of: Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F2IM7JQH}},
  note         = {Machine review of arXiv:2509.02175}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose RocketScience, an open-source contrastive VLM benchmark that tests for spatial relation understanding. It is comprised of entirely new real-world image-text pairs covering mostly relative spatial understanding and the order of objects. The benchmark is designed to be very easy for humans and hard for the current generation of VLMs, and this is empirically verified. Our results show a striking lack of spatial relation understanding in open source and frontier commercial VLMs and a surprisingly high performance of reasoning models. Additionally, we perform a disentanglement analysis to separate the contributions of object localization and spatial reasoning in chain-of-thought-based models and find that the performance on the benchmark is bottlenecked by spatial reasoning and not object localization capabilities. We release the dataset with a CC-BY-4.0 license and make the evaluation code available at: https://github.com/nilshoehing/rocketscience

Figures

Figures reproduced from arXiv: 2509.02175 by Anthony Ventresque, Ellen Rushe, Mayug Maniparambil, Nils Hoehing, Noel E. O'Connor.

Figure 1
Figure 1. Figure 1: Left: A contrastive pair from our RocketScience benchmark, showing two images and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the contrastive RocketScience dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the RocketScience benchmark. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Disentanglement experiments of (a) localisation and (b) CoT reasoning [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Function of CLIP-like models: they embed each image and text into a vector independently [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Function of VLMs: they receive either an image and two captions and have to choose a [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples for scene diversity in RocketScience. [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Dataset distribution, relative proportions [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Physical Object Counts 25 [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Adjective Counts 26 [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Human baseline interface with two images [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Human baseline interface with two captions [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

100 extracted references · 32 canonical work pages · 6 internal anchors

  1. [1]

    On the limitations of vision-language models in understanding image transforms, 2025

    Ahmad Mustafa Anis, Hasnain Ali, and Saquib Sarfraz. On the limitations of vision-language models in understanding image transforms, 2025. URL https://arxiv.org/abs/2503. 09837

    2025

  2. [2]

    Claude sonnet 3.7

    Anthropic. Claude sonnet 3.7. https://www.anthropic.com/news/claude-3-7-sonnet ,

  3. [3]

    Vismin: Visual minimal-change understanding, 2025

    Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal-change understanding, 2025. URL https://arxiv.org/abs/2407.16772

    Pith/arXiv arXiv 2025

  4. [4]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv.org/abs/2308.12966

    Pith/arXiv arXiv 2023

  5. [5]

    Qwen2.5-vl technical report,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report,

  6. [6]

    Worldsense: A synthetic benchmark for grounded reasoning in large language models, 2023

    Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models, 2023

    2023

  7. [7]

    Paligemma: A versatile 3b vlm for transfer, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

    Pith/arXiv arXiv 2024

  8. [8]

    Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images, 2023

    Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images, 2023. URL https://arxiv.org/abs/ 2303.07274. 10

    Pith/arXiv arXiv 2023

  9. [9]

    Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models

    Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: a commonsense and world knowledge challenge for large vision and language models, 2024. URL https: //arxiv.org/abs/2407.19474

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Spatialbot: Precise spatial understanding with vision language models, 2025

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models, 2025. URL https://arxiv.org/abs/2406.13642

    Pith/arXiv arXiv 2025

  11. [11]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024. URL https://arxiv.org/abs/2401.12168

    Pith/arXiv arXiv 2024

  12. [12]

    Are we on the right way for evaluating large vision-language models?, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models?, 2024. URL https://arxiv.org/abs/2403.20330

    Pith/arXiv arXiv 2024

  13. [13]

    Spatialrgpt: Grounded spatial reasoning in vision language models, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models, 2024. URL https://arxiv.org/abs/2406.01584

    Pith/arXiv arXiv 2024

  14. [14]

    Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. CoRR, abs/2202.04053, 2022. URL https://arxiv.org/abs/2202.04053

    Pith/arXiv arXiv 2022

  15. [15]

    TraVLR: Now you see it, now you don’t! a bimodal dataset for evaluating visio-linguistic reasoning

    Keng Ji Chow, Samson Tan, and Min-Yen Kan. TraVLR: Now you see it, now you don’t! a bimodal dataset for evaluating visio-linguistic reasoning. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 3322–3347, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics

    2023

  16. [16]

    Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models,

    Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models,

  17. [17]

    Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations, 2024

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations, 2024. URL https://arxiv.org/abs/2406.11171

    Pith/arXiv arXiv 2024

  18. [18]

    Metamorphic testing for pose estimation systems, 2025

    Matias Duran, Thomas Laurent, Ellen Rushe, and Anthony Ventresque. Metamorphic testing for pose estimation systems, 2025. URL https://arxiv.org/abs/2502.09460

    Pith/arXiv arXiv 2025

  19. [19]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024. URL https://arxiv.org/abs/2404.12390

    Pith/arXiv arXiv 2024

  20. [20]

    Benchmarking spatial relationships in text-to-image generation, 2022

    Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation, 2022

    2022

  21. [21]

    Gemini 2.5

    Google. Gemini 2.5. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ , 2025. Accessed: May 2025

    2025

  22. [22]

    Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment

    Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment, 2024. URL https://arxiv.org/abs/2312.03766

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d, 2020

    Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d, 2020. URL https://arxiv.org/abs/2012. 01634

    2020

  24. [24]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325–6334,

    2017

  25. [25]

    What’s left can’t be right – the remaining positional incompetence of contrastive vision-language models, 2023

    Nils Hoehing, Ellen Rushe, and Anthony Ventresque. What’s left can’t be right – the remaining positional incompetence of contrastive vision-language models, 2023. URL https://arxiv. org/abs/2311.11477

    Pith/arXiv arXiv 2023

  26. [26]

    Sugar- crepe: Fixing hackable benchmarks for vision-language compositionality, 2023

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugar- crepe: Fixing hackable benchmarks for vision-language compositionality, 2023

    2023

  27. [27]

    Jehanzeb Mirza, Jacob A

    Irene Huang, Wei Lin, M. Jehanzeb Mirza, Jacob A. Hansen, Sivan Doveh, Victor Ion Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuehne, Trevor Darrell, Chuang Gan, Aude Oliva, Rogerio Feris, and Leonid Karlinsky. Conme: Rethinking evaluation of compositional reasoning for modern vlms, 2024. URL https://arxiv.org/abs/2406.08164

    Pith/arXiv arXiv 2024

  28. [28]

    Visfactor: Benchmarking fundamental visual cognition in multimodal large language models, 2025

    Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, and Zhaopeng Tu. Visfactor: Benchmarking fundamental visual cognition in multimodal large language models, 2025. URL https://arxiv.org/abs/2502. 16435

    2025

  29. [29]

    Mmevalpro: Calibrating multimodal benchmarks towards trustworthy and efficient evaluation, 2025

    Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, and Ming Zhang. Mmevalpro: Calibrating multimodal benchmarks towards trustworthy and efficient evaluation, 2025. URL https://arxiv.org/abs/2407.00468

    Pith/arXiv arXiv 2025

  30. [30]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    2017

  31. [31]

    What’s “up” with vision-language models? investigating their struggle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9161–9175, Singapore, December 2023. Association for Computational L...

    doi:10.18653/v1/2023.emnlp-main.568 2023

  32. [32]

    Naturalbench: Evaluating vision-language models on natural adversarial samples, 2024

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples, 2024. URL https://arxiv.org/ abs/2410.14669

    Pith/arXiv arXiv 2024

  33. [33]

    An Inverse Scaling Law for CLIP Training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training, 2023. URL https://arxiv.org/abs/2305.07017

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Omnibench: Towards the future of universal omni-language models, 2024

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, and Chenghua Lin. Omnibench: Towards the future of universal omni-language models, 2024. URL https:...

    arXiv 2024

  35. [35]

    A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges, 2025

    Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges, 2025. URL https://arxiv.org/abs/2501.02189

    Pith/arXiv arXiv 2025

  36. [36]

    Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models, 2024

    Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models, 2024. URL https://arxiv.org/abs/2409.09788

    Pith/arXiv arXiv 2024

  37. [37]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  38. [38]

    Visual spatial reasoning, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023. 12

    2023

  39. [39]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URL https://arxiv.org/abs/2307.06281

    Pith/arXiv arXiv 2024

  40. [40]

    Wildvision: Evaluating vision-language models in the wild with human preferences, 2024

    Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences, 2024. URL https://arxiv.org/abs/2406.11069

    Pith/arXiv arXiv 2024

  41. [41]

    Meta. Llama 4. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ ,

  42. [42]

    BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval

    Imanol Miranda, Ander Salaberria, Eneko Agirre, and Gorka Azkune. Bivlc: Extending vision-language compositionality evaluation with text-to-image retrieval, 2024. URL https: //arxiv.org/abs/2406.09952

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  43. [43]

    Spartqa: : A textual question answering benchmark for spatial reasoning, 2021

    Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa: : A textual question answering benchmark for spatial reasoning, 2021

    2021

  44. [44]

    Romero, Misha Smelyanskiy, Shuran Song, Lyne Tchapmi, Andrew Z

    NVIDIA, :, Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, Jinwei Gu, Siddharth Gururani, Imad El Hanafi, Zekun Hao, Jacob Huffman, Jingyi Jin, Brendan Johnson, Rizwan Khan, George Kurian, Elena Lantz, Nayeon Lee, Zhaoshuo Li, Xuan Li, Tsung-Yi Lin,...

    Pith/arXiv arXiv 2025

  45. [45]

    o4-mini system card

    o4 mini. o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf ,

  46. [46]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, A...

    Pith/arXiv arXiv 2024

  47. [47]

    Teaching clip to count to ten, 2023

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten, 2023. URL https://arxiv.org/abs/2302.12066

    Pith/arXiv arXiv 2023

  48. [48]

    Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena, 2022

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena, 2022

    2022

  49. [49]

    Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jaeho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, ...

    Pith/arXiv arXiv 2025

  50. [50]

    Forest: Frame of reference evaluation in spatial reasoning tasks, 2025

    Tanawan Premsri and Parisa Kordjamshidi. Forest: Frame of reference evaluation in spatial reasoning tasks, 2025. URL https://arxiv.org/abs/2502.17775

    arXiv 2025

  51. [51]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machi...

    2021

  52. [52]

    Vision language models are blind, 2024

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind, 2024. URL https://arxiv.org/abs/2407.06581

    Pith/arXiv arXiv 2024

  53. [53]

    Atkinson, Aaditya Baranwal, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D

    Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang, Anh Totti Nguyen, David I. Atkinson, Aaditya Baranwal, Alexan...

    Pith/arXiv arXiv 2025

  54. [54]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. O...

    work page 2022

  55. [55]

    An empirical analysis on spatial reasoning capabilities of large multimodal models, 2024

    Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models, 2024. URL https://arxiv.org/abs/2411.06048

    Pith/arXiv arXiv 2024

  56. [56]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics,

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics,

    work page

  57. [57]

    A corpus for reasoning about natural language grounded in photographs, 2019

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs, 2019

    work page 2019

  58. [58]

    Eva-clip: Improved training techniques for clip at scale, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale, 2023

    2023

  59. [59]

    Space3D-Bench: Spatial 3D Question Answering Benchmark

    Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, and Marc Pollefeys. Space3d-bench: Spatial 3d question answering benchmark, 2024. URL https://arxiv.org/ abs/2408.16662

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  60. [60]

    URL https://arxiv.org/abs/2411.16537

    arXiv

  61. [61]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5238–5248, June 2022

    work page 2022

  62. [62]

    Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving, 2025

    Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving, 2025. URL https://arxiv.org/abs/2504.03164

    Pith/arXiv arXiv 2025

  63. [63]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. URL https://arxiv. org/abs/2401.06209

    Pith/arXiv arXiv 2024

  64. [64]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025

    V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, J...

    Pith/arXiv arXiv 2025

  65. [65]

    Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features,

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features,

    work page

  66. [66]

    NL-Eye: Abductive NLI for Images

    Mor Ventura, Michael Toker, Nitay Calderon, Zorik Gekhman, Yonatan Bitton, and Roi Reichart. Nl-eye: Abductive nli for images, 2024. URL https://arxiv.org/abs/2410.02613

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  67. [67]

    Fei Wang, Xingyu Fu, James Y . Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi-image understanding,

    work page

  68. [68]

    Image captioners are scalable vision learners too, 2023

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too, 2023

    work page 2023

  69. [69]

    Spatial457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models, 2025

    Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Spatial457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models, 2025. URL https://arxiv.org/abs/2502.08636

    work page arXiv 2025

  70. [70]

    URL https://arxiv.org/abs/2502.14786

    Pith/arXiv arXiv

  71. [71]

    Kiva: Kid-inspired visual analogies for testing large multimodal models, 2024

    Eunice Yiu, Maan Qraitem, Charlie Wong, Anisa Noor Majhi, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal models, 2024. URL https://arxiv.org/abs/2407.17773

    arXiv 2024

  72. [72]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id= Ee277P3AYC

    work page 2022

  73. [73]

    URL https://arxiv.org/abs/2406.09411

    Pith/arXiv arXiv

  74. [74]

    Is a picture worth a thousand words? delving into spatial reasoning for vision language models,

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models,

  75. [75]

    URL https://arxiv.org/abs/2406.14852

    Pith/arXiv arXiv

  76. [76]

    Humaneval-v: Benchmarking high-level visual reasoning with complex diagrams in coding tasks, 2025

    Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Benchmarking high-level visual reasoning with complex diagrams in coding tasks, 2025. URL https://arxiv.org/abs/2410.12381

    Pith/arXiv arXiv 2025

  77. [77]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  78. [80]

    When and why vision-language models behave like bags-of-words, and what to do about it?, 2022

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2022. URL https://arxiv.org/abs/2210.01936. 19

    Pith/arXiv arXiv 2022

  79. [81]

    A benchmark for compositional visual reasoning, 2022

    Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning, 2022

    work page 2022

  80. [82]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    2023

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.