Pith. sign in

REVIEW 3 major objections 7 minor 66 references

Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Almost no vision-language model can reason about multi-step spatial deformations; 3D defeats even the strongest models.

desk verdict A valuable new deformation-reasoning benchmark whose headline negative result is undercut by missing item-uniqueness checks and a missing chance baseline. read the letter →

arxiv 2507.02978 v1 pith:H3I4FEQY submitted 2025-07-01 cs.CV

Jiahuan Zhang , Shunwen Bai , Tianheng Wang , Kaiwen Guo , Kai Han , Guozheng Rao , Kaicheng Yu This is my paper
classification cs.CV
keywords spatialdeformationreasoningvision-languagemodelsbenchmark3DRubik'sCubeladdercompetitionforwardandinverse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that current vision-language models lack stable, generalizable spatial deformation reasoning — the ability to predict how a shape changes through sequenced operations like cutting, rotating, stacking, and recoloring. To test this, it builds Inf-Bench, an endlessly scalable benchmark that grades difficulty by the number of deformation steps and probes both forward prediction and reverse inference across 2D, 2.5D, and 3D. Across 18 models spanning scales and reasoning styles, human evaluators outperform every model, and in 3D most models cannot reliably complete even a single deformation step. Supervised fine-tuning and mainstream reasoning-enhancement methods (chain-of-thought, ReAct, self-reflection, tools) fail to close the 3D gap, which the authors read as evidence of an architectural bottleneck rather than a mere data or prompting shortfall.

What carries the argument

The load-bearing instrument is the Infinite Ladder competition, which defines difficulty entirely by the number of deformation steps. A model starts at one-step problems and ascends a level only by answering at least three of five questions correctly, descending on a second failure; the final level, R, is the reported score. The paper's rule-based data engine generates unlimited initial shapes and action sequences deterministically and assembles forward questions (operations to final shape) and inverse questions (final shape to operations) with distractor action lists produced by random replacement. This machinery converts 'how many sequential deformations can a model reliably track' into a single, unbounded ranking, applied here to 18 vision-language models and 100 human volunteers.

What would settle it

Run the paper's data engine on a large sample of generated questions and execute every option's action list on the initial shape; if a non-negligible fraction of questions has two options producing identical final states, the benchmark's unique-correct-answer assumption fails and the reported reasoning depths underestimate true model ability. Alternatively, exhibit one concrete question where the model's chosen distractor produces the same final shape as the target.

Watch

Extended reading notes

Core claim

The central discovery is that spatial deformation reasoning, as formalized by Inf-Bench, lies largely outside current vision-language model competence. On the ladder metric, humans reach average depths of 31.5 in 2D forward reasoning and 6.7 in 3D forward reasoning, while the best evaluated model (OpenAI o3) reaches 27.0 and 4.2, respectively; most other models score near zero in 3D. The paper further shows this is not solely a perception problem: when shapes are pre-encoded as text, 2D and 2.5D performance rises sharply while 3D performance remains near floor, and neither fine-tuning on thousands of solved examples nor built-in reasoning scaffolds produces robust 3D state tracking. The authors conclude that current VLMs do not possess the multi-step, high-dimensional state tracking that spatial deformation demands.

Load-bearing premise

Each generated multiple-choice question is assumed to have exactly one correct answer, but distractor action lists are produced by random replacement without verifying that they cannot yield the same final shape as the target operation list; with symmetric shapes, two options can be equally correct.

Editorial extensions

If this is right

  • Robotic and embodied systems that rely on VLMs for spatial planning cannot currently be expected to handle multi-step shape transformations, such as assembling parts or rearranging blocks, without external verification.
  • Static benchmarks will saturate as models improve; the ladder format offers an unbounded measure of reasoning depth that remains discriminative at arbitrarily high difficulty.
  • Because chain-of-thought, ReAct, self-reflection, and tool invocation do not meaningfully improve 3D deformation performance, progress on this capability is unlikely to come from inference-time reasoning tricks alone.
  • Supervised fine-tuning does extend 2D and 2.5D reasoning beyond the difficulty level seen in training, but 3D forward depth stays near five steps even with ten-step training data, pinpointing a concrete ceiling for current architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's unique-correct-answer assumption deserves scrutiny: since distractor action lists are created by random replacement without checking final-state equivalence, symmetric initial shapes could make two options produce identical final shapes, and a model picking the 'wrong' but semantically valid option would be incorrectly penalized, biasing the reported R values downward.
  • Because the 3D tasks are Rubik's Cube rotations — a formally structured, deterministic group — the 3D failure is especially diagnostic: models cannot track state in a domain where ground truth is computable, pointing to a deficit in compositional sequential reasoning rather than in visual perception.
  • The infinite-ladder design could transfer to other parametrized reasoning domains, such as multi-query logic or hierarchical planning, as a general way to measure reasoning ceiling without benchmark saturation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Inf-Bench, a procedurally generated benchmark for spatial deformation reasoning in vision-language models. It covers 2D, 2.5D, and 3D tasks, each with forward reasoning (operations to final state) and inverse reasoning (final state to operations), and organizes difficulty through an 'infinite ladder' competition whose metric R is the highest level a model reaches. The authors evaluate 18 VLMs plus 100 human volunteers, report that human performance exceeds all models, and find especially low 3D performance. They further test supervised fine-tuning and several prompting/enhancement methods, concluding that current VLMs lack stable, generalizable spatial deformation reasoning, particularly in 3D.

Significance. If the central result is robust, Inf-Bench would be a useful addition to VLM spatial-reasoning evaluation: the data engine is deterministic and algorithmic, the ladder format avoids fixed-difficulty saturation, and the human comparison provides an external anchor. The breadth of the evaluation (18 models across families, plus SFT and prompting ablations) is a strength. However, the paper's headline negative claim depends on the validity of its multiple-choice generation, and the current pipeline does not guarantee that each question has a unique correct answer. Because the ladder metric advances on 3-of-5 accuracy, even a modest rate of ambiguous questions could shift reported R values and undermine the conclusion that 'almost no model demonstrates plausible spatial deformation reasoning abilities.' The benchmark's contribution is therefore conditional on a uniqueness verification step, which is a fixable but load-bearing gap.

major comments (3)
  1. [§3.2, Steps 3–4; Figure 5(a)] The distractor construction does not verify uniqueness of the ground-truth option. Step 3 generates distractor action lists by random replacement, and Step 4 executes both target and distractor lists on the initial shape, but nothing checks that a distractor list cannot produce the same final shape as the target in forward tasks, or that no distractor action list is an equally valid solution in inverse tasks. With symmetric initial shapes—for example, the four identical red windmills in Figure 5(a)—rotations, mirroring, and some color operations become equivalent, so two different action lists can yield identical renders. A model that selects such an alternative is scored as wrong. Since advancement in §4.1 requires 3-of-5 correct answers per level, even a small proportion of ambiguous questions can lower R by a ladder level. The reported low values, especially in 3D, are therefore not yet a settled measure of model ability. The authors should add a uniqueness filter at generation time, report the fraction of generated questions that fail it, or otherwise prove that every distractor is visually distinct and non-equivalent as an action sequence.
  2. [Abstract; §4.3, Table 1] The abstract's claim that 'almost no model demonstrates plausible spatial deformation reasoning abilities' is inconsistent with the paper's own results. Section 4.3 states that models 'generally show a competitive performance in 2D and 2.5D tasks,' and Table 1 reports 2D forward R values of 27.0 for O3, 11.5 for Gemini-2.5-flash-preview, and 8.9 for Claude 3.7 Sonnet. The negative result is primarily a 3D phenomenon, and the wording should be restricted accordingly, or the authors should define a threshold for 'plausible' reasoning depth. This is not merely cosmetic: the abstract frames the paper's contribution, and overstatement affects the interpretation of the benchmark's significance.
  3. [§4.1, ladder formula] The ladder competition rule is under-specified, which matters because R is the paper's only metric. The text says a model that fails is 'downgraded to the same level,' while the displayed recurrence decrements R to R−1 when c<3; it is unclear whether a failure at level 1 yields R=0 or R=1, and how 'failing at the same level twice' is counted when a model moves down and then back up. The stopping condition 'R = 0' is also ambiguous. Additionally, Table 1 reports only the mean R over 10 runs, with no variance, confidence intervals, or per-run distribution; for 3D tasks where mean values are 0.0–0.3, a single ladder outcome can move the mean by 0.1, so differences such as 0.1 vs. 0.3 are not interpretable. The authors should provide a precise state-transition definition of the ladder and report dispersion or raw counts.
minor comments (7)
  1. [Tables 3 and 4] The word 'Vanilia' should be 'Vanilla' in both tables.
  2. [Figure 5 caption] 'Comparision' should be 'Comparison'.
  3. [Figure 4 caption] The caption contains typos: 'groundtruth' should be 'ground truth' and 'excute' should be 'execute'.
  4. [Appendix C.1.1] The 2D encoding section says a shape can consist of up to four layers, but 2D tasks are described as single-layer in Section 3.1; clarify whether the four-layer description applies only to 2.5D.
  5. [Prompts in Appendix D] The 2D/2.5D system persona asks the model to output a detailed explanation, while the user prompt says 'You only need to output the letter corresponding to your choice.' This contradiction should be resolved so all models receive the same output instruction.
  6. [§3.1, 3D action space] The text says the Rubik's Cube has 54 visible faces and then 'The task allows all 54 basic rotations,' which conflates face count with rotation count; the action space elsewhere suggests face rotations, wide-layer rotations, and middle-layer rotations, so the number should be stated precisely.
  7. [§5.2, Table 2] Human performance on encoded text input is lower than that of several models, which reverses the paper's main human-superiority narrative; this deserves more than the single word 'Interestingly' and should be discussed as a phenomenon about human versus model information processing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is generated by a self-contained rule engine with an external human anchor, and no load-bearing claim reduces to its own inputs or to self-citation.

full rationale

The paper's central claims are empirical benchmark results, not derivations from fitted parameters or from the authors' prior results. The Inf-Bench data engine randomly generates initial shapes, target action lists, and distractor action lists, then deterministically executes them (§3.2, Steps 1-5), so the ground-truth labels are produced by the engine itself rather than by the evaluated models. The ladder metric R is defined directly from observed correct answers and is not fitted to any predicted outcome. Human performance is measured on the same tasks and used as an external anchor, so the conclusion that models underperform is not definitionally forced. The SFT experiments train and evaluate on the same benchmark, but they are presented as controlled intervention studies, not as independent predictions, and the reported limitations in 3D forward reasoning are empirical findings rather than consequences of construction. The paper does not invoke a uniqueness theorem, does not adopt an ansatz from the authors' prior work via citation, and does not rename a known result as a new contribution. Concerns about whether the multiple-choice options always have a unique correct answer, or whether the benchmark generalizes beyond the authors' task family, are validity questions rather than circularity: they do not amount to the benchmark's conclusion being equivalent to its inputs by construction. No circular step satisfying the required evidentiary standard is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The only introduced artifact is the benchmark itself, which is not an invented entity in the sense of a new force, particle, or conserved quantity. The ledger instead captures the benchmark design choices and unverified assumptions about the generator and the tasks, which are the main burden the central claim rests on.

free parameters (4)
  • Ladder pass threshold = 3 of 5 questions correct
    Chosen by hand to define advance and downgrade; the reported reasoning depth R and all conclusions about model capability depend on this threshold.
  • Allowed failures at a level = 2
    Hand-chosen stopping rule; it affects the final R and makes R a noisy estimate of capability.
  • Distractor count and replacement count = k = 3 distractors implied; r unspecified
    Distractor action lists are generated by RandReplace with unspecified r; this controls difficulty and the potential ambiguity of multiple-choice questions.
  • SFT dataset sizes and Smax levels = 20,000 samples per group; Smax in {1,5,10}
    Hand-chosen training mix; the claim that SFT cannot unlock 3D forward reasoning depends on these scales rather than an exhaustive training study.
assumptions (5)
  • domain assumption The procedural engine S correctly implements the documented deformation semantics for all operations and dimensions.
    Section 3.2 and Appendix C.3 describe fixed Python functions, but code is not shipped; a bug in rotation, stacking, or cube face handling would invalidate all ground truth labels.
  • domain assumption Every generated multiple-choice question has a unique correct answer under the stated rules.
    Steps 4 and 5 in Section 3.2 generate distractors by replacing random operations without checking that a distractor action list cannot produce the same final shape (forward) or that no alternative option equals the target operation list (inverse). Symmetric shapes make rotations and mirrors equivalent, so uniqueness is not guaranteed.
  • domain assumption The 3D cube images convey all 54 facelet states unambiguously to the model.
    Appendix C.1.3 defines each face as a 3x3 matrix, but the actual image presentation of the cube is not specified; a perspective rendering hides faces and makes rotation questions underspecified.
  • domain assumption Human performance with unlimited time is a fair reference for model capability.
    Appendix D.2 allows humans unlimited time and one answer, while models are zero-shot with greedy decoding; the comparison conflates reasoning ability with time and compute budget.
  • domain assumption Training and evaluation samples drawn from the same procedural generator do not constitute leakage.
    Section 3.2 claims no data leakage because generation is random and unlimited, but Section 5.3 trains on the same distribution as evaluation; distributional overlap is a form of leakage for generalization claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models." pith.science (2026). https://pith.science/paper/H3I4FEQY

@misc{pith2026250702978,
  author       = {Pith},
  title        = {Pith review of: Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3I4FEQY}},
  note         = {Machine review of arXiv:2507.02978}
}
read the original abstract

Humans naturally possess the spatial reasoning ability to form and manipulate images and structures of objects in space. There is an increasing effort to endow Vision-Language Models (VLMs) with similar spatial reasoning capabilities. However, it remains unclear whether these models truly understand and manipulate spatial objects or not. To address this question, we propose a new evaluation framework aimed at assessing the performance of VLMs in spatial deformation reasoning tasks. Specifically, we construct a benchmark for spatial deformation reasoning from 2D to 3D. Leveraging our data engine, we can generate unlimited evaluation problem pairs with infinite steps, without any data leakage. We explore whether the model can effectively perform spatial deformation reasoning from two directions: forward reasoning (given the operations, find the final state) and reverse reasoning (given the final state, determine the operations). We adopt a ladder competition format, using the number of deformation steps as the level classification criterion, with the goal of exploring the boundaries of the model's deformation reasoning capabilities. Interestingly, the benchmarking results reveal that almost no model demonstrates plausible spatial deformation reasoning abilities. Furthermore, even after applying targeted training and mainstream reasoning enhancement methods, the models are still unable to perform well on 3D spatial deformation reasoning.

Figures

Figures reproduced from arXiv: 2507.02978 by the authors.

Figure 1
Figure 1. Existing spatial reasoning benchmarks [39, 61, 46, 40] for Vision-Language Models (VLMs) focus on tasks like ‘which is higher’ or ‘find the shortest path,’ with a static level of difficulty, which may easily get outdated with the fast evolving of VLMs. Our Inf-Bench introduces tasks requiring forward and reverse spatial deformation reasoning (e.g., shape changes after cutting/rotating/stacking) across 2D-3D. Our lad… view at source ↗
Figure 2
Figure 2. Shape Explanation. 2D shapes are presented on a flat plane, while 2.5D introduces additional (up to four) layers of dimension. 3D shapes, based on a Rubik’s Cube, introduce depth and spatial orientation. Action Space. Actions like rotation and coloring are common in 2D and 2.5D, with 2D including “filling” and 2.5D adding “stacking.” The 3D section focuses on Rubik’s Cube rotations and movements in three dimensions.… view at source ↗
Figure 3
Figure 3. Inf-Bench Task Examples. Tasks are presented from 2D to 3D, with forward tasks (left) and reverse tasks (right). Note: All problems have been simplified to enhance clarity and conciseness. Each spatial level includes two types of tasks: Forward Reasoning and Inverse Reasoning. In Forward Reasoning Tasks, the model transforms an object from its initial state to the target state through deformation steps. These tasks … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Data Generation Pipeline. The process consists of five steps: First, a shape is generated, and then a target deformation operation list is created. Next, an interference list is generated by modifying the target list. Target and interference shapes are then produced, a…
Figure 5
Figure 5. Figure 5: Case Study. (a) illustrates the direct pairing method, where valuable information is extracted from the prompt and used to pair options directly. (b) shows the example of step-by-step reasoning execution, where the model first encodes the statement, abstracts actions i…
Figure 6
Figure 6. Figure 6: Encoded Shape Example. As noted earlier, models primarily rely on shape encoding for spatial deformation reasoning. To minimize perceptual encoding errors, we pre-encode all images uniformly, allowing models to process pure text input for reasoning. The encoding method…
Figure 7
Figure 7. Figure 7: Example of the Option of Forward Task D.2 Human Evaluation Setup When evaluating human-level performance on Inf-Bench, human evaluators are allowed unlimited time to answer the questions but can only submit one answer. They receive both the question and the correspondi…
Figure 8
Figure 8. Figure 8: Prompt of Inf-Bench (2D) E More Details about Inf-Bench. E.1 Shapes and Colors In the Shapez game, shapes can be combined using different types and colors. The shape types include circles, rectangles, windmills, sectors, and stars. Each shape can be represented in vari…
Figure 9
Figure 9. Figure 9: Prompt of Inf-Bench (2.5D) • Filling: Filling empty quadrants with a specified shape, such as filling the blank quadrant with a rectangle or circle. • Mirror: Performing a horizontal mirror operation on the entire shape, swapping quadrants 1 and 4, and quadrants 2 and …
Figure 10
Figure 10. Figure 10: Prompt of Inf-Bench (3D) RcCpSgCc WpWgWgWp: --Ww--Rc {'U': [['y', 'y', 'y'], ['o', 'b', 'o'], ['y', 'y', 'y']], 'D': [...], 'L': [...], ...} Code Explanation U face : Row 1: Yellow (y), Yellow (y), Yellow (y) Row 2: Orange (o), Blue (b), Orange (o) Row 3: Yellow (y), …
Figure 12
Figure 12. Figure 12: All Shapes and Colors • B2: Rotate the back face 180◦ Wide Layer Rotations These operations rotate two layers of the Rubik’s Cube at once, affecting adjacent layers. • r: Rotate the right two layers 90◦ clockwise • u: Rotate the upper two layers 90◦ clockwise • f: Rot…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 21 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. Anthropic Blog, 5 2024. https://www.anthropic.com/claude

    work page 2024

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  4. [4]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024

    2024

  5. [5]

    An introduction to vision-language modeling

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247, 2024

    arXiv 2024

  6. [6]

    Spatialbot: Precise spatial understanding with vision language models

    Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024

    arXiv 2024

  7. [7]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  8. [8]

    Evlm: An efficient vision-language model for visual understanding

    Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, et al. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177, 2024

    arXiv 2024

Show all 66 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. 10

    2024 arXiv

  2. [10]

    Spatialrgpt: Grounded spatial reasoning in vision language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.01584, 2024

    2024 arXiv

  3. [11]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  4. [12]

    A survey of spatial deformation from a user-centered perspective

    James Gain and Dominique Bechmann. A survey of spatial deformation from a user-centered perspective. ACM Transactions on Graphics (TOG), 27(4):1–21, 2008

    2008

  5. [13]

    Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J

    Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David ...

    2021 arXiv

  6. [14]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions

    Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214, 2024

    2024

  7. [15]

    A review on vision-language-based approaches: Challenges and applications

    Huu-Tuong Ho, Luong Vuong Nguyen, Minh-Tien Pham, Quang-Huy Pham, Quang-Duong Tran, Duong Nguyen Minh Huy, and Tri-Hai Nguyen. A review on vision-language-based approaches: Challenges and applications. Computers, Materials & Continua, 82(2), 2025

    2025

  8. [16]

    Cogvlm2: Visual language models for image and video understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  9. [17]

    Do vision-language models really understand visual language? arXiv preprint arXiv:2410.00193, 2024

    Yifan Hou, Buse Giledereli, Yilei Tu, and Mrinmaya Sachan. Do vision-language models really understand visual language? arXiv preprint arXiv:2410.00193, 2024

    2024 arXiv

  10. [18]

    Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. In 8th Annual Conference on Robot Learning, 2024

    2024

  11. [19]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  12. [20]

    Imagine while reasoning in space: Multimodal visualization-of-thought

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli ´c, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542, 2025

    2025 arXiv

  13. [21]

    Benchmark evalua- tions, applications, and challenges of large vision language models: A survey

    Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. Benchmark evalua- tions, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189, 1, 2025

    2025 arXiv

  14. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  15. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  16. [24]

    MMBench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Yike Yuan, Wangbo Zhao, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player?, 2024

    2024

  17. [25]

    Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning.arXiv preprint arXiv:2501.10074, 2025

    Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, et al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning.arXiv preprint ...

    2025 arXiv

  18. [26]

    ivispar– an interactive visual-spatial reasoning benchmark for vlms

    Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, and Elia Bruni. ivispar– an interactive visual-spatial reasoning benchmark for vlms. arXiv preprint arXiv:2502.03214, 2025

    2025

  19. [27]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, april 2025, 2025

    AI Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, april 2025, 2025

    2025

  20. [28]

    Compositional chain-of- thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024

    2024

  21. [29]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  22. [30]

    Bryan L. M. de Oliveira, Murilo L. da Luz, Bruno Brandão, Luana G. B. Martins, Telma W. de L. Soares, and Luckeciano C. Melo. Sliding puzzles gym: a scalable benchmark for state representation in visual reinforcement learning, February 2025. arXiv:2410.14038 [cs]

    2025

  23. [31]

    Openai o1 system card

    OpenAI. Openai o1 system card. OpenAI Website, 12 2024. https://cdn.openai.com/o1-system- card-20241205.pdf, Accessed: 2025-05-14

    2024

  24. [32]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. OpenAI Website, 2024. https://openai.com/index/o3-o4-mini-system-card/, Accessed: 2025-05-14

    2024

  25. [33]

    Benchmarking visual language models on standardized visualization literacy tests

    Saugat Pandey and Alvitta Ottley. Benchmarking visual language models on standardized visualization literacy tests. arXiv preprint arXiv:2503.16632, 2025

    2025 arXiv

  26. [34]

    Xiao, Katherine M

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, and Bernhard Schölkopf. Can large language models understand symbolic graphics programs? In The Thirteenth International Conference on Learnin...

    2025

  27. [35]

    Does spatial cognition emerge in frontier models? In The Thirteenth International Conference on Learning Representations, 2025

    Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  28. [36]

    Sat: Spatial aptitude training for multimodal language models

    Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2412.07755, 2024

    2024

  29. [37]

    Doan, and Chandan K

    Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D. Doan, and Chandan K. Reddy. LLM-SRBench: a new benchmark for scientific equation discovery with large language models, April 2025. arXiv:2504.10415 [cs]

    2025 arXiv

  30. [38]

    Tobias Springer. Shapez. https://store.steampowered.com/app/1318690/shapez/,

  31. [39]

    Lego-puzzles: How good are mllms at multi-step spatial reasoning? arXiv preprint arXiv:2503.19990, 2025

    Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. Lego-puzzles: How good are mllms at multi-step spatial reasoning? arXiv preprint arXiv:2503.19990, 2025

    2025 arXiv

  32. [40]

    Sparkle: mastering basic spatial capabilities in vi- sion language models elicits generalization to spatial reasoning, March 2025

    Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, and Jinhua Zhao. Sparkle: mastering basic spatial capabilities in vi- sion language models elicits generalization to spatial reasoning, March 2025. arXiv:2410.16162 [cs]

    2025

  33. [41]

    Doubao 1.5 pro

    Doubao Team. Doubao 1.5 pro. Doubao Website, 2025. URL: https://team.doubao.com/ en/special/doubao_1_5_pro, Accessed: 2025-05-14

    2025

  34. [42]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 12

    2023 arXiv

  35. [43]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186, 2025

    2025 arXiv

  36. [44]

    Solving olympiad geometry without human demonstrations

    Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024

    2024

  37. [45]

    Alphazero-like tree-search can guide large language model decoding and training

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, 2024

    2024

  38. [46]

    Pulsecheck457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models

    Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Pulsecheck457: A diagnostic benchmark for 6d spatial reasoning of large multimodal models. arXiv e-prints, pages arXiv–2502, 2025

    2025

  39. [47]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pages 396–416. Springer, 2024

    2024

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  41. [49]

    Boosting multimodal reasoning with mcts-automated structured thinking

    Jinyang Wu, Mingkuan Feng, Shuai Zhang, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. Boosting multimodal reasoning with mcts-automated structured thinking. arXiv preprint arXiv:2502.02339, 2025

    2025

  42. [50]

    Mind’s eye of llms: Visualization-of-thought elicits spatial reasoning in large language models

    Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of llms: Visualization-of-thought elicits spatial reasoning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  43. [51]

    Grok 3: Advancing real-time reasoning in ai

    xAI. Grok 3: Advancing real-time reasoning in ai. Tech Rep. 1, xAI, 2025

    2025

  44. [52]

    Geox: Geometric problem solving through unified formalized vision-language pre-training

    Renqiu Xia, Mingsheng Li, Hancheng Ye, Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng, Xinyu Cai, Xiangchao Yan, Bin Wang, Conghui He, Botian Shi, Tao Chen, Junchi Yan, and Bo Zhang. Geox: Geometric problem solving through unified formalized vision-language pre-training....

    2025

  45. [53]

    Vision language models for spreadsheet understanding: Challenges and opportunities

    Shiyu Xia, Junyu Xiong, Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Mengyu Zhou, Yeye He, Shi Han, and Dongmei Zhang. Vision language models for spreadsheet understanding: Challenges and opportunities. arXiv preprint arXiv:2405.16234, 2024

    2024 arXiv

  46. [54]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279, 2025

    2025 arXiv

  47. [55]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  48. [56]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  49. [57]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  50. [58]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 13

    2023

  51. [59]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  52. [60]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  53. [61]

    Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space

    Weichen Zhan, Zile Zhou, Zhiheng Zheng, Chen Gao, Jinqiang Cui, Yong Li, Xinlei Chen, and Xiao-Ping Zhang. Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space. arXiv preprint arXiv:2503.11094, 2025

    2025

  54. [62]

    –”. Layers are separated by colons. For example, {“Layer 1

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024. 14 A Appendix Outline In the appendix, we provide the follo...

    2024 arXiv

  55. [64]

    First, the model generates a reasoning process in the thinking space, outlining the problem- solving approach

  56. [65]

    Second, based on this reasoning, the model decides on the appropriate action to take

  57. [66]

    Third, it observes the result of the action and integrates this information into the context. This structured reasoning-action-observation cycle allows the model to handle complex problems more systematically, especially tasks that require multi-step reasoning and tool collabo...

  58. [2020]

    PC, Released June 7, 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.