Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that current multimodal large language models cannot effectively perform multi-step visual simulation: they score close to random chance on cube net folding and tangram puzzles, and intermediate visual steps help only…

desk verdict A genuinely useful benchmark whose headline claim runs ahead of its own perception probes. read the letter →

arxiv 2506.04633 v1 pith:W4UCAIR2 submitted 2025-06-05 cs.CV

Linjie Li , Mahtab Bigverdi , Jiawei Gu , Zixian Ma , Yinuo Yang , Ziang Li , Yejin Choi , Ranjay Krishna This is my paper
classification cs.CV
keywords spatialcognitionvisualsimulationmultimodallargelanguagemodelsbenchmarkcubenetfoldingtangrampuzzlesmentalimageryreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STARE is a new benchmark of about 4,000 spatial tasks built to test whether multimodal AI models can do what humans do naturally: run a step-by-step mental picture of a transformation rather than just talking through it. The paper finds that models handle simple 2D transformations well but score close to random chance on cube net folding and tangram puzzles, which require several mental moves in 3D. Showing models intermediate visual steps helps on some tasks and hurts on others, and the best model tested still lands below 60% overall, while humans reach the mid-90s. The authors conclude that current models cannot effectively perform visual simulation, and they use the benchmark to locate the failure in 3D perception and multi-step integration rather than in basic color or flat-shape recognition.

What carries the argument

The carrying mechanism is STARE's programmatic generator, which synthesizes each task with an explicit ground-truth simulation: a folding algorithm that rotates cube-net faces 90 degrees about shared edges and checks for overlaps and disconnections, a recursive segmentation-and-scramble routine for tangram puzzles, and Matplotlib and Blender renderings of every intermediate state. That generator lets the authors produce matched without-visual-simulation and with-visual-simulation versions of the same item, turning the benchmark into a controlled intervention where the only difference between the two conditions is the intermediate imagery. The explicit step structure of each task is what makes the claim about visual simulation testable.

What would settle it

Run an independent audit of the generated labels: brute-force fold every cube net (or enumerate all 11 cube nets) and run an exact-cover solver on every tangram puzzle. If the mislabel rate is above a couple of percent, the near-chance model scores could reflect broken ground truth rather than missing simulation; if it is zero, the claim stands on the generated data.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art multimodal large language models do not perform multi-step visual simulation the way humans do. Across the benchmark's 2D and 3D transformations, cube net folding, tangram puzzles, temporal frame reasoning, and perspective reasoning, models stay within a few points of random chance on the multi-step spatial tasks, even when they are given explicit intermediate visualizations. The one systematic exception is straightforward 2D transformation, where accuracy reaches the 80-90% range. Human testers scored 87.5-99% on the same items, and their response times dropped by 7.5 seconds on average when intermediate steps were shown, evidence that the tasks genuinely are solved by running mental simulations. Because models improve inconsistently with those same visual aids, and because a probing test shows they fail specifically at judging whether a face has been folded into depth, the paper concludes the bottleneck is the capacity to simulate spatial change step by step, not the ability to see or describe the shapes.

Load-bearing premise

The load-bearing premise is that STARE's ground-truth labels are correct: the cube-net folding algorithm's overlap and disconnection checks and the tangram invalid-case construction must never mislabel a puzzle, since near-chance scores only mean 'no visual simulation' if the labels are right.

Editorial extensions

If this is right

  • Success on 2D transformation tasks should not be read as general spatial competence: the same models drop to near-chance on multi-step 3D tasks, so benchmarks that stop at 2D overstate ability.
  • Intermediate visual simulation is not a reliable assist: models that improve on some tasks and decline on others (GPT-4o and o1 on tangrams, Claude and Gemini Flash on cube nets) have not internalized the visual steps.
  • The strong correlation (r≈0.88) between synthetic-task performance and real-world task performance implies that gains on these abstract simulation tasks should transfer to practical settings such as navigation and assembly.
  • Because explicit verbal reasoning steps do not help cube net folding and actively hurt tangram performance, chain-of-thought prompting cannot substitute for visual simulation.
  • A specific deficit in 3D perception, with GPT-4o identifying a folded face at only 57.4% accuracy while being perfect on color, locates the bottleneck in depth-aware perception rather than in 2D vision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the deficit is architectural: fine-tune a model on pairs of initial and intermediate visual states, then measure cube-net accuracy held out; if it stays near chance, missing simulation is a hard architectural limit rather than an experience gap.
  • Because models can reach about 75% on tangram question-only items by comparing piece areas, the near-chance numbers on the full tangram set may understate the failure: on items where the area heuristic cannot work, models likely do even worse.
  • The human response-time data suggest a new evaluation signal: compare a model's time-to-answer with and without intermediate visuals; a model that answers about as fast without simulation is plausibly pattern-matching rather than simulating.
  • The same generator-controlled design of matched with and without intermediate states extends naturally to deformable bodies, articulated mechanisms, and physical-prediction tasks, where the intermediate states are equally well-defined.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces STARE, a benchmark of approximately 4,000 programmatically generated tasks spanning 2D and 3D geometric transformations, cube net folding, tangram puzzles, and real-world perspective and temporal reasoning. Ten multimodal large language models are evaluated in settings with and without intermediate visual simulations, alongside two human participants. The main findings are that models perform near chance on cube net folding and tangram puzzles, that gains from intermediate visual simulations are inconsistent across models and tasks, and that humans achieve near-perfect accuracy but take longer without visual guidance. The paper attributes these gaps to a lack of multi-step visual simulation ability in current MLLMs.

Significance. If the central claim holds, STARE would be a useful diagnostic for spatial reasoning in multimodal models, complementing existing verbal-reasoning benchmarks. The programmatic generation makes the benchmark reproducible and extensible, and the error analysis, including the 3D folded-face perception probe, is a valuable attempt to decompose model failures. However, the central claim as stated conflates perception with simulation: the paper's own probe shows GPT-4o at 57.4% on a 3D perception task, which undermines the interpretation that near-chance cube net scores reflect a pure simulation deficit. The human baseline is also too thin to support the strong comparative claims. With these issues addressed, the benchmark could be a solid contribution to evaluating spatial cognition in MLLMs.

major comments (4)
  1. [§3.3, Table 2, Q3; Abstract; §4] The paper's own perception probe shows GPT-4o at 57.4% on 'has face 6 been folded?', essentially chance, while 2D color and connectivity are near ceiling; providing the final folded form raises cube net accuracy to 100%. The text acknowledges that these perceptual errors explain the limited benefits from visual simulations, yet the Abstract and Conclusion state that 'models cannot effectively perform visual simulation.' This conflates a 3D perception deficit with a simulation deficit: a model that cannot perceive whether a face is folded has failed at perception, not necessarily at simulation. The central claim needs to be reframed, or supported by a condition that isolates simulation from perception (e.g., symbolic state descriptions or non-visual intermediate representations), before the paper can claim evidence about visual simulation ability.
  2. [§3.1; Table 1] The human baseline consists of two undergraduates, and no error bars, repeated runs, or per-subject variability are reported for either humans or models. The strong claims about human near-perfect accuracy and response-time reductions (e.g., 28.9s down to 17.1s on tangram puzzles) rest on an essentially anecdotal sample. Report at least per-subject scores and confidence intervals, or increase the number of participants, to make the human-model comparison statistically meaningful.
  3. [§2.2/E.4; §3.3 Q5, Table 4] The paper explicitly states that the tangram question-only set has a 'bias': models can achieve about 75% accuracy by comparing total piece areas. This means the 'without visual simulation' tangram condition in Table 1 is not a pure spatial-simulation test, and the near-chance F1 scores in that condition are not clean evidence about simulation ability. The dataset should be redesigned so that area-sum is non-diagnostic (e.g., equal-area solvable and unsolvable instances), or the analysis should exclude or condition on those instances.
  4. [§E.3, E.4; §2.2] The validity of the entire benchmark depends on programmatic generators: the cube net algorithm's overlap and disconnection checks and the tangram invalid-case construction must never mislabel a puzzle. The paper describes these checks but provides no formal verification, no independent audit, and no human-validation statistics on the generated labels. Given that near-chance scores could also arise from systematic mislabeling, the authors should report a human audit of a random sample (e.g., 100 instances per task) or otherwise verify label correctness.
minor comments (5)
  1. [Table 1] The header row splits 'Temp-oral' and 'Pers-pective' awkwardly; ensure the table is readable in the final version.
  2. [§3.2, Appendix G] The claim of r≈0.88 across 11 models with p≈5e-4 is fragile; with only 11 data points and a drop to r≈0.58 when open-source models are removed, the wording 'strong correlation' overstates the evidence. Hedge the claim or use rank correlation.
  3. [References] The reference list contains malformed entries (e.g., '[11] et al. Johnson, Justin' and several others with misplaced author names), which should be cleaned before publication.
  4. [§3.1, Evaluation Metrics] The paper uses accuracy for multiple-choice tasks and F1 for binary tasks, then macro-averages across tasks; since these metrics have different ranges and chance levels, the 'overall' column is hard to interpret. State this limitation explicitly.
  5. [§2.1, Table 6] The text describes the benchmark as containing ~4K instances, while Table 6 sums to 3,937; align the wording to avoid a minor inconsistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STARE is an empirical benchmark; model scores are measured against programmatically generated labels with no fitted parameters, and self-citations appear only in related work.

full rationale

The paper's central claim—that MLLMs score near chance on cube-net folding and tangram puzzles and benefit inconsistently from intermediate visual simulations—is an empirical measurement, not a derivation. No parameter is fitted to model outputs, no constant is derived from the target conclusion, and no result is defined in terms of the models being evaluated. The ground-truth labels for cube nets and tangrams are produced by the generators described in Appendix E.3/E.4, with near-perfect human performance (Table 1) serving as an independent check; a possible bug in the validity checks would be a correctness risk, not circular self-definition. The related-work section cites prior papers by overlapping authors (NaturalBench [26], BLINK [29], perception tokens [71]), but these citations merely contextualize the benchmark and are not used to justify STARE's scores or its validity. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The perception-versus-simulation confound raised by the paper's own 3D perception probe (Table 2) is an interpretive concern about what the benchmark isolates, not a circular reduction of the conclusion to the input. Accordingly, no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted: the benchmark is a measurement instrument, not a predictive model. The load-bearing axioms are correctness of the synthetic labels, the interpretation of tasks as visual simulation, the thin human baseline, and the quality of the real-world datasets. No new entities are postulated.

assumptions (4)
  • domain assumption The synthetic generators produce correct ground-truth labels: cube-net folding validity checks are sound, and tangram invalid puzzles are truly unsolvable.
    The entire benchmark's model scores depend on these labels; Section 2.2 describes the algorithms but provides no external or formal verification that the validity checks never mislabel.
  • domain assumption The tasks are best solved by multi-step visual simulation, so performance measures spatial simulation rather than perception or language priors.
    Introduction uses cognitive science (Shepard and Metzler, Hegarty) to motivate this, but models might fail for perception reasons; Q3 shows perception errors contribute.
  • domain assumption Two undergraduate participants provide a reliable human ceiling.
    Section 3.1 reports averaged performance over two students, which is a thin sample for near-100% claims.
  • domain assumption Objectron and HM3D provide sufficiently unambiguous temporal and perspective reasoning stimuli.
    Temporal frames use Objectron and perspective uses HM3D plus human filtering; ambiguous cases may remain despite filtering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations." pith.science (2026). https://pith.science/paper/W4UCAIR2

@misc{pith2026250604633,
  author       = {Pith},
  title        = {Pith review of: Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4UCAIR2}},
  note         = {Machine review of arXiv:2506.04633}
}
read the original abstract

Spatial cognition is essential for human intelligence, enabling problem-solving through visual simulations rather than solely relying on verbal reasoning. However, existing AI benchmarks primarily assess verbal reasoning, neglecting the complexities of non-verbal, multi-step visual simulation. We introduce STARE(Spatial Transformations and Reasoning Evaluation), a benchmark designed to rigorously evaluate multimodal large language models on tasks better solved through multi-step visual simulation. STARE features 4K tasks spanning foundational geometric transformations (2D and 3D), integrated spatial reasoning (cube net folding and tangram puzzles), and real-world spatial reasoning (perspective and temporal reasoning), reflecting practical cognitive challenges like object assembly, mechanical diagram interpretation, and everyday spatial navigation. Our evaluations show that models excel at reasoning over simpler 2D transformations, but perform close to random chance on more complex tasks like 3D cube net folding and tangram puzzles that require multi-step visual simulations. Humans achieve near-perfect accuracy but take considerable time (up to 28.9s) on complex tasks, significantly speeding up (down by 7.5 seconds on average) with intermediate visual simulations. In contrast, models exhibit inconsistent performance gains from visual simulations, improving on most tasks but declining in specific cases like tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0 Flash), indicating that models may not know how to effectively leverage intermediate visual information.

Figures

Figures reproduced from arXiv: 2506.04633 by the authors.

Figure 1
Figure 1. Visual simulations play a crucial role in real-world tasks, from assembling complex structures to interpreting mechanical diagrams and predicting spatial interactions. Different from how humans would ap￾proach a cube net folding problem, existing multimodal models rely heavily on textual simulation, which is not sufficient for reaching human-level spatial cognition. The above example shows how textual simulations of… view at source ↗
Figure 2
Figure 2. Overview of STARE. STARE consists of 3 levels of tasks, 2D Transformation and 3D Transfor￾mation for foundational spatial reasoning skills, tangram puzzle and cube net folding for integrated spatial reasoning, temporal frame inference and perspective reasoning to mimic real-world scenarios. The interme￾diate steps for completing tasks in the first two levels can be explicitly simulated, while the more real-word spat… view at source ↗
Figure 3
Figure 3. The different variants in the Tangram Puzzle task. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: GPT-4o performance on individual 2D/3D trans￾formation types, with and without Visual Simulation (Vis￾Sim). Question: Check out a net with 6 square faces. Can the net be folded to form a cube, yes or no? Perception Error Error Analysis: Claude: Looking at the image, I …
Figure 6
Figure 6. Figure 6: GPT-4o performance vs. task complexity (left: difficulty levels and right: number of transformation steps) with or without Visual Simulation (VSim). Q3: Do model failures originate from basic visual perception errors? To determine if model fail￾ures originate from fund…
Figure 7
Figure 7. Figure 7: Data Statistics of STARE. Below, we summarize the design space of data curation for synthetic tasks, including (1) 2D Trans￾formations (§E.1); (2) 3D Transformations (§E.2); (3) Cube Net Folding (§E.3); and (4) Tangram Puzzles (§E.4); E.1 2D Transformations Shape gener…
Figure 8
Figure 8. Figure 8: Design space of 2D Transformations (1). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Design space of 2D Transformations (2). E.2 3D Transformations Shape generation. 3D objects are loaded from external blend files and instantiated with random properties defined in a JSON file. Their attributes include: • Types: Various 3D models such as cube, sphere, c…
Figure 10
Figure 10. Figure 10: Examples of Tangram Puzzle under “without Visual Simulations" Evaluation Setting (top: question￾only, bottom: question+assembly steps) [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Example of Tangram Puzzle under “with Visual Simulations" Evaluation Setting. You see 4 sequential frames of a video, but one is missing (marked with "?'). Choose which of the images in the second row correctly fills the missing frame. Remember, the camera only moves …
Figure 12
Figure 12. Figure 12: Examples of Temporal Frame Reasoning under “without Visual Simulations" Evaluation Setting. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Exemplary questions on cube nets to probe model performance on 2D and 3D perception. F.4 Visualizations of STARE Task in Different Representations Figures 14–17 provide concrete examples of the input modalities evaluated in STARE. For every task family we visualize th…
Figure 14
Figure 14. Figure 14: Visualizations of 2D transformations (w/ text instructions) in different representations (upper left: image-only, lower left: text-only, right: image+text). Image-only Imagine transforming a pyramid step by step. Follow these changes: • Move the object up After these …
Figure 15
Figure 15. Figure 15: Visualizations of 3D transformations (w/ text instructions) in different representations (upper left: image-only, lower left: text-only, right: image+text). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Visualizations of cube net folding in different representations (upper left: image-only, lower left: text-only, right: image+text). Image-only Te x t-only Tangram-Style Text Puzzle Can you use all of the following pieces—rotating or flipping them as needed—to exactly …
Figure 17
Figure 17. Figure 17: Visualizations of tangram in different representations (left: image-only, right: text-only). F.5 Case Study [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Error cases of o1 and GPT-4o on 2D transformation [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]
Figure 19
Figure 19. Figure 19: presents error cases on 3D transformation tasks from o1 and GPT-4o. The o1 response accurately places the torus up and left of the origin and selects panel A, but it overlooks that mate￾rial/appearance differences could also rule out that choice. GPT-4o correctly sees…
Figure 20
Figure 20. Figure 20: A error case from GPT-4o on cube net folding, where the model fails to understand the intermediate visual simulation. For tangram puzzles, the left example of [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Left: A perception error case from Claude on tangram puzzle. Right: An error case from Claude on tangram puzzle, which failed to simulate the intermediate steps even when step-by-step instructions are given [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Error case on temporal frame reasoning [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: Error case on perspective reasoning. G Detailed Analysis Results Correlation Analysis between Synthetic tasks and Real tasks. In Section 3.2, we briefly dis￾cussed the correlation between averaged model performance on synthetic tasks (including 2D trans￾formation, 3D …
Figure 24
Figure 24. Figure 24: Correlation between model performance on synthetic tasks and that on real-world tasks. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumanCLAW: Can Vision-Language Models Act Through a Body?

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Off-the-shelf VLMs fail closed-loop whole-body find-navigate-interact tasks (best 16.8%) because they lack embodied self-awareness, not target recognition.

  2. MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Visual thoughts — latent tokens, interleaved images, or video rollouts — do not currently improve multi-step reasoning over text-only baselines in frontier models.

Reference graph

Works this paper leans on

72 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    Thinking with sketches

    Barbara Tversky and Masaki Suwa. Thinking with sketches. 2009. 1

    work page 2009

  2. [2]

    When it all falls down: The relationship between intuitive physics and spatial cognition.Cognitive research: principles and implications, 5:1–13, 2020

    Alex Mitko and Jason Fischer. When it all falls down: The relationship between intuitive physics and spatial cognition.Cognitive research: principles and implications, 5:1–13, 2020. 1

    work page 2020

  3. [3]

    Pip: Physical interaction pre- diction via mental simulation with span selection

    Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan. Pip: Physical interaction pre- diction via mental simulation with span selection. InEuropean Conference on Computer Vision, pages 405–421. Springer, 2022

    work page 2022

  4. [4]

    Spatial ability for stem domains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance.Journal of Educational Psy- chology, 101(4):817, 2009

    Jonathan Wai, David Lubinski, and Camilla P Benbow. Spatial ability for stem domains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance.Journal of Educational Psy- chology, 101(4):817, 2009

    work page 2009

  5. [5]

    Simulation as an engine of physical scene understanding.Proceedings of the National Academy of Sciences, 110(45):18327–18332

    Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding.Proceedings of the National Academy of Sciences, 110(45):18327–18332. 1

    work page

  6. [6]

    Mental rotation of three-dimensional objects.Science, 171 (3972):701–703, 1971

    Roger N Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects.Science, 171 (3972):701–703, 1971. 1, D

    work page 1971

  7. [7]

    Mental animation: Inferring motion from static displays of mechanical systems.Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(5):1084–1102, 1992

    Mary Hegarty. Mental animation: Inferring motion from static displays of mechanical systems.Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(5):1084–1102, 1992. 1

    work page 1992

  8. [8]

    Training spatial cognition enhances mathematical learning in a randomized study of 17,000 children.Nature Human Behaviour, 5(11):1548–1554, 2021

    Nicholas Judd and Torkel Klingberg. Training spatial cognition enhances mathematical learning in a randomized study of 17,000 children.Nature Human Behaviour, 5(11):1548–1554, 2021. 1

    work page 2021

Show all 72 references
  1. [9]

    Bo T Christensen and Christian D Schunn. The role and impact of mental simulation in design.Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 23(3):327–344, 2009

    2009

  2. [10]

    Mechanical reasoning by mental simulation.Trends in Cognitive Sciences, 8(6):280– 285, 2004

    Mary Hegarty. Mechanical reasoning by mental simulation.Trends in Cognitive Sciences, 8(6):280– 285, 2004. ISSN 1364-6613. doi: https://doi.org/10.1016/j.tics.2004.04.001. URLhttps://www. sciencedirect.com/science/article/pii/S1364661304001007. 1

    2004 doi

  3. [11]

    Johnson, Justin

    et al. Johnson, Justin. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910, 2017. 1, D

    2017

  4. [12]

    C. et al. Zhang. Raven: A dataset for relational and analogical visual reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5317–5327, 2019. 2, D

    2019

  5. [13]

    et al. Ji, W. Abstract visual reasoning with tangram shapes. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2350–2360, 2022

    2022

  6. [14]

    Space: A simulator for physical interactions and causal learning in 3d environments

    Jiafei Duan, Samson Yu, and Cheston Tan. Space: A simulator for physical interactions and causal learning in 3d environments. InProceedings of the ieee/cvf international conference on computer vision, pages 2058–2063, 2021

    2021

  7. [15]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    1911 arXiv

  8. [16]

    S. K. Ramakrishnan, E. Wijmans, P. Krahenbuhl, and V . Koltun. Does spatial cognition emerge in frontier models?arXiv preprint arXiv:2410.06468, 2024. 1, 2, D

    2024 arXiv

  9. [18]

    Spatial perspective taking: Effects of social, directional, and interactive cues.Memory & cognition, 47:1031–1043, 2019

    Peri Gunalp, Tara Moossaian, and Mary Hegarty. Spatial perspective taking: Effects of social, directional, and interactive cues.Memory & cognition, 47:1031–1043, 2019

    2019

  10. [19]

    Shepard and Christine Feng

    Roger N. Shepard and Christine Feng. A chronometric study of mental paper folding.Cognitive Psychol- ogy, 3(2):228–243, 1972. ISSN 0010-0285. doi: https://doi.org/10.1016/0010-0285(72)90005-9. URL https://www.sciencedirect.com/science/article/pii/0010028572900059

    1972

  11. [20]

    Identifying cognitive processes and neural substrates of spatial transformation in a mental folding task with cognitive modeling

    Kai Preuss, Christopher Hilton, Klaus Gramann, and Nele Russwinkel. Identifying cognitive processes and neural substrates of spatial transformation in a mental folding task with cognitive modeling. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 4...

    2024

  12. [21]

    Tangram solved? prefrontal cortex activation analysis during geometric problem solving

    Hasan Ayaz, Patricia A Shewokis, Meltem ˙Izzeto˘glu, Murat P Çakır, and Banu Onaral. Tangram solved? prefrontal cortex activation analysis during geometric problem solving. In2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 47...

    2012

  13. [22]

    Smith, Elizabeth Bonawitz, and Tomer D

    Ilona Bass, Kevin A. Smith, Elizabeth Bonawitz, and Tomer D. Ullman. Partial mental simulation explains fallacies in physical reasoning.Cognitive Neuropsychology, 2022. 1, D

    2022

  14. [23]

    Allen, Samuel J

    Tony Chen, Kelsey R. Allen, Samuel J. Cheyette, Joshua B. Tenenbaum, and Kevin A. Smith. ¨Just in time¨representations for mental simulation in intuitive physics. InProceedings of the 45th Annual Meeting of the Cognitive Science Society (CogSci), 2023. 1, D

    2023

  15. [24]

    IsoBench: Benchmarking multimodal foundation models on isomorphic representations,

    Deqing Fu ∗, Ghazal Khalighinejad∗, Ollie Liu∗, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomorphic representations,

  16. [25]

    Inter- gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter- gps: Interpretable geometry problem solving with formal language and symbolic reasoning. InThe 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021

    2021

  17. [26]

    Naturalbench: Evaluating vision-language models on natural adversarial samples.European Conference on Computer Vision, 2024

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples.European Conference on Computer Vision, 2024. 2

    2024

  18. [27]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms.arXiv preprint arXiv:2401.06209, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms.arXiv preprint arXiv:2401.06209, 2024. 2

    2024 arXiv

  19. [28]

    V*: Guided visual search as a core mechanism in multimodal llms.ArXiv, abs/2312.14135, 2023

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms.ArXiv, abs/2312.14135, 2023. URLhttps://api.semanticscholar.org/CorpusID:266436019. D

    2023 arXiv

  20. [29]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390, 2024. 2, 3.2, D

    2024 arXiv

  21. [30]

    Kiva: Kid-inspired visual analogies for testing large multimodal models.arXiv preprint arXiv:2407.17773, 2024

    Eunice Yiu, Maan Qraitem, Charlie Wong, Anisa Noor Majhi, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal models.arXiv preprint arXiv:2407.17773, 2024. 2, D

    2024

  22. [31]

    Stratified rule-aware network for abstract visual reasoning

    Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1567–1574, 2021. 2, D

    2021

  23. [32]

    Turtlebench: A visual pro- gramming benchmark in turtle geometry.arXiv preprint arXiv:2411.00264, 2024

    Sina Rismanchian, Yasaman Razeghi, Sameer Singh, and Shayan Doroudi. Turtlebench: A visual pro- gramming benchmark in turtle geometry.arXiv preprint arXiv:2411.00264, 2024. 2, D

    2024 arXiv

  24. [33]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces.arXiv preprint arXiv:2412.14171, 2024. 2, D

    2024 arXiv

  25. [34]

    Matplotlib: Visualization with python.https://matplotlib.org/, 2012

    Matplotlib. Matplotlib: Visualization with python.https://matplotlib.org/, 2012. 2.2

    2012

  26. [35]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 2.2

    2017

  27. [36]

    Blender is free software.https://www.blender.org/

    Blender. Blender is free software.https://www.blender.org/. 2.2

  28. [37]

    Objectron: A large scale dataset of object-centric videos in the wild with pose annotations.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021

    Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021. 2.2

    2021

  29. [38]

    Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environ- ments for embodied AI

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale ...

    2021 arXiv

  30. [39]

    Habitat: A Platform for Embodied AI Research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. InProceedings of the IEEE/CVF International Conferenc...

    2019

  31. [40]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir V ondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, J...

    2021

  32. [41]

    Habitat 3.0: A co-habitat for humans, avatars and robots, 2023

    Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang, Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimír V ondruš, Vincent- Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishna...

    2023

  33. [42]

    Hello gpt-4o.https://openai.com/index/hello-gpt-4o/

    OpenAI. Hello gpt-4o.https://openai.com/index/hello-gpt-4o/. 3.1

  34. [43]

    Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet

    Anthropic. Claude 3.5 sonnet.https://www.anthropic.com/news/claude-3-5-sonnet. 3.1

  35. [44]

    Introducing gemini 2.0: our new ai model for the agentic era.https://blog

    Google Deepmind. Introducing gemini 2.0: our new ai model for the agentic era.https://blog. google/technology/google-deepmind/google-gemini-ai-update-december-2024//, . 3.1

    2024

  36. [45]

    Gemini 2.0 flash thinking mode.https://ai.google.dev/gemini-api/docs/ thinking-mode,

    Google Deepmind. Gemini 2.0 flash thinking mode.https://ai.google.dev/gemini-api/docs/ thinking-mode, . 3.1

  37. [46]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024 arXiv

  38. [47]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024. 3.1 12

    2024 arXiv

  39. [48]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

    arXiv

  40. [49]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  41. [50]

    Structure-mapping: A theoretical framework for analogy.Cognitive Science, 7(2):155– 170, 1983

    Dedre Gentner. Structure-mapping: A theoretical framework for analogy.Cognitive Science, 7(2):155– 170, 1983. D

    1983

  42. [51]

    Carpenter, M.A

    P.A. Carpenter, M.A. Just, and P. Shell. What one intelligence test measures: A theoretical account of the processing in the raven progressive matrices test.Psychological Review, 97(3):404–431, 1990. D

    1990

  43. [52]

    Lovett and K

    A. Lovett and K. Forbus. Modeling visual problem solving as analogical reasoning.Psychological Review, 124(1):60–90, 2017. D

    2017

  44. [53]

    Holyoak, Alan Yuille, and Hongjing Lu

    Nicholas Ichien, Qing Liu, Shuhao Fu, Keith J. Holyoak, Alan Yuille, and Hongjing Lu. Visual analogy: Deep learning versus compositional models. InProceedings of the 43rd Annual Meeting of the Cognitive Science Society (CogSci), 2021. D

    2021

  45. [54]

    Webb, Shuhao Fu, Trevor Bihl, Keith J

    Taylor W. Webb, Shuhao Fu, Trevor Bihl, Keith J. Holyoak, and Hongjing Lu. Zero-shot visual reasoning through probabilistic analogical mapping.arXiv preprint arXiv:2209.15087, 2022

    2022 arXiv

  46. [55]

    Ichien, Q

    N. Ichien, Q. Liu, S. Fu, K.J. Holyoak, A. Yuille, and H. Lu. Two computational approaches to visual analogy: Task-specific models versus domain-general mapping.Cognitive Science, 47(4):e13347, 2023. D

    2023

  47. [56]

    M. Hegarty. Mechanical reasoning by mental simulation.Trends in Cognitive Sciences, 8(6):280–285,

  48. [57]

    Barsalou

    L.W. Barsalou. Grounded cognition.Annual Review of Psychology, 59:617–645, 2008. D

    2008

  49. [58]

    Battaglia, J.B

    P.W. Battaglia, J.B. Hamrick, and J.B. Tenenbaum. Simulation as an engine of physical scene understand- ing.Proceedings of the National Academy of Sciences, 110(45):18327–18332, 2013. D

    2013

  50. [59]

    Tenenbaum, T.L

    J.B. Tenenbaum, T.L. Griffiths, and C. Kemp. Theory-based bayesian models of inductive learning and reasoning.Trends in Cognitive Sciences, 10(7):309–318, 2006

    2006

  51. [60]

    Ullman, E.S

    T.D. Ullman, E.S. Spelke, P. Battaglia, and J.B. Tenenbaum. Mind games: Game engines as an architec- ture for intuitive physics.Trends in Cognitive Sciences, 21(9):649–665, 2017. D

    2017

  52. [61]

    Neural prediction errors enable analogical visual reasoning in human standard intelligence tests

    Lingxiao Yang, Hongzhi You, Zonglei Zhen, Dahui Wang, Xiaohong Wan, Xiaohua Xie, and Ru-Yuan Zhang. Neural prediction errors enable analogical visual reasoning in human standard intelligence tests. InProceedings of the 40th International Conference on Machine Learning (ICML), 2023. D

    2023

  53. [62]

    Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick

    Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology.Nature Human Behaviour, 6(9):1257–1267,

  54. [63]

    Bear, Elias Wang, Damian Mrowca, Felix J

    Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu F. Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical predicti...

    2022 arXiv

  55. [64]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. InICCV, 2015. D

    2015

  56. [65]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502, 2023

    Xiyao Yue, Yifan Ni, Kai Zhang, Tao Zheng, Ruixuan Liu, Wen Chen, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.16502, 2023. D

    2023 arXiv

  57. [66]

    Wang, Robert Gerbicz, John-Clark Levin, Serguei Popov, Fiona 13 Feng, Steven Y

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Tung Nguyen, Daro...

    2025 arXiv

  58. [67]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2024

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2024. D

    2024 arXiv

  59. [68]

    Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing

    Wenxuan Zhang, Sharifah M. Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2023. D

    2023

  60. [69]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. InarXiv preprint arXiv:2306.13394, 2023. D

    2023 arXiv

  61. [70]

    MMBench: Is your multi-modal model an all-around player? InProceedings of the European Conference on Computer Vision (ECCV), 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? InProceedings of the European Conference on Computer Vision (ECCV), 2024. D 15

    2024

  62. [71]

    Perception tokens enhance visual reasoning in multimodal language models.arXiv preprint arXiv:2412.03548, 2024

    Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ran- jay Krishna. Perception tokens enhance visual reasoning in multimodal language models.arXiv preprint arXiv:2412.03548, 2024. D

    2024 arXiv

  63. [72]

    Gon- zalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

  64. [73]

    just in time,

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Ma...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.