Pith. sign in

REVIEW 5 major objections 5 minor 56 references

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Counting boxes without the parts: a spatial hierarchy exposes shortcut behavior in vision models

desk verdict Well-built spatial diagnostic whose headline 'shortcut' finding outruns the evidence from its own independent-query design. read the letter →

arxiv 2607.22864 v1 pith:YCICNK5S submitted 2026-07-24 cs.CV cs.AI

Patrick Rim , Tom Long , Ekta Prashnani , Ruth Rosenholtz , Ben Boudaoud , Peter Xenopoulos , Alex Wong , Joohwan Kim
show 1 more author
Jae-Hyun Jung
This is my paper · ORCID
classification cs.CVcs.AI
keywords spatialintelligencehierarchicalbenchmarkobjectcountingmultimodallargelanguagemodelschain-of-thoughtreinforcementlearningwithverifiablerewards3Dstructureocclusionreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current multimodal large language models do not solve spatial tasks like counting boxes in a stack the way humans do: they reach the right total without going through the lower-level perceptual and cognitive steps that produce the total. The authors introduce Spatial-IQ, a diagnostic framework that decomposes 3D object counting into nine sub-tasks arranged in a hierarchy, and show that every frontier model fails many of those sub-tasks even when its final count is correct. They then show that training a model on the hierarchy itself, by asking it to output the sub-task answers in order before the total, plus reinforcement learning on verifiable rewards, sharply improves both intermediate and final accuracy. If right, this means a spatial benchmark must report more than a final accuracy score, and that decomposing a skill into its human-developmental prerequisites is itself a useful training signal.

What carries the argument

The central machine is a pre-specified causal hierarchy over one target task: counting objects in a gravity-supported 3D stack. The hierarchy has nine sub-tasks (object categorization, cluster count, column count, layer count, visible object count, top layer, direct support, support column, hidden object count) joined by four testable relations: the Visible Count Hierarchy, the Hidden Count Hierarchy, the Internal Referential Chain (top layer to direct support to support column), and the Summation Mechanism (visible count + hidden count = total). The data are ~80,000 procedurally generated scenes in Isaac Sim, with per-task ground truth derived from voxel control; hidden objects are defined by a depth-buffer test that keeps an occluded object only if it supports a visible object. The paper evaluates models in three output formats and a human baseline, and the training phase uses the hierarchy as a chain-of-thought template plus RLVR (DAPO) with a gated integer-distance reward.

What would settle it

Train a model on total counts only, then compare its Visible × Hidden conditional matrix to a decomposition-trained model's matrix: if the endpoint-only model matches the decomposition-trained model's row and column pattern (highest in both-correct, lowest in both-wrong), the matrix signature would not diagnose shortcut behavior. Conversely, if scenes with zero hidden objects are solved by a model while matched hidden-heavy scenes fail, the hidden-inference step is load-bearing.

Watch

Extended reading notes

Core claim

The Spatial-IQ framework demonstrates that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure.

Load-bearing premise

The load-bearing premise is that the nine sub-tasks are not just useful labels but a causal dependency chain for object counting, so that a model which reaches the correct total without completing that chain is taking a shortcut rather than using another legitimate strategy.

Editorial extensions

If this is right

  • Benchmarks should report sub-task or component accuracy, not just final accuracy: two models with identical total counts can differ by 30+ points on the intended mechanism.
  • Chain-of-thought supervision over a spatial sub-task hierarchy plus RLVR moves a Qwen2.5-VL-32B from 2.9% to 62.6% Object Counting accuracy, closing much of the gap to the 82.1% human baseline on this benchmark.
  • A model trained on the final answer alone reaches the same total count as a decomposition-trained model at 7B scale, but collapses to zero on every hierarchy relation, so endpoint-only training teaches a shortcut that does not preserve the intended competence.
  • The trained model generalizes to unseen object categories with only a small drop, indicating the hierarchy encodes a general spatial competence rather than a memorized cube-specific answer.
  • The full hierarchical decomposition is specific to Object Counting; Mental Rotation behaves as an independent spatial probe that does not share the same sub-task chain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The observable signature of human-like spatial reasoning in this framework is the Summation Mechanism's matrix (both visible and hidden correct well above both wrong); a model that reproduces that matrix without training on the hierarchy would still satisfy the diagnostic even if it never explicitly sums its two sub-answers.
  • The results generalize to other spatially decomposable tasks (paper folding, water-level prediction, perspective taking) by instrumenting the same pattern: identify prerequisites, encode them as a chain of tasks, and reward correct intermediate steps, an extension the paper itself points to.
  • The hidden-object support rule encodes a modeling commitment: hidden objects are defined by a support-based occlusion convention rather than any occluded volume, so hidden count is a logical inference about gravity support rather than a purely perceptual occlusion fact, and is the most likely place a competing benchmark would depart.
  • A direct next test is whether the SFT-CoT/DAPO gains persist under viewpoint, layout, and texture perturbations beyond the categories already tested, since training used one object category and evaluations on others were held out.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Spatial-IQ introduces a hierarchical diagnostic benchmark for spatial intelligence in multimodal LLMs. The target task is object counting in procedurally generated stacks of 3D objects, decomposed into nine sub-tasks (object categorization, cluster/column/layer counts, visible count, top-layer/direct-support/support-column, hidden count) organized into pre-specified dependency relations, plus a mental-rotation probe. Roughly 80,000 scenes are generated in NVIDIA Isaac Sim with analytic per-task ground truth, evaluated in text, multiple-choice, and image-editing formats, and anchored by a human baseline. The paper reports that frontier models reach only 2–18% on the counting target while showing variable sub-task performance, that conditional analyses reveal target-correct responses occurring without correct visible/hidden counts, and that causal claims about 'shortcut behavior' are inferred from these dissociations. A training study on Qwen2.5-VL shows that chain-of-thought supervision over the hierarchy plus RLVR improves target accuracy and hierarchy consistency much more than plain SFT on the final integer.

Significance. If the claims hold, the paper would contribute a useful diagnostic instrument: a large, procedurally generated dataset with voxel-level ground truth, three response formats, a human baseline, and pre-specified hierarchy relations whose statistical treatment (McNemar tests, FDR correction, within-scene lifts) is careful and reproducible. The direct strengths are the analytic ground truths, the difficulty-control validation, the format-stable Internal Referential Chain, and the training result showing that decomposition-aware supervision outperforms final-answer-only SFT. However, the central headline—that models 'often succeed at the target task without succeeding on the lower-level sub-tasks'—is overstated relative to the raw accuracies, and the inference from independent sub-task probes to 'shortcut behavior' is an interpretive leap rather than a measured fact. The benchmark remains valuable as a diagnostic tool for dissociations under isolated probes, with the shortcut/causal language needing substantial revision or additional chained-query evidence.

major comments (5)
  1. [Abstract; §5, Fig. 7, Table 1] The headline claim that top-performing models "often succeed at the target task without succeeding on the lower-level sub-tasks" is not supported as stated. The best model reaches 17.7% on Object Counting (Table 1, Fig. 4), so target-task success is not frequent in absolute terms. The data do support a conditional claim: among the small set of target-correct responses, a substantial fraction occur with incorrect Visible or Hidden Object Count responses (e.g., for Gemini, about 56% of target-correct trials come from cells where Visible Object Count is wrong). As written, the absolute-frequency reading overstates model competence and should be reworded to "when models succeed at the target, they often fail the isolated sub-task probes" or to a conditional formulation.
  2. [§3, §5, Fig. 7] Because all sub-tasks and target tasks are queried independently of one another, the conditional analyses show only that a model that fails an isolated sub-task query can still answer the target query correctly. They do not demonstrate that the target computation bypassed the sub-task competence; a model could use an internal enumeration, 3D reconstruction, or perceptual-individuation route that is not captured by the wording or format of the isolated sub-task prompt. The label "shortcut behavior" in the abstract and Section 7 is therefore an interpretation rather than a measured fact. I recommend either adding a chained-query condition (feeding model- or ground-truth-generated sub-task answers into the target prompt) or replacing "shortcut behavior" with "dissociation under independent probes" throughout the paper.
  3. [§3, Fig. 3, Appendix C] The hierarchy is introduced as a "causal dependency graph" in which sub-tasks are prerequisites for the target task. This is a normative commitment drawn from developmental psychology, not an empirical fact about either humans or machines. The paper's own data—models passing the target while failing prerequisites—are equally compatible with the hierarchy being an arbitrary decomposition for these models. The authors should either test the causal reading directly (for example by manipulating sub-task competence through training or by fitting a hierarchical diagnostic model) or explicitly restrict their conclusions to the pre-specified ordering, presenting it as a hypothesis rather than a demonstrated causal chain.
  4. [Appendix A, Definition 6; Appendix D] The scene-validity convention removes any hidden object that does not support a visible object, so Hidden Object Count and the hidden branch of Object Counting concern only support-required occlusion. The abstract's general claim about counting "including those occluded from view" is broader than what the benchmark actually measures. This is a defensible design choice for unambiguous ground truth, but it should be stated prominently in the main text and abstract so readers do not generalize the results to arbitrary occlusion or to hidden objects that are not physically required for support.
  5. [§6, Appendix H.3] The training experiment demonstrates that CoT supervision over the benchmark's own ground-truth labels improves performance on that benchmark, but it is a single-family (Qwen2.5-VL), single-modality (text) demonstration. Because SFT-CoT and DAPO-tight are trained and evaluated on the same task distribution, the large gains on hierarchy columns may partly reflect learning the exact sub-task answer format rather than a genuinely general spatial competence. The out-of-distribution object-category result is encouraging, but it does not establish that the decomposition transfers to other spatial task families. I would ask the authors to label this as a demonstration for the Spatial-IQ task family rather than a general training principle, and to report the missing 4% of the 32B DAPO-tight evaluation more explicitly.
minor comments (5)
  1. [Figures 2, 4] The figure labels contain the artifact "VLA/uni00AD0" in the model ordering; this should be corrected to "VLA-0".
  2. [Appendix B, Table 6] Appendix B does not report the number of annotators or the number of human responses per task, and it is unclear whether the human row in Table 6 uses the same n=3000 scenes or a smaller human-only sample. Please state the human sample size explicitly and use it in the Wilson confidence intervals.
  3. [Appendix D] The difficulty-calibration procedure was tuned on pilot frontier models; this selection information is important for interpreting the benchmark and should be summarized in the main text near the dataset description rather than only in an appendix.
  4. [Appendix F.2] The explanation of Qwen's negative chance-adjusted Hidden Object Count artifact is asserted but not accompanied by the supporting data; please include the quantitative evidence or remove the claim.
  5. [Appendix H.2] The text refers to "accompanying configuration files" for hyperparameters, but these are not included in the manuscript; please provide them in the supplement so the training conditions are fully reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's ground truths are procedurally defined, the human baseline is external, and the training loop includes independent controls.

full rationale

The benchmark's target and sub-task ground truths are derived from voxel-level scene generation and a depth-buffer occlusion test, not from any fitted parameter or from the model outputs being explained, so there is no self-definitional or fitted-input-called-prediction reduction in the dataset construction. The 'Summation Mechanism' relation (Visible Object Count + Hidden Object Count = Object Counting) is indeed definitional in the ground truth, but the paper uses it as a conditional-consistency diagnostic on model responses rather than as a predicted quantity derived from fit, and the human baseline externally anchors the hierarchy through developmental psychology. The closest self-referential element is the training experiment, where the SFT-CoT trace is the same additive composition that the Summation Mechanism metric scores; however, the structure-level train/benchmark split, the SFT-plain control condition, the total-only reward in DAPO-tight, and the out-of-distribution object generalization results give the training claim independent empirical content. The 'shortcut behavior' label is an interpretive description of the observed target-success/sub-task-failure dissociation rather than a definitional equivalence, so it is a validity concern about inference from independent probes, not a circularity. No load-bearing self-citation or author-imported uniqueness theorem was found; the cited developmental and psychometric literature is external and used to motivate rather than to force the hierarchy. The paper's own stated limitations, such as single-model-family training and cross-modality operationalization differences, reduce external validity but are disclosed and do not constitute circular derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on a normative decomposition of spatial reasoning (the causal hierarchy) and a scene-validity convention for hidden objects, plus hand-chosen difficulty and reward weights. No new physical entities are introduced; the 'five core spatial capabilities' of Appendix C are a descriptive taxonomy, not an invented entity. The core reason these axioms matter is that the benchmark's explanatory power depends on the hierarchy being the right decomposition, and its ground truth depends on the hidden-object validity convention.

free parameters (2)
  • Object-count sampling distribution = peak at n=11, slow falloff through n=30, exponential decay beyond
    Tuned during piloting on frontier models so the evaluation set separates models rather than saturating; this is a fitted property of the benchmark construction (Appendix D).
  • RLVR reward weights = format weight 0.1, integer-distance weight 0.9, hard gate on answer tags
    Chosen by hand in Appendix H.2; the reported training gains depend on this weighting and would need a sensitivity analysis to be fully interpretable.
assumptions (3)
  • domain assumption The nine sub-tasks form a causal prerequisite graph for object counting.
    Stated in Section 3 as pre-specified relations inspired by Piaget-Inhelder; the paper does not prove causality, and its own model results violate the prerequisite ordering for machines. The human baseline is used as the reference hierarchy signature.
  • ad hoc to paper A valid structure may contain only hidden objects that support at least one visible object; other occluded blocks are removed from the scene.
    This convention in Appendix A (definition 6) and Appendix D makes hidden-object ground truth well-defined from a single image; it is a modeling choice, not an observable fact, and it defines the target answer for all models and humans.
  • domain assumption Human in-house annotator responses are a representative reference for human spatial hierarchy.
    Appendix B reports that annotators are NVIDIA employees paid by salary, without reporting sample size, demographics, or inter-annotator agreement; the human baseline is treated as the gold-standard hierarchical signature throughout Section 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests." pith.science (2026). https://pith.science/paper/YCICNK5S

@misc{pith2026260722864,
  author       = {Pith},
  title        = {Pith review of: Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCICNK5S}},
  note         = {Machine review of arXiv:2607.22864}
}
read the original abstract

Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.

Figures

Figures reproduced from arXiv: 2607.22864 by the authors.

Figure 1
Figure 1. Humans count stacked 3D objects by hierarchically decomposing the structure into columns, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. [Left] The nine sub-tasks (S1–S9) and two target tasks (T1, T2) of Spatial-IQ. [Right] An [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We pre-specify these four sub-task relations for hierarchy dependency analysis in Section 5. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Per-task accuracy, text free-response, ordered according to our task taxonomy (Figure 2). [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Wrong-answer preferences in the five-choice MCQ condition. Each column denotes a [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Object Counting accuracy as a function of five difficulty controls in the text modality. Total objects, hidden objects, layers, columns, and fill ratio all affect difficulty in the direction expected (more is harder, except fill ratio where denser is easier), for both …
Figure 7
Figure 7. Figure 7: Object Counting accuracy, conditioned on correct performance on the Visible Object Count (V) and Hidden Object Count (H) tasks (text free-response modality). Rows and columns correct (✓) or incorrect (×) performance on the corresponding sub-task; cell color and label i…
Figure 8
Figure 8. Figure 8: Examples of the five-choice MCQ option grids for two procedurally generated scenes. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Human-study interface for the text free-response modality. The annotator is shown the same [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Human-study interface for the multiple-choice modality, illustrated on a five-choice [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Human-scoring interface for the image-editing modality, shown on [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: The three developmental frameworks tagged on each task panel of Figure 2: Piaget’s four [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Cross-view dependency correlations. Naive [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Cross-view bundle versus Object Counting mechanism comparison. Aggregate within￾scene Pearson r between each task bundle and Object Counting (left: model-averaged; right: per￾model heatmap). The Mental Rotation bundle (90◦ + 180◦ , r ≈ 0.19) is roughly 2–3× weaker tha…
Figure 15
Figure 15. Figure 15: Chance-adjusted per-task accuracy under five-, four-, and three-choice MCQ for the [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Mixed-CQ wrong-answer preference heatmap. Each cell reports [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Image-output per-task accuracy for Gemini Flash Image, Qwen-Image-Edit, and [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Within-structure camera-parameter effects per task in text. Top: perspective deltas (strong [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Object Counting accuracy as a function of the five candidate difficulty controls in the image-output modality (Gemini Flash Image, Qwen-Image-Edit, HunyuanImage-Instruct), paralleling [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Object Counting accuracy by object type for the human baseline and all text models, with Wilson 95% confidence intervals. Object identity modulates accuracy but does not reverse the overall human-versus-model ordering; the human baseline remains above 78% on every obj…
Figure 21
Figure 21. Figure 21: Aggregated text-modality task performance by object type (left) and by [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: Composition diagnostics for the Summation Mechanism analysis in text. Per responder: [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: Conditional Object Counting accuracy under the Visible Object Count × Hidden Object Count → Object Counting mechanism in MCQ. Each panel is a single responder’s 2 × 2 joint, ordered left-to-right by raw Object Counting accuracy. Each cell reports both the con￾ditional…
Figure 24
Figure 24. Figure 24: Conditional Object Counting accuracy under the Visible Object Count × Hidden Object Count → Object Counting mechanism in image output for the three image-editing models. Headline Object Counting accuracy appears in the panel title; each cell reports the conditional ac…
Figure 25
Figure 25. Figure 25: Per-task accuracy across the trained-text intervention subset, redrawn in the same task [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]
Figure 26
Figure 26. Figure 26: Object Counting accuracy by object type for the trained checkpoints (DAPO-tight and SFT-CoT/SFT-plain at 7B and 32B), the two zero-shot Qwen2.5-VL backbones, and the human baseline, with 95% Wilson confidence intervals. Cube is the only object category seen during tra…
Figure 27
Figure 27. Figure 27: Conditional Object Counting accuracy under the Visible Object Count × Hidden Object Count → Object Counting mechanism for the trained-text intervention subset, in the same one-row 2 × 2 format as [PITH_FULL_IMAGE:figures/full_fig_p039_27.png]
Figure 28
Figure 28. Figure 28: Composition diagnostics for the trained-text mechanism analysis (companion to Figure 27). [PITH_FULL_IMAGE:figures/full_fig_p039_28.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 31 canonical work pages

  1. [1]

    Claude opus 4.6, 2026

    Anthropic. Claude opus 4.6, 2026. Large language model

    work page 2026

  2. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    arXiv 2025

  3. [3]

    Object permanence in 31/2-and 41/2-month-old infants.Developmental psychology, 23(5):655, 1987

    Renee Baillargeon. Object permanence in 31/2-and 41/2-month-old infants.Developmental psychology, 23(5):655, 1987

    work page 1987

  4. [4]

    Simulation as an engine of physical scene understanding.Proceedings of the national academy of sciences, 110(45):18327–18332, 2013

    Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding.Proceedings of the national academy of sciences, 110(45):18327–18332, 2013

    work page 2013

  5. [5]

    Hunyuanimage 3.0 technical report.arXiv preprint arXiv:2509.23951, 2025

    Siyu Cao, Hangting Chen, Peng Chen, Yiji Cheng, Yutao Cui, Xinchi Deng, Ying Dong, Kipper Gong, Tianpeng Gu, Xiusen Gu, et al. Hunyuanimage 3.0 technical report.arXiv preprint arXiv:2509.23951, 2025

    arXiv 2025

  6. [6]

    Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    arXiv 2026

  7. [7]

    The kaufman assessment battery for children—second edition and kabc-ii normative update.Contemporary intellectual assessment: Theories, tests, and issues, pages 333–359, 2018

    Lisa Whipple Drozdick, Jennie Kaufman Singer, Elizabeth O Lichtenberger, et al. The kaufman assessment battery for children—second edition and kabc-ii normative update.Contemporary intellectual assessment: Theories, tests, and issues, pages 333–359, 2018

    work page 2018

  8. [8]

    Worldscore: A unified evaluation benchmark for world generation

    Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. Worldscore: A unified evaluation benchmark for world generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 27713–27724, 2025

    2025

Show all 56 references
  1. [9]

    Counting stacked objects

    Corentin Dumery, Noa Etté, Aoxiang Fan, Ren Li, Jingyi Xu, Hieu Le, and Pascal Fua. Counting stacked objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19774–19783, 2025

    2025

  2. [10]

    Functional neuroanatomy of intuitive physical inference.Proceedings of the national academy of sciences, 113(34):E5072–E5081, 2016

    Jason Fischer, John G Mikhael, Joshua B Tenenbaum, and Nancy Kanwisher. Functional neuroanatomy of intuitive physical inference.Proceedings of the national academy of sciences, 113(34):E5072–E5081, 2016

    2016

  3. [11]

    Space-10: A comprehensive benchmark for multimodal large language models in compositional spatial intelligence.ArXiv, abs/2506.07966, 2025

    Ziyang Gong, Wenhao Li, Olivera Martínez Ma, Songyuan Li, Jiayi Ji, Xue Yang, Gen Luo, Junchi Yan, and Rongrong Ji. Space-10: A comprehensive benchmark for multimodal large language models in compositional spatial intelligence.ArXiv, abs/2506.07966, 2025

    2025

  4. [12]

    Gemini 3 flash image, 2026

    Google DeepMind. Gemini 3 flash image, 2026. Generative text-to-image model (Internal codename: Nano Banana 2)

    2026

  5. [13]

    Gemini 3.1 pro, 2026

    Google DeepMind. Gemini 3.1 pro, 2026. Multimodal large language model

    2026

  6. [14]

    Vla-0: Building state-of-the-art vlas with zero modification.arXiv preprint arXiv:2510.13054, 2025

    Ankit Goyal, Hugo Hadfield, Xuning Yang, Valts Blukis, and Fabio Ramos. Vla-0: Building state-of-the-art vlas with zero modification.arXiv preprint arXiv:2510.13054, 2025

    2025

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  8. [16]

    Egoexobench: A benchmark for first-and third-person view video understanding in mllms.arXiv preprint arXiv:2507.18342, 2025

    Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang. Egoexobench: A benchmark for first-and third-person view video understanding in mllms.arXiv preprint arXiv:2507.18342, 2025

    2025 arXiv

  9. [17]

    Kaufman assessment battery for children.Psychological Assessment, 1983

    Alan S Kaufman and Nadeen L Kaufman. Kaufman assessment battery for children.Psychological Assessment, 1983. 10

    1983

  10. [18]

    Vision as a user interface

    Jan Koenderink. Vision as a user interface. InHuman vision and electronic imaging XVI, volume 7865, pages 18–30. SPIE, 2011

    2011

  11. [19]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  12. [20]

    The attribute hierarchy method for cognitive assessment: A variation on tatsuoka’s rule-space approach.Journal of educational measurement, 41(3):205– 237, 2004

    Jacqueline P Leighton, Mark J Gierl, and Stephen M Hunka. The attribute hierarchy method for cognitive assessment: A variation on tatsuoka’s rule-space approach.Journal of educational measurement, 41(3):205– 237, 2004

    2004

  13. [21]

    Worldmodelbench: Judging video generation models as world models.arXiv preprint arXiv:2502.20694, 2025

    Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E Gonzalez, et al. Worldmodelbench: Judging video generation models as world models.arXiv preprint arXiv:2502.20694, 2025

    2025 arXiv

  14. [22]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe twelfth international conference on learning representations, 2023

    2023

  15. [23]

    Towards world simulator: Crafting physical commonsense-based benchmark for video generation.arXiv preprint arXiv:2410.05363, 2024

    Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation.arXiv preprint arXiv:2410.05363, 2024

    arXiv

  16. [24]

    Nvidia isaac sim, 2024

    NVIDIA. Nvidia isaac sim, 2024. Robotics simulation platform

    2024

  17. [25]

    Replicator tutorials — omniverse isaac sim 4.2.0 documentation

    NVIDIA. Replicator tutorials — omniverse isaac sim 4.2.0 documentation. https://docs.isaacsim. omniverse.nvidia.com/4.2.0/replicator_tutorials/index.html, 2024. Accessed: April 27, 2026

    2024

  18. [26]

    Routledge & Kegan Paul, London,

    Jean Piaget and Bärbel Inhelder.The Child’s Conception of Space. Routledge & Kegan Paul, London,

  19. [27]

    Basic Books, New York, 1969

    Jean Piaget and Bärbel Inhelder.The Psychology of the Child. Basic Books, New York, 1969

    1969

  20. [28]

    Capture: Evaluating spatial reasoning in vision language models via occluded object counting

    Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision language models via occluded object counting. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8001–8010, 2025

    2025

  21. [29]

    Worldsimbench: Towards video generation models as world simulators.arXiv preprint arXiv:2410.18072, 2024

    Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, et al. Worldsimbench: Towards video generation models as world simulators.arXiv preprint arXiv:2410.18072, 2024

    2024 arXiv

  22. [30]

    Sat: Dynamic spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 2024

    Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, et al. Sat: Dynamic spatial aptitude training for multimodal language models.arXiv preprint arXiv:2412.07755, 2024

    2024

  23. [31]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  24. [32]

    Shepard and Jacqueline Metzler

    Roger N. Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects.Science, 171(3972):701–703, 1971

    1971

  25. [33]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    2025 arXiv

  26. [34]

    Sources of uncertainty in intuitive physics.Topics in cognitive science, 5(1):185–199, 2013

    Kevin A Smith and Edward Vul. Sources of uncertainty in intuitive physics.Topics in cognitive science, 5(1):185–199, 2013

    2013

  27. [35]

    Origins of knowledge

    Elizabeth S Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychological review, 99(4):605, 1992

    1992

  28. [36]

    Mental rotation and orientation-dependence in shape recognition

    Michael J Tarr and Steven Pinker. Mental rotation and orientation-dependence in shape recognition. Cognitive psychology, 21(2):233–282, 1989

    1989

  29. [37]

    Rule space: An approach for dealing with misconceptions based on item response theory.Journal of educational measurement, pages 345–354, 1983

    Kikumi K Tatsuoka. Rule space: An approach for dealing with misconceptions based on item response theory.Journal of educational measurement, pages 345–354, 1983. 11

    1983

  30. [38]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  31. [39]

    Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies.Psychometrika, 79(2):317–339, 2014

    Jonathan Templin and Laine Bradshaw. Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies.Psychometrika, 79(2):317–339, 2014

    2014

  32. [40]

    Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025

    2025 arXiv

  33. [41]

    Spatialtree: How spatial abilities branch out in mllms

    Yuxi Xiao, Longfei Li, Shen Yan, Xinhang Liu, Sida Peng, Yunchao Wei, Xiaowei Zhou, and Bingyi Kang. Spatialtree: How spatial abilities branch out in mllms. InThe First Workshop on Efficient Spatial Reasoning, 2026

    2026

  34. [42]

    Spatialbench: Benchmarking multimodal large language models for spatial cognition.arXiv preprint arXiv:2511.21471, 2025

    Peiran Xu, Sudong Wang, Yao Zhu, Jianing Li, Gege Qi, and Yunjian Zhang. Spatialbench: Benchmarking multimodal large language models for spatial cognition.arXiv preprint arXiv:2511.21471, 2025

    2025 arXiv

  35. [43]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv e-prints, pages arXiv–2412, 2024

    2024

  36. [44]

    Thinking in space: How multimodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025

    2025

  37. [45]

    Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al. Mmsi-bench: A benchmark for multi-image spatial intelligence.arXiv preprint arXiv:2505.23764, 2025

    2025 arXiv

  38. [46]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  39. [47]

    3viewsense: Spatial and mental perspective reasoning from orthographic views in vision-language models.arXiv preprint arXiv:2603.07751, 2026

    Shaoxiong Zhan, Yanlin Lai, Zheng Liu, Hai Lin, Shen Li, Xiaodong Cai, Zijian Lin, Wen Huang, and Hai-Tao Zheng. 3viewsense: Spatial and mental perspective reasoning from orthographic views in vision-language models.arXiv preprint arXiv:2603.07751, 2026

    2026 arXiv

  40. [48]

    Glm-4.6, 2026

    Zhipu AI. Glm-4.6, 2026. Large language model. 12 A Prompts, Definitions, and Task Specifications This appendix reports the exact text of all prompts used in the Spatial-IQ benchmark. Every text free-response, multiple-choice, and image-editing query is composed by concatenati...

    2026

  41. [50]

    ACOLUMNis a vertical stack of one or more objects in which each object lies either on the ground or solely on top of the object immediately beneath it

  42. [51]

    The objects within the same layer could be laterally detached or attached

    ALAYERis a horizontal group of one or more objects at the same vertical height in the structure. The objects within the same layer could be laterally detached or attached. A single object also counts as a layer

  43. [52]

    AVISIBLE OBJECTis an object with at least one face fully or partially visible in the image

  44. [53]

    ACLUSTERis a group of one or more objects that are connected through direct contact, possibly across multiple layers. Two objects are considered connected if they are in direct physical contact (i.e., directly adjacent horizontally or vertically with no gap between them); diag...

  45. [54]

    A DIRECTLY SUPPORTING OBJECTis the supporting object in immediate contact beneath a given object

    ASUPPORTING OBJECTis any object that is beneath a given object in the same column. A DIRECTLY SUPPORTING OBJECTis the supporting object in immediate contact beneath a given object. If an object is directly supported by the ground, then it has no supporting objects

  46. [55]

    Which option (A–E) correctly depicts

    AHIDDEN OBJECTis an object with no visible faces in the image. Any hidden object must be a supporting object of at least one visible object. Otherwise, such an object does not exist in a valid structure. A.3 Task-Specific Queries The body of each prompt is the task-specific qu...

  47. [56]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  48. [1956]

    Translated by F. J. Langdon and J. L. Lunzer; original French edition 1948

    1948

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.