Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models, tested on a new 23-dimension atomic benchmark, score near chance on motion trajectory, spatial reconstruction, temporal extension, and compositional physics, and so lack robust internal world models.

desk verdict Useful diagnostic benchmark with solid synthetic-world evidence of VLM blind spots, but the real-world recast in §5.4 is too thin to carry the generalization claim. read the letter →

arxiv 2506.21876 v1 pith:D5G4CPAY submitted 2025-06-27 cs.CL cs.AIcs.CV

Qiyue Gao , Xinyu Pi , Kevin Liu , Junrong Chen , Ruolan Yang , Xinqi Huang , Xinyu Fang , Lu Sun
show 16 more authors
Gautham Kishore Bo Ai Stone Tao Mengyang Liu Jiaxi Yang Chao-Jung Lai Chuanyang Jin Jiannan Xiang Benhao Huang Zeming Chen David Danks Hao Su Tianmin Shu Ziqiao Ma Lianhui Qin Zhiting Hu
This is my paper
classification cs.CLcs.AIcs.CV
keywords worldmodelsvision-languagebenchmarkevaluationintuitivephysicsspatialreasoningtemporalperceptionmotiontrajectorycounterfactualgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether vision-language models (VLMs) carry a genuine internal world model, and answers with a systematic, atomic test. The authors propose a two-stage framework, perception then prediction, and build WM-ABench: 23 fine-grained dimensions, over 100,000 instances from six simulators, with counterfactual wrong answers so models cannot win by visual shortcuts. Across 660 experiments on 15 commercial and open-source VLMs, they report that almost all models are near random when judging motion trajectories, weak on spatial and temporal judgment, poor at predicting collisions and multi-step changes, and entangled in how they represent attributes, with some models treating blue objects as faster than green ones. If the benchmark is a fair measure, current VLMs lack robust, independent internal world models, and the paper supplies a checklist for where to improve them.

What carries the argument

The load-bearing object is WM-ABench itself, a benchmark that decomposes world modeling into 23 atomic tasks organized by a two-stage framework: perception (visual, spatial, temporal, quantitative, motion) and prediction (mechanistic simulation, transitive inference, compositional inference). The key mechanism that makes the scores interpretable is counterfactual option generation: every question's wrong answers are produced either by perturbing the action while fixing the true previous state, or by perturbing the previous state while fixing the action, so the false states look visually similar to the true next state. That controls spurious cues and forces a model to simulate world dynamics rather than match surface features. Hard negative generation plus single-variable control is what lets the benchmark attribute failures to atomic dimensions, and a standardized relative entanglement score measures how much changing one attribute, such as color, shifts performance on an unrelated task.

What would settle it

Render the same controlled physical events and counterfactual wrong answers with photorealistic images, or with real video that has exact ground truth, and rerun the trajectory, temporal-extension, and collision tasks. If models jump to human-level scores while the physical content is unchanged, the near-random results are a rendering artifact; if they stay near random, the missing-world-model claim holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art VLMs do not have robust internal world models in the mechanistic sense: they can name shapes and colors and count objects, but they largely fail when asked to reconstruct 3D relations from multiple views, compare durations or speeds across episodes, trace a moving object's trajectory, or predict the next state after physical events and agent actions. The strongest evidence is the near-random motion-trajectory results, and the entanglement analysis showing that performance shifts when irrelevant attributes like color change. The authors also show the deficit is not only perceptual: filtering to cases where models answered all perceptual verification questions correctly, next-state prediction for slide and drop improves only slightly, or decreases for collide, implying missing physical knowledge rather than mere perception errors. A real-world recast of eight dimensions replicates the main pattern, giving the authors reason to think the simulator-based findings generalize.

Load-bearing premise

The conclusions assume that answering these simulator-rendered, counterfactual questions fairly measures the world-modeling ability that would matter with real-world images; the paper itself cautions that the synthetic images are not realistic, so if models fail mainly because the visuals are unfamiliar, the near-random scores would overstate the real-world gap.

Editorial extensions

If this is right

  • Current VLMs should not be trusted as general-purpose world simulators in robotics, driving, or other planning systems that depend on tracking trajectories and predicting collisions, since motion-trajectory accuracy near random is a direct warning.
  • Improving perception alone will not be enough: once the authors filter to cases where models had correct state perception, next-state prediction for slide and drop barely improves and collision prediction worsens, pointing to missing physical knowledge.
  • Static perception strengths are not a proxy for world modeling: state-of-the-art models are near-perfect on color, shape, and counting while failing spatial positioning, temporal extension, and compositional prediction.
  • Frontier models close some gaps, such as near-human size comparison and driving navigation, but not others such as multi-view 3D reconstruction, temporal extension, and multi-step manipulation, so the deficits are selective rather than a single general limitation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that WM-ABench's single-variable perturbation design could double as training data: fine-tuning on the hard negatives might teach models to stop using color as a proxy for speed, a failure mode the entanglement analysis exposes.
  • A reader wanting to locate the bottleneck could run the same questions with text-only state descriptions; if scores stay equally low, the missing ability is in language-grounded world knowledge, whereas if scores collapse further, vision grounding is the main culprit.
  • The relative entanglement metric could be ported to other VLM benchmarks: computing how much color, position, or material shifts reported accuracy on unrelated tasks would turn any multiple-choice benchmark into a check for shortcut reliance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a two-stage conceptual framework, decomposing world modeling into perception (visual, spatial, temporal, quantitative, motion) and prediction (mechanistic simulation, transitive inference, compositional inference), and introduces WM-ABench, a large simulator-generated benchmark with 23 fine-grained dimensions, controlled counterfactual options, and human baselines. The authors evaluate 15 VLMs in 660 experiments, reporting that models are strong at some static visual tasks but near-random on motion trajectory and several prediction tasks, and that their representations entangle orthogonal attributes such as color and speed. A small real-world recast of 8 dimensions is used to argue that simulation results transfer.

Significance. If the benchmark and transfer claim hold, this is a useful diagnostic contribution: the controlled counterfactual design, the use of simulator ground truth, and the inclusion of human baselines give the central negative findings about synthetic-scene performance a solid evidentiary basis. The paper is also commendably concrete about its limitations, including the realism gap of synthetic images. However, the headline conclusions in the abstract and conclusion are stated about VLMs' 'basic world modeling abilities' in general, and the only direct evidence that the simulator results transfer to real-world settings is a small, underpowered recast in Section 5.4. The benchmark itself appears to be a genuine resource, and the framework is clearly presented, but the generalizability claims need stronger support or explicit tempering.

major comments (3)
  1. [Section 5.4 / Table 4 / Limitations] The real-world validation is underpowered for the strong conclusion drawn from it. Table 4 reports approximately 50 data points per dimension and no confidence intervals; for example, Motion Trajectory accuracies range from 23.3% to 40.0% against a 25% random baseline, and with n=50 the 95% binomial confidence interval half-width is roughly 14 percentage points, making several of these values statistically indistinguishable from chance. Spatial Positioning values (26.7–53.3%) are similarly wide. The text then concludes that 'bias from our use of simulation data is likely to be minimal,' but no significance tests, confidence intervals, or explicit alignment criteria are provided. Since the Limitations section concedes that the synthetic images 'do not look realistic' and may be out-of-distribution, the abstract and conclusion should either restrict their claims to simulator-rendered tasks or be accompanied by a statistically adequate real-world validation.
  2. [Section 4.4 / Tables 1 and 2] Frontier-model results are based on only 100 instances per subtask and are reported without confidence intervals or significance tests. For instance, OpenAI o3 achieves 48% on Motion Trajectory in Table 1, but with n=100 the standard error for a proportion near 0.5 is 5 percentage points, so the claim that this is 'near-random' is compatible with true accuracy well above 50%. Likewise, the 'human parity' claims for static perception must be qualified by the uncertainty in both the model and human estimates. The authors should report exact binomial confidence intervals or equivalent uncertainty quantification for all asterisked entries, and should temper statements such as 'Frontier Models Reach Human Parity in Static Perception' accordingly.
  3. [Section 5.3 / Table 3 / Appendix D.2] The perception-filtering analysis that underpins 'VLMs struggle with intuitive physics even under accurate state perception' is underspecified. Appendix D.2 states that the protocol excluded approximately 30% of instances, but Section 5.3 describes the filter as instances 'where all models correctly answer all relevant perception questions,' without clarifying whether the filter is model-specific or global, which tasks it applies to, or how the three listed queries map onto each prediction task. Table 3 reports only accuracy differences between filtered and unfiltered inputs, with no sample sizes, no per-condition accuracies, and no confidence intervals. As a result, the quantitative conclusion that 'limited perception capability is not the only cause' is not adequately supported for the three tasks shown, and the generalization to prediction tasks generally is even less supported. The authors should report the filtered and unfiltered accuracies separately, with uncertainties, and should state the filtering rule precisely.
minor comments (5)
  1. [Section 4.4] There is a typo in the first sentence: 'Depite' should be 'Despite'.
  2. [Appendix C.2] The sentence 'We use both ManiSkill framework version 2 (Gu et al., 2023) and 3 (Tao et al., 2024))' has an unmatched closing parenthesis; remove the extra parenthesis.
  3. [Table 3] The caption uses the abbreviation 'filtered (correct state perception)' without defining the filtering procedure or the sample size; please define it in the caption or refer to a precise appendix section.
  4. [Appendix A.3] The compositional inference display contains mismatched parentheses in 'S(i)t ∼ Compose(S(1)t , . . . S(n)t )'; this should be cleaned up for readability.
  5. [Section 4.1] The human evaluation uses 50 samples per task with three annotators, but the reported human accuracies in Tables 1 and 2 are point estimates; adding a simple binomial confidence interval or noting the sample size alongside each human row would help the reader judge the human baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical benchmark measurements scored against simulator ground truth and human baselines, not derivations from fitted inputs or self-cited theorems.

full rationale

WM-ABench is an evaluation artifact rather than a derivation. The dual-stage framework is a taxonomic decomposition sourced from comparative psychology and cognitive science, and no equation in the paper defines a predicted quantity in terms of the quantity being predicted. The headline findings (e.g., near-random Motion Trajectory accuracy, color-speed entanglement, weak transitive/compositional prediction) are computed by scoring fixed model outputs against simulator ground truth and human solvability baselines; no parameter is fitted to a subset of results and then renamed as a prediction. The paper's self-citations (Zhang et al. 2025, Pi et al. 2024, Hu and Shu 2023, Ai et al. 2024, Tian et al. 2025, and the ManiSkill/TDW simulator papers) are either corroborative context, related-work comparisons, or externally maintained artifacts; none is the sole justification for the benchmark's conclusions, and none imports a uniqueness theorem that forces the authors' choice. Section 5.4's real-world recast is an underpowered but direct empirical check, and the Limitations passage conceding that synthetic images may be out-of-distribution is an external-validity caveat, not a circularity. Accordingly, there are no circular steps to report.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper contributes an evaluation artifact, not a derivation. It introduces no fitted free parameters and no new physical entities. Its central conclusion rests on three domain assumptions: the cognitive taxonomy, simulator fidelity, and the validity of counterfactual hard negatives.

assumptions (3)
  • domain assumption The two-stage perception/prediction decomposition from comparative psychology is a valid and sufficiently complete decomposition of world modeling.
    Section 2 and Appendix A adopt Spelke, Knill and Pouget, and Ha and Schmidhuber to define perception and prediction; the choice of five perception and three prediction dimensions is a modeling assumption, not proven.
  • domain assumption Simulators (ThreeDWorld, ManiSkill, Physion, Carla, Habitat) faithfully instantiate the physical and agentic dynamics targeted by the questions.
    Data collection in Section 3 and Appendix C assumes that simulator transitions are valid ground truth; the human baseline supports solvability but not full physical fidelity.
  • domain assumption Counterfactual option generation (perturbing action or previous state) yields hard negatives that require genuine dynamics understanding rather than low-level image shortcuts.
    Section 3, 'Fighting Shortcuts and Spurious Correlations,' relies on this premise to interpret accuracy as world modeling ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation." pith.science (2026). https://pith.science/paper/D5G4CPAY

@misc{pith2026250621876,
  author       = {Pith},
  title        = {Pith review of: Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5G4CPAY}},
  note         = {Machine review of arXiv:2506.21876}
}
read the original abstract

Internal world models (WMs) enable agents to understand the world's state and predict transitions, serving as the basis for advanced deliberative reasoning. Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs' fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses Perception (visual, spatial, temporal, quantitative, and motion) and Prediction (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, almost all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding -- e.g., some models tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.

Figures

Figures reproduced from arXiv: 2506.21876 by the authors.

Figure 1
Figure 1. WM-ABench overview. Grounded in comparative psychology and cognitive science, our framework delineates the functioning of world models into two distinct stages: (1) perception stage, encompassing visual, spatial, temporal, quantitative, and motion perceptions, and (2) prediction stage, involving mechanistic simulation, transitive inference, and compositional inference. Large-scale evaluation reveals VLMs’ limitation… view at source ↗
Figure 2
Figure 2. Overview of WM-ABench tasks. The Perception stage (top) covers Spatial, Temporal, Visual, Quantity, and Motion dimensions, each shown with example questions and outputs. The Prediction stage (bottom) includes Mechanistic Simulation, which covers Intuitive Physics (e.g., drop), Agent Navigation (e.g., turn left), and Agent Manipulation (e.g., push), plus Transitivity and Compositionality tasks that build on these tra… view at source ↗
Figure 3
Figure 3. Model performance with respect to increasing stimulus differences, as discussed in Section [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Heatmap showing s-RE scores that quantify the relative impact of different physical dimensions on [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning

    cs.MM 2026-07 conditional novelty 7.0 of 10

    Current VLMs fail retrospective physical reachability and causal reconstruction on RetroHolmes; a simple analysis-by-synthesis loop with video simulation reduces bias and belief-conflict sensitivity.

  2. Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Preserving pretrained VLM features with layer-wise distillation plus supervising the language head on discretized action directions improves OOD generalization of VLA policies on LIBERO, CALVIN, and a real xArm7.

  3. DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

    cs.LG 2026-07 accept novelty 6.0 of 10

    Schema-guided interleaved state-transition pretraining with selective attention and reweighted loss improves hierarchical visual dynamics modeling for narrative generation and world simulation.

  4. Egocentric Bias in Vision-Language Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Most vision-language models fail Level-2 visual perspective taking: they report the camera's view rather than the 180°-rotated string, even though they often recognize that another agent sees differently.

Reference graph

Works this paper leans on

45 extracted references · 32 canonical work pages · cited by 4 Pith papers

  1. [1]

    Any valid program can be expressed as an equivalent topologically sorted computational graph based on dependency structures

    work page

  2. [2]

    All world simulators function as programs

    work page

  3. [3]

    The topological order of the graph is determined by time and world dynamics

    Therefore, all world simulators have equivalent graphs. The topological order of the graph is determined by time and world dynamics. Within this framework, future state prediction involves computing values for future nodes based on historical nodes, using the current time step as a cutoff. We then identify three necessary and collectively sufficient condi...

    work page 2013

  4. [4]

    These multi-view tasks emphasize the model’s ability to synthesize distinct viewpoints into a coherent three-dimensional representation of object arrangements

    Spatial Positioning (SP): Given the front and side view of a set of object arrangements, let the model infer the top view of the objects. These multi-view tasks emphasize the model’s ability to synthesize distinct viewpoints into a coherent three-dimensional representation of object arrangements. Temporal Perception

    work page

  5. [6]

    International Journal of Computer Vision, 127(4):398–414

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127(4):398–414. Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. 2023. Maniskill2:...

    arXiv 2023

  6. [10]

    Preprint, arXiv:2304.03843

    Why think step by step? reasoning emerges from the locality of experience. Preprint, arXiv:2304.03843. Viorica P ˘atr˘aucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Han...

    arXiv 2023

  7. [11]

    Preprint, arXiv:2407.06581

    Vision language models are blind. Preprint, arXiv:2407.06581. Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada

    arXiv

  8. [12]

    In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    AI and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Rajesh PN Rao and Dana H Ballard. 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79–87. H...

    arXiv 1999

Show all 45 references
  1. [13]

    science, 332(6033):1054– 1059

    Pure reasoning in 12-month-old infants as probabilistic inference. science, 332(6033):1054– 1059. Susan P Thompson and Elissa L Newport. 2007. Statistical learning of syntax: The role of transitional probability. Language learning and development , 3(1):1–42. Tongxuan Tian, Ha...

    2007 arXiv

  2. [15]

    In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models

    MM-spubench: Towards better understanding of spurious biases in multimodal LLMs. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 20...

    1910 arXiv

  3. [17]

    inaction

    Vl-checklist: Evaluating pre-trained vision- language models with objects, attributes and relations. Preprint, arXiv:2207.00221. Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. 2024. Robodreamer: Learning compositional world models for robot ima...

    2023 arXiv

  4. [22]

    This task evaluates whether the model can accurately discern spatial relationships based on visual cues

    Spatial Relation (SR): Given the front and top view of two objects on a table, let the model infer the relative position of one object with respect to the other. This task evaluates whether the model can accurately discern spatial relationships based on visual cues

  5. [23]

    Thus testing the model’s understanding of spatial constraints

    Spatial Vacancy (SE-V): Given an object and a structure with a hollow space in the middle, let the model infer whether the object can fit into the hollow space. Thus testing the model’s understanding of spatial constraints

  6. [24]

    Thus testing the model’s understanding of object size

    Spatial Occupancy (SE-O): Given two objects with different sizes, let the model infer which object is larger. Thus testing the model’s understanding of object size

  7. [26]

    Temporal positioning (TP): Compare two episodes of object motions from different views, and let the model differentiate which motion started first

  8. [27]

    Collectively, these tasks investigate the aptitude of a model to maintain consistent temporal representations, estimate durations, and infer the correct order of events

    Temporal extension (TE): Compare two episodes of object motions from different perspectives, and let the model differentiate which motion lasts longer. Collectively, these tasks investigate the aptitude of a model to maintain consistent temporal representations, estimate durat...

  9. [28]

    Color (V-C): Differentiating whether two objects have the same color, or identifying which color the object is

  10. [29]

    Shape (V-S): Determine the object’s shape; another task involves differentiating whether two objects have the same shape

  11. [30]

    Material (V-M): Differentiating whether the two objects have the same material or the model determine the material of the given object. By isolating these fundamental visual features, the tasks provide targeted evaluations of how effectively a model can parse and differentiate...

  12. [31]

    This setup tests the model’s capacity to identify the moving action of objects

    Motion Identification (MI): Given an episodes of motions of an object and a set of static objects, let the model decide which object is moving. This setup tests the model’s capacity to identify the moving action of objects

  13. [32]

    This setup tests the model’s capacity to track position changes over time and estimate relative velocity

    Motion Speed (MS): Given episodes of motions of two objects, let the model decide which object moves faster. This setup tests the model’s capacity to track position changes over time and estimate relative velocity

  14. [33]

    This setup tests the model’s 20 capacity to track position changes over time and estimate relative moving direction

    Motion Direction (MD): Given one moving object and a set of static objects, let the model determine which static object the moving object is heading towards. This setup tests the model’s 20 capacity to track position changes over time and estimate relative moving direction

  15. [34]

    Quantitative Perception

    Motion Trajectory (MT): Given an episode of motions of two objects, let the model decide the moving trajectory of a specified object, assessing its aptitude for higher-level spatiotemporal pattern recognition and object-specific path tracking across multiple frames. Quantitati...

  16. [35]

    This setup evaluates the model capacity for discrete numerical estimation

    Discrete Quantity (DQ): Given the top view of objects on the table, let the model count the objects. This setup evaluates the model capacity for discrete numerical estimation

  17. [36]

    Continuous Quantity (CQ): Given the top view of two objects with the same color theme, let the model determine which object has a darker shade

  18. [37]

    This setup probes counting skills, numerical reasoning, and perceptual comparisons in a visual context

    Relative Quantity (RQ): Given the top view of objects with different colors, determine which color group has more objects. This setup probes counting skills, numerical reasoning, and perceptual comparisons in a visual context. B.1.2 Prediction Mechanistic Knowledge

  19. [38]

    This setup evaluates the model capacity in physical reasoning

    Intuitive Physics (M-IP): Given a sequence of images showing consecutive states of the environment in which two objects move towards each other, let the model choose the most probable prediction of the next state. This setup evaluates the model capacity in physical reasoning

  20. [39]

    This setup evaluates the model capacity in predictive reasoning

    Agent Navigation (M-Nav): Given an image of the start state, let the model choose what is most likely to be the final state after the robot/vehicle moves in a certain direction. This setup evaluates the model capacity in predictive reasoning

  21. [40]

    This setup evaluates the model capacity in predictive reasoning for robot manipulation

    Agent Manipulation (M-Man): Given an image of the start state, let the model choose what is most likely to be the final state after the robot arm does certain movements toward the object. This setup evaluates the model capacity in predictive reasoning for robot manipulation. T...

  22. [41]

    This setup evaluates the model capacity in predictive reasoning for autonomous navigation

    Agent Navigation (T-Nav): Given an image of the start state, let the model choose what is most likely to be the final state after the robot/vehicle moves through multiple directions in sequence. This setup evaluates the model capacity in predictive reasoning for autonomous navigation

  23. [42]

    This setup evaluates the model capacity in multi-step predictive reasoning for robotic manipulation

    Agent Manipulation (T-Man): Given an image of the start state, let the model choose what is most likely to be the final state after the robot arm performs two actions in sequence. This setup evaluates the model capacity in multi-step predictive reasoning for robotic manipulati...

  24. [43]

    This task evaluates the model’s ability to perform compositional inferences about physical causality and object behavior

    Multi-Object Intuitive Physics (C-IP): Given several images of two balls colliding with a third object at the same time, let the model predict the state after the collision occurred. This task evaluates the model’s ability to perform compositional inferences about physical cau...

  25. [44]

    This setup evaluates the model capacity in concurrent action predictive reasoning for robotic manipulation

    Multi-Agent Manipulation (C-Man): Given an image of the start state, let the model choose what is most likely to be the final state after two robot arms do certain actions on one object simultaneously. This setup evaluates the model capacity in concurrent action predictive rea...

    2024

  26. [45]

    yes” or “no

    to generate images for the perception tasks, with intuitive physics (e.g., collisions) modeled using the Physion framework (Bear et al., 2022). We first select a curated set of pre-packaged scenes, objects, and materials from ThreeDWorld. For most questions, we spawn selected ...

    2022

  27. [2004]

    John Wiley & Sons Hoboken, NJ

    Sensation and perception. John Wiley & Sons Hoboken, NJ. Carl Craver, James Tabery, and Phyllis Illari. 2024. Mechanisms in Science. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy, Fall 2024 edition. Metaphysics Research Lab, Stanford Uni...

    2024

  28. [2010]

    S(n) t to denote the whole relationship between n component states and complex states

    St = Compose S(1) t , . . . S(n) t to denote the whole relationship between n component states and complex states. Then compositional inference can 19 be expressed as: St ∼ Compose S(1) t , . . . S(n) t ) where S(i) t ∼ Pθ(S(i) t | S(i) t−1, at−1, . . . S(i) t−h, at−h . The co...

  29. [2011]

    Animal behaviour, 82(2):359–367

    Ontogeny of object permanence and object tracking in the carrion crow, corvus corone. Animal behaviour, 82(2):359–367. Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. 2023. Gaia-1: A generative world model f...

    2023 arXiv

  30. [2016]

    Social and Personality Psychology Compass, 10(7):405–420

    Mental simulation as substitute for experience. Social and Personality Psychology Compass, 10(7):405–420. 12 Edna L Kaufman, Miles W Lord, Thomas Whelan Reese, and John V olkmann. 1949. The discrimination of visual number. The American journal of psychology, 62(4):498–525. Kor...

    1949 arXiv

  31. [2018]

    In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 2127–21273

    Active vision dataset benchmark. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 2127–21273. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision...

    2018 arXiv

  32. [2019]

    Preprint, arXiv:1908.02660

    Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition. Preprint, arXiv:1908.02660. Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. 2023. Learning interactive real-world simulators. arXiv prepri...

    1908 arXiv

  33. [2020]

    In Advances in Neural Information Processing Systems, volume 33, pages 10514–10525

    Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. In Advances in Neural Information Processing Systems, volume 33, pages 10514–10525. Curran Associates, Inc. Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  34. [2021]

    In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual

    Threedworld: A platform for interactive multi- modal physical simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. Qiaozi Gao, Shaohua Yang, Joyce Chai, and Luc...

    2021 arXiv

  35. [2023]

    Preprint, arXiv:2305.18654

    Faith and fate: Limits of transformers on compositionality. Preprint, arXiv:2305.18654. Johanna Eckert, Hannes Rakoczy, Shona Duguid, Esther Herrmann, and Josep Call. 2021. The ape lottery: Chimpanzees fail to consider spatial information when drawing statistical inferences. A...

    2021 arXiv

  36. [2024]

    arXiv preprint arXiv:2411.02385

    How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385. Immanuel Kant, John Miller Dow Meiklejohn, Thomas Kingsmill Abbott, and James Creed Meredith. 1934. Critique of pure reason. JM Dent London. Heather Barry Kappes and Care...

    1934 arXiv

  37. [2025]

    In The Thirteenth International Conference on Learning Representations

    Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities. In The Thirteenth International Conference on Learning Representations. Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.