Pith. sign in

REVIEW 4 major objections 6 minor 10 cited by

IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art video models perform near chance on IntPhys 2, a video benchmark probing four core intuitive-physics principles, while humans achieve near-perfect accuracy.

desk verdict Useful benchmark, but the headline claim that models lack physical world models is undercut by the absence of a perceptual control showing models can even see and track the objects in these videos. read the letter →

arxiv 2506.09849 v1 pith:EJRO2ZXF submitted 2025-06-11 cs.CV

Florian Bordes , Quentin Garrido , Justine T Kao , Adina Williams , Michael Rabbat , Emmanuel Dupoux This is my paper
classification cs.CV
keywords intuitivephysicsvideobenchmarkviolationofexpectationworldmodelsphysicalreasoningmultimodallargelanguagesyntheticenvironmentsobjectpermanence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IntPhys 2 is a video benchmark that asks AI models to tell physically possible events apart from impossible ones, across four principles known to guide human infants: object permanence, immutability, spatio-temporal continuity, and solidity. The paper evaluates state-of-the-art multimodal large language models and predictive video models on a set of 1,416 photorealistic synthetic videos, each paired in counterbalanced quadruplets so that the only intended difference is physical plausibility. Most models score near chance (49–58 percent on the main set), with the best model at 55.63 percent, while human annotators reach 96.44 percent. The authors take this gap as evidence that current video models have not learned a usable physical world model, and that the benchmark offers a way to measure progress toward that capability.

What carries the argument

The evaluative machinery is the violation-of-expectation framework applied to counterbalanced video quadruplets: each scene contains two possible and two impossible videos, constructed so that the same footage is possible in one setting and impossible in the other (for example, a ball that changes course only when an obstacle is present). This structure is intended to remove low-level perceptual shortcuts so that success requires tracking objects over occlusions and reasoning about the four core principles. For predictive models, surprise is operationalized as prediction error over sliding windows; for multimodal language models, plausibility is elicited via prompts. The benchmark also varies camera motion, realism, and occlusion duration to demand short-term memory and robust reasoning.

What would settle it

A controlled test would compare a model's accuracy on IntPhys 2 against a version of the benchmark in which the possible/impossible labels are randomized while keeping the videos identical: if a model trained on the mislabeled version still reaches above-chance performance on the original held-out set, then low-level cues, not physics, are driving the decisions. Equivalently, a pixel-level classifier or human perceptual study that can distinguish the two video classes better than chance would undercut the claim that models are at chance because they lack physical world knowledge.

Watch

Extended reading notes

Core claim

The central claim is that state-of-the-art AI systems, both multimodal large language models and self-supervised predictive video models, fail to exhibit intuitive physics understanding on IntPhys 2, performing at or slightly above chance across all difficulty splits. The best overall accuracy on the main set is 55.63 percent (Gemini 2.5 Flash); the best predictive model, V-JEPA 2, reaches 57.51 percent; and both fall far below the human majority-vote accuracy of 96.44 percent. The authors conclude from this that current models have not learned a good physical world model, and that the short-term memory demands and photorealistic complexity of IntPhys 2, unlike the saturated original IntPhys benchmark, expose the gap between artificial and human-like physical reasoning.

Load-bearing premise

The entire conclusion—that current models lack intuitive physics understanding—rests on the assumption that the possible and impossible videos in each quadruplet differ only in physical plausibility and not in low-level visual cues, an assumption the paper states but does not verify with pixel-level analysis.

Editorial extensions

If this is right

  • Models that excel on the original IntPhys benchmark, such as V-JEPA, drop to near-chance performance on IntPhys 2, showing that prior results do not extend to more complex, photorealistic scenes.
  • The held-out split, released without metadata, gives a way to confirm that future performance gains come from genuine physical reasoning rather than from training on benchmark labels.
  • The finding that feeding more frames often hurts multimodal-model accuracy points to short-term memory and long-context processing as a specific bottleneck for video-based physical reasoning.
  • The benchmark positions four infant-tested principles—permanence, immutability, continuity, and solidity—as a diagnostic axis that separates human-like physical understanding from current model behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the counterbalancing assumption holds exactly, IntPhys 2 could double as a probe of object-centric representation: success on permanence and continuity would require models to track object identity through occlusion rather than rely on motion or appearance statistics.
  • A pixel-level check for shortcut cues—such as rendering noise or motion discontinuities that correlate with the possible/impossible label—would directly test whether near-chance accuracy reflects missing physics or missing low-level artifacts; the paper does not report such an analysis.
  • The protocol's sensitivity to prompts and frame counts suggests that model rankings on video physics tasks are partly an artifact of evaluation choices, so future comparisons should report distributions over prompts and context lengths rather than a single best run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces IntPhys 2, a synthetic video benchmark for evaluating intuitive physics understanding in machine learning models. It builds on the original IntPhys benchmark, using a violation-of-expectation design with quadruplets of videos (two possible, two impossible) across four principles (permanence, immutability, spatio-temporal continuity, solidity), and features photorealistic Unreal Engine scenes, static and moving cameras, and three difficulty splits. The authors evaluate humans, several multimodal large language models (GPT-4o, Qwen-VL, Gemini variants), and four prediction-based models (VideoMAEv2, Cosmos-Predict-4B, V-JEPA, V-JEPA 2). They report near-chance model performance (roughly 50-58% on the main set) in stark contrast to near-perfect human accuracy (96.44%), and conclude that current models have not learned a good physical world model. The paper also provides ablation studies on prompts, input frames, and generation artifacts, and releases code, data, and a leaderboard for the held-out set.

Significance. If the central claim is supported, IntPhys 2 is a valuable contribution to intuitive physics benchmarking: it extends the scope of physical principles, introduces more realistic occlusions and camera dynamics, and provides a multi-family model evaluation protocol. The paper's strengths include the counterbalanced quadruplet design, human annotation, the explicit study of prompt and frame-count sensitivity, the release of code/data with a leaderboard, and the transparency about hyperparameter grids and run costs. These are praiseworthy. However, the headline inference—that near-chance model accuracy reflects an absence of physical understanding—is underdetermined unless models can be shown to perceive and track the relevant objects in these exact videos, and unless the single-video classification protocol is fully specified. The benchmark may still be a useful resource for the community, but the empirical claim about model understanding requires additional controls and analyses.

major comments (4)
  1. [Section 4.2 and Appendix E] The inference from near-chance accuracy to an absence of physical understanding is underdetermined because the paper does not provide a positive perceptual or memory control on the same videos. Appendix E.1 (Fig. 9) shows V-JEPA's surprise rising when the ball enters the frame but not differentiating possible from impossible reappearance, and Appendix E.2 (Figs. 10-11) shows Cosmos failing to reconstruct the ball after a few frames even while it is still visible. These observations indicate that models may fail at perception or short-term tracking rather than at physical reasoning. The authors should add a control task, e.g., object tracking or reappearance detection on the same IntPhys 2 videos, and report whether models can encode and follow the critical object. Without such a control, near-chance scores are ambiguous for the paper's central conclusion.
  2. [Section 3 and Appendix D.1] The single-video classification protocol is not fully defined. The paper defines AvgSurprise and MaxSurprise but does not specify the decision rule (e.g., a threshold) that converts a scalar surprise into a binary possible/impossible judgment. If the AUROC shown in Figure 4 is used instead, the accuracy values in Table 2 need to be reconciled with that metric. Because the threshold is a free parameter, its unspecified choice could affect all reported predictive-model accuracies. The authors should state exactly how single-video accuracy was computed, or report AUROC consistently and justify why the reported accuracy is not threshold-dependent.
  3. [Table 2 and Section 4] Table 2 reports the best accuracy over many hyperparameter runs and prompts for each model, without error bars, confidence intervals, or a description of the selection rule. This best-run selection biases scores upward and makes quantitative comparisons across models and subsets unreliable. Although the direction of the human-model gap is likely robust, the precise values (e.g., Gemini 2.5 Flash at 55.63% vs. V-JEPA 2 at 57.51%) are not reliable as stated. The authors should report mean and standard deviation (or confidence intervals) over runs, and clarify whether the best run is selected per column, per model, or globally.
  4. [Section 2] The claim that the counterbalanced quadruplet design is 'instrumental in mitigating low-level perceptual biases' is asserted but not validated. No analysis is provided to show that possible and impossible videos are indistinguishable on low-level pixel statistics, such as rendering noise, texture mismatch, or motion discontinuities at the violation site. If such artifacts exist, near-chance model performance could reflect the absence of shortcuts rather than the absence of physical understanding. The authors should provide a control, e.g., a simple pixel-based classifier or a human-perceptual rating of low-level differences, to demonstrate that the intended physical event, not a surface artifact, is the only systematic difference.
minor comments (6)
  1. [Throughout] Please fix inconsistencies in benchmark naming: 'IntPhys2', 'IntPhys 2', and 'IntPhys 2' are used interchangeably across headings, captions, and text.
  2. [Table 2] Several entries in the VideoMAEv2-g row are concatenated without separators (e.g., '46.0058.5052.73 53.75'); please format all table cells consistently.
  3. [Abstract and Section 1] There are typos and missing spaces, e.g., 'wether' (whether), 'physic' (physical), 'plausibler' (plausible), 'reprent' (represent), and 'These datasetsshould not be usedfor generative modeling purposes' in the datasheet.
  4. [Appendix A.4] The datasheet says the debug set has 'around 100 videos', while Table 1 states 60 videos; please reconcile these numbers.
  5. [Equation (1)] The notation in the surprise definition is hard to parse due to missing indices and spacing (e.g., 'Surprisew =d(p(f(V w:w+C )), f′ (Vw+C:w+M ))'); please rewrite with clear subscripts and define all variables (w, C, T, M, S).
  6. [Figure 3] The leftmost plot (Debug set) would benefit from error bars or repeated-run markers; the current plot does not convey run-to-run variability, which is relevant given the best-run reporting issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark evaluation is empirical and self-contained.

full rationale

The paper's central claim is that state-of-the-art models perform near chance on a newly constructed benchmark while humans perform near ceiling. This is an empirical measurement, not a derivation. The benchmark's possible/impossible labels come from a physical simulator, and human judgments are collected independently as a reference standard; no parameter is fitted to any subset of model outputs and then renamed as a prediction. The evaluation protocol for predictive models is adopted from the authors' prior work [21], but it is not itself the quantity being predicted, and the paper explicitly checks protocol invariance in Appendix D.2 (Table 7), finding equivalent performance under the prior protocol. The use of self-trained V-JEPA models and self-authored prompts is a potential source of bias, but it does not force the near-chance results; the same protocol and family of models perform well on the original IntPhys benchmark, showing that the protocol can in principle yield high scores. The absence of a positive perceptual control is a legitimate validity concern, but it is an underdetermination issue rather than a circularity: failing to perceive the ball could explain chance performance, but this does not mean the paper's conclusion is definitionally equivalent to its inputs. The benchmark construction, human evaluation, and model evaluations are all externally grounded, so no circular step can be exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark construction rests on several domain assumptions: counterbalanced video quadruplets remove low-level cues, prediction error is a valid surprise proxy, human ratings are ground truth, and Unreal Engine physics approximates real physics. The only numerical free parameter we could identify is the unstated threshold used to convert surprise scores into binary accuracy. No new physical entities are introduced.

free parameters (1)
  • Binary plausibility threshold for surprise-to-accuracy conversion
    Single-video classification accuracy requires thresholding continuous surprise scores (Eq. 1-2 in Appendix D), but the threshold value and calibration are not reported, so exact accuracies and chance-level comparisons are not fully reproducible.
assumptions (4)
  • domain assumption Video quadruplet counterbalancing eliminates low-level perceptual biases, so accuracy reflects physical understanding.
    Section 2 asserts this design is instrumental in mitigating low-level perceptual biases, but no control analysis demonstrates that possible and impossible videos are indistinguishable except for physics.
  • domain assumption Prediction error is a valid proxy for surprise and therefore for physical expectation violation.
    Section 3 and Appendix D define surprise as prediction error; this is inherited from prior work and not independently validated on IntPhys 2.
  • domain assumption Human Likert ratings and majority vote provide ground-truth plausibility labels.
    Section 3 describes annotation but does not report inter-annotator agreement or checks for order and familiarization effects.
  • domain assumption Unreal Engine physics simulation is a faithful analog of real-world physical laws for these macroscopic scenarios.
    Appendix A describes rendering and physics in Unreal Engine; the paper relies on this to define possible versus impossible events.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments." pith.science (2026). https://pith.science/paper/EJRO2ZXF

@misc{pith2026250609849,
  author       = {Pith},
  title        = {Pith review of: IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJRO2ZXF}},
  note         = {Machine review of arXiv:2506.09849}
}
read the original abstract

We present IntPhys 2, a video benchmark designed to evaluate the intuitive physics understanding of deep learning models. Building on the original IntPhys benchmark, IntPhys 2 focuses on four core principles related to macroscopic objects: Permanence, Immutability, Spatio-Temporal Continuity, and Solidity. These conditions are inspired by research into intuitive physical understanding emerging during early childhood. IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments. Alongside the benchmark, we provide performance evaluations of several state-of-the-art models. Our findings indicate that while these models demonstrate basic visual understanding, they face significant challenges in grasping intuitive physics across the four principles in complex scenes, with most models performing at chance levels (50%), in stark contrast to human performance, which achieves near-perfect accuracy. This underscores the gap between current models and human-like intuitive physics understanding, highlighting the need for advancements in model architectures and training methodologies.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

    cs.CV 2026-07 conditional novelty 7.0 of 10

    A law-grounded benchmark, Apple-PI, grades video models stage-by-stage on physics reasoning and finds they top out at 0.473, well short of reliable simulation.

  2. The TIME Machine: On The Power of Motion for Efficient Perception

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    TIME is a motion-based embedding from point tracks, trained only on synthetic data via masked autoencoding, that matches state-of-the-art video model performance with up to 10,000x less training data.

  3. PhiZero: A World Model Built Around Physical Language

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A self-supervised discrete physical-language bottleneck plus a VLM reasoner lets a world model predict state transitions before rendering video, improving physical coherence and enabling zero-shot motion transfer.

  4. PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Structured physical memories (captions, event graphs, rule cues) retrieved and verified at inference time consistently improve frozen VLMs on physical-plausibility benchmarks without fine-tuning.

  5. VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.

  6. ChronoVision: Temporal Reasoning via Latent State Reconstruction

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ChronoVision reaches 74.8% in-domain and 71.6% out-of-domain exact-match accuracy on the new Vbvr-VQA frame-ordering benchmark plus 55.0% on IntPhys2, with a 9B model trained on one million same-suite examples.

  7. SiPhy: Single-Image Physical Property Reasoning

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A single-image vision-language pipeline reports state-of-the-art mass, density, and stiffness predictions by combining CLIP features, a fine-tuned VLM, and depth-adaptive pseudo-voxel sampling.

  8. Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models

    cs.CL 2025-07 reject novelty 5.0 of 10

    Even the newest multimodal LLMs score near chance on intuitive physics videos, and the paper's probing evidence that vision encoders hold the relevant information is confounded by scene identity.

  9. Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective

    cs.AI 2025-09 conditional novelty 4.0 of 10

    Agent spatial intelligence is organized into six neuroscience-inspired modules, and the field is reviewed through that lens without any experimental validation.

  10. Embodied AI Agents: Modeling the World

    cs.AI 2025-06 conditional novelty 4.0 of 10

    Embodied AI agents should be built around physical world models plus a mental world model of the user, with virtual, wearable, and robotic agents sharing this core.

Reference graph

Works this paper leans on

56 extracted references · 36 canonical work pages · cited by 10 Pith papers

  1. [1]

    Agarwal, A

    N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    arXiv 2025

  2. [2]

    Assran, A

    M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas. V-jepa 2: Self-supervised vid...

    work page 2025

  3. [3]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Baillargeon and J

    R. Baillargeon and J. DeVos. Object permanence in young infants: Further evidence.Child Development, 62(6): 1227, Dec. 1991. ISSN 00093920. doi: 10.2307/1130803

    work page doi:10.2307/1130803 1991

  5. [5]

    Baillargeon and S

    R. Baillargeon and S. Hanko-Summers. Is the top object adequately supported by the bottom object? young infants’ understanding of support relations.Cognitive Development, 5(1):29–53, Jan. 1990. ISSN 08852014. doi: 10.1016/0885-2014(90)90011-H

    work page doi:10.1016/0885-2014(90)90011-h 1990

  6. [6]

    Baillargeon, E

    R. Baillargeon, E. S. Spelke, and S. Wasserman. Object permanence in five-month-old infants.Cognition, 20(3): 191–208, 1985

    work page 1985

  7. [7]

    Baillargeon, A

    R. Baillargeon, A. Needham, and J. Devos. The development of young infants’ intuitions about support.Early Development and Parenting, 1(2):69–78, Jan. 1992. ISSN 1057-3593, 1099-0917. doi: 10.1002/edp.2430010203

    doi:10.1002/edp.2430010203 1992

  8. [8]

    Bakhtin, L

    A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick. Phyre: A new benchmark for physical reasoning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/pa...

    work page 2019

Show all 56 references
  1. [9]

    Bansal, Z

    H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y. Bitton, C. Jiang, Y. Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation.arXiv:2406.03520, 2024

    2024 arXiv

  2. [10]

    Bardes, Q

    A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. Revisiting feature prediction for learning visual representations from video.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. Featured Certification

    2024

  3. [11]

    P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327–18332, Nov. 2013. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1306572110

    2013 doi

  4. [12]

    D. Bear, E. Wang, D. Mrowca, F. Binder, H.-Y. Tung, P. RT, C. Holdaway, S. Tao, K. Smith, F.-Y. Sun, F.-F. Li, N. Kanwisher, J. Tenenbaum, D. Yamins, and J. Fan. Physion: Evaluating physical prediction from vision in humans and machines. In J. Vanschoren and S. Yeung, editors,...

    2021

  5. [13]

    Benchekroun, M

    Y. Benchekroun, M. Dervishi, M. Ibrahim, J.-B. Gaya, X. Martinet, G. Mialon, T. Scialom, E. Dupoux, D. Hupkes, and P. Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models. arXiv:2311.15930, 2023

    2023 arXiv

  6. [14]

    Y. Bisk, R. Zellers, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI conference on artificial intelligence, 34:7432–7439, 2020

    2020

  7. [15]

    Bordes, S

    F. Bordes, S. Shekhar, M. Ibrahim, D. Bouchacourt, P. Vincent, and A. S. Morcos. PUG: Photorealistic and semantically controllable synthetic data for representation learning. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 20...

    2023

  8. [16]

    Bordes, R

    F. Bordes, R. Y. Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Mañas, Z. Lin, A. Mahmoud, B. Jayaraman, M. Ibrahim, M. Hall, Y. Xiong, J. Lebensold, C. Ross, S. Jayakumar, C. Guo, D. Bouchacourt, H. Al-Tahan, K. Padthe, V. Sharma, H. Xu, X. E. Tan, M. Richards, S. Lavoie, ...

    2024 arXiv

  9. [17]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Che...

    1901

  10. [18]

    Dosovitskiy, G

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017

    2017

  11. [19]

    PUG environments and datasets

    EpicGames. URL https://www.unrealengine.com. Unreal Engine is a copyright of Epic Games, Inc. and its affiliates (collectively, “Epic”). Any use of images, datasets, or other content made available by Epic, including without limitation through the Unreal Engine Marketplace or ...

  12. [20]

    Epstein, B

    D. Epstein, B. Chen, and C. Vondrick. Oops! predicting unintentional action in video. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 919–929, 2020

    2020

  13. [21]

    Garrido, N

    Q. Garrido, N. Ballas, M. Assran, A. Bardes, L. Najman, M. Rabbat, E. Dupoux, and Y. LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos.arXiv preprint arXiv:2502.11831, 2025

    2025 arXiv

  14. [22]

    Gebru, J

    T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

  15. [23]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URLhttps://arxiv.org/abs/2312. 11805

    2024

  16. [24]

    Girdhar and D

    R. Girdhar and D. Ramanan. Cater: A diagnostic dataset for compositional actions & temporal reasoning. InInter- national Conference on Learning Representations, 2020. URLhttps://openreview.net/forum?id=HJgzt2VKPB

    2020

  17. [25]

    Ha and J

    D. Ha and J. Schmidhuber. World models, 2018

    2018

  18. [26]

    Hafner, T

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. InInternational Conference on Machine Learning, pages 2555–2565, 2019

    2019

  19. [27]

    Jassim, M

    S. Jassim, M. Holubar, A. Richter, C. Wolff, X. Ohmer, and E. Bruni. Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. In K. Larson, editor, Proceedings of the Thirty-Third International Joint Conferenc...

    2024 doi

  20. [28]

    Kolve, R

    E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    2017 arXiv

  21. [29]

    Kuratov, A

    Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024. URLhttps://arxiv.org/abs/2406.10149

    2024 arXiv

  22. [30]

    Y. LeCun. A path towards autonomous machine intelligence. Technical Report, Meta AI, 2022

    2022

  23. [31]

    Y. Li, T. Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In H. D. III and A. Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Re...

    2020

  24. [32]

    D. X. Long, H. N. Ngoc, T. Sim, H. Dao, S. Joty, K. Kawaguchi, N. F. Chen, and M.-Y. Kan. Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms, 2025. URL https://arxiv.org/abs/2408.08656

    2025 arXiv

  25. [33]

    Margoni, L

    F. Margoni, L. Surian, and R. Baillargeon. The violation-of-expectation paradigm: A conceptual overview. Psychological Review, 131(3):716–748, Apr. 2024. ISSN 1939-1471, 0033-295X. doi: 10.1037/rev0000450

    2024 doi

  26. [34]

    Motamed, L

    S. Motamed, L. Culp, K. Swersky, P. Jaini, and R. Geirhos. Do generative video models understand physical principles?arXiv preprint arXiv:2501.09038, 2025. 11

    2025 arXiv

  27. [35]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  28. [36]

    Piaget.The Construction of Reality in the Child

    J. Piaget.The Construction of Reality in the Child. Basic Books, 1954

    1954

  29. [37]

    L. S. Piloto, A. Weinstein, P. Battaglia, and M. Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology.Nature Human Behaviour, 6(9):1257–1267, July 2022. ISSN 2397-3374. doi: 10.1038/s41562-022-01394-8

    2022 doi

  30. [38]

    W. Qiu, Q. Zhou, C. Chen, and A. Yuille. Unrealcv: Virtual worlds for computer vision.ACM Multimedia Open Source Software Competition, 2017

    2017

  31. [39]

    Riochet, M

    R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning.CoRR, abs/1803.07616, 2018. URLhttp://arxiv.org/abs/1803. 07616

    2018 arXiv

  32. [40]

    Riochet, J

    R. Riochet, J. Sivic, I. Laptev, and E. Dupoux. Occlusion resistant learning of intuitive physics from videos. arXiv:2005.00069, 2020

    2005 arXiv

  33. [41]

    Sanchez-Gonzalez, J

    A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. W. Battaglia. Learning to simulate complex physics with graph networks. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

  34. [42]

    Smith, L

    K. Smith, L. Mei, S. Yao, J. Wu, E. Spelke, J. Tenenbaum, and T. Ullman. Modeling expectation violation in intuitive physics with coarse probabilistic object representations.Advances in neural information processing systems, 32, 2019

    2019

  35. [43]

    E. S. Spelke. Preferential-looking methods as tools for the study of cognition in infancy. In G. Gottlieb and N. A. Krasnegor, editors,Measurement of audition and vision in the first year of postnatal life: A methodological overview, pages 323–363. Ablex Publishing, 1985

    1985

  36. [44]

    E. S. Spelke, K. Breinlinger, J. Macomber, and K. Jacobson. Origins of knowledge.Psychological Review, 99(4): 605–632, 1992. ISSN 1939-1471, 0033-295X. doi: 10.1037/0033-295X.99.4.605

    1992 doi

  37. [46]

    E. S. Spelke, R. Kestenbaum, D. J. Simons, and D. Wein. Spatiotemporal continuity, smoothness of motion and object identity in infancy.British Journal of Developmental Psychology, 13(2):113–142, June 1995. ISSN 0261-510X, 2044-835X. doi: 10.1111/j.2044-835X.1995.tb00669.x

    1995 doi

  38. [47]

    J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu. Roformer: enhanced transformer with rotary position embedding. corr abs/2104.09864 (2021).arXiv:2104.09864, 2021

    2021 arXiv

  39. [48]

    H.-Y. Tung, M. Ding, Z. Chen, D. Bear, C. Gan, J. B. Tenenbaum, D. L. Yamins, J. E. Fan, and K. A. Smith. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties. InThirty-seventh Conference on Neural Information Proce...

    2023

  40. [49]

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao. Videomae v2: Scaling video masked autoencoders with dual masking.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549–14560, June 2023

    2023

  41. [50]

    Compositional4ddynamicscenesunderstanding with physics priors for video question answering

    X.Wang, W.Ma, A.Wang, S.Chen, A.Kortylewski, andA.Yuille. Compositional4ddynamicscenesunderstanding with physics priors for video question answering. InInternational Conference on Learning Representations (ICLR),

  42. [51]

    Watters, A

    N. Watters, A. Tacchetti, T. Weber, R. Pascanu, P. Battaglia, and D. Zoran. Visual Interaction Networks. arXiv:1706.01433, June 2017

    2017 arXiv

  43. [52]

    Weihs, A

    L. Weihs, A. Yuile, R. Baillargeon, C. Fisher, G. Marcus, R. Mottaghi, and A. Kembhavi. Benchmarking progress to infant-level physical reasoning in AI.Transactions on Machine Learning Research, 2022

    2022

  44. [53]

    T. Wilcox. Object individuation: Infants’ use of shape, size, pattern, and color.Cognition, 72(2):125–166, 1999

    1999

  45. [54]

    Wilcox and C

    T. Wilcox and C. Chapa. Priming infants to attend to color and pattern information in an individuation task. Cognition, 90(3):265–302, 2004. 12

    2004

  46. [55]

    T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface’s transformers: State-of-...

    2020 arXiv

  47. [56]

    surprise over time curve

    K. Yi*, C. Gan*, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HkxYzANYDB. 13 Appendix A IntPhys2 Ben...

    2020

  48. [2025]

    URLhttps://openreview.net/pdf?id=6Vx28LSR7f

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.