Pith. sign in

REVIEW 3 major objections 5 minor 9 cited by

A cheap add-on converts pretrained video generators into action-conditioned world models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:51 UTC pith:Q6KWIF6L

load-bearing objection A clean, credible trick for turning pretrained video generators into action-conditioned world models; the headline 65–80% claim needs a more trustworthy metric before I'd believe it. the 3 major comments →

arxiv 2601.15284 v2 pith:Q6KWIF6L submitted 2026-01-21 cs.CV

Walk through Paintings: Egocentric World Models from Internet Priors

Anurag Bagchi , Zhipeng Bao , Homanga Bharadhwaj , Yu-Xiong Wang , Pavel Tokmakov , Martial Hebert This is my paper
classification cs.CV
keywords world modelsvideo diffusionaction conditioningegocentric video predictiontimestep embeddinghumanoid controlnavigationgenerative video
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a pre-trained video diffusion model—trained only on passive internet video—can be turned into a world model that predicts futures contingent on the agent's actions, without training a bespoke model from scratch. The recipe is to encode action sequences into embeddings and inject them into the model's existing timestep-conditioning pathway via lightweight scale-and-shift layers. If this works, world modeling becomes a fine-tuning problem rather than a data-collection problem: modest action-observation data plus rich internet priors yields models that follow commands, generalize across embodiments, and even simulate navigation inside paintings. The paper also introduces a metric, Structural Consistency Score (SCS), intended to measure action-following independently of visual appearance, and reports up to 65% improvement over a prior state-of-the-art navigation world model.

Core claim

By adding an action embedding to the denoising timestep embedding at every modulation block, the authors keep the base video diffusion architecture untouched and still achieve fine-grained control over the generated future. The conditioning mechanism is deliberately generic: MLPs project actions of any dimensionality (3-DoF or 25-DoF) into a shared latent space, and 1D convolutions align them with temporally compressed latents. On navigation datasets the approach beats the specialized Navigation World Model on LPIPS, DreamSim, and SCS at all horizons, and on 25-DoF humanoid navigation and manipulation it produces coherent rollouts; the pretrained variant substantially outperforms a from-scra

What carries the argument

The timestep-conditioning pathway: in video diffusion models, the denoising timestep embedding modulates network activations through learned scale, shift, and gate projections. EgoWM piggybacks on this universal pathway by adding action embeddings (and, for humanoids, the initial agent state embedding) to the timestep embedding at every modulation block, so the same mechanism works across U-Net and DiT backbones. The action projection module and the SCS metric are secondary machinery: the former maps arbitrary action spaces into the latent, the latter tracks hand-selected stable objects with SAM2 and averages mask IoU between predicted and ground-truth frames.

Load-bearing premise

The entire evaluation rests on SCS being a faithful measure of action-following; if the manual selection of 'key scene structures' or the frame-trimming rule introduces bias, the reported 65% improvement over NWM may not hold.

What would settle it

Take a fine-tuned EgoWM model and swap the true action sequence for a semantically inverted one (e.g., rotate clockwise instead of counter-clockwise) while keeping all other inputs fixed. If SCS does not collapse relative to rollouts with correct actions, SCS is not capturing action-following. A complementary check: replace the manual object selection in SCS with an automatic criterion (e.g., all static points from a tracker) and see whether the reported gap to NWM survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • World models for new embodiments or tasks reduce to fine-tuning an existing video generator with a modest amount of paired action–observation data, sidestepping large-scale data collection.
  • High-dimensional control signals (25-DoF joint angles) can drive coherent egocentric video prediction, opening the door to humanoid planning from pixels.
  • Because the base architecture is unchanged, improvements in video diffusion backbones transfer directly to world modeling performance.
  • SCS provides a way to benchmark action-following separately from visual fidelity, useful for future world-model comparisons.
  • The lower latency of parallel-frame generation (vs autoregressive rollout) makes closed-loop planning more practical.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's observation that SVD (no temporal compression) follows actions more faithfully than Cosmos (temporal compression) suggests an untested hypothesis: temporal downsampling of the action channel may blur fast or precise motions; one could test by reducing the compression factor k and measuring SCS.
  • If the timestep pathway is a general conditioning bus, then other control signals (e.g., IMU readings, language commands, future goal images) might be injected the same way; that is my extension, not the paper's claim.
  • The painting-generalization result hints that large video priors encode abstract spatial-navigation dynamics independent of photorealism; a testable extension would be to measure SCS in procedurally-generated abstract mazes where ground-truth structure is known.
  • SCS's dependence on manually marking objects limits automated development; a learned regressor trained on SCS-like labels could replace the manual step, which the paper does not propose.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EgoWM, a method that converts pre-trained video diffusion models into action-conditioned world models by injecting action embeddings through the timestep-conditioning pathway. It evaluates on 3-DoF navigation (RECON, SCAND, TartanDrive) and 25-DoF humanoid navigation/manipulation (1X), comparing against Navigation World Models (NWM). The paper also introduces a new Structural Consistency Score (SCS) to measure action-following independent of appearance, claims up to 65% SCS improvement over NWM, and shows qualitative generalization to paintings and unseen real scenes.

Significance. If the results hold, the paper provides a simple and potentially scalable recipe for turning large passive video-generation models into action-conditioned world models, with evidence spanning embodiments, action-space dimensionality, and out-of-distribution visual domains. The architecture-agnostic conditioning mechanism is attractively lightweight, and the inclusion of a from-scratch ablation strengthens the pretraining claim. However, the central quantitative evidence for action-following relies on the newly proposed SCS metric, whose manual components demand additional validation and reproducibility support before the headline improvements can be fully accepted.

major comments (3)
  1. [Section 4, Eq. (7)] The SCS metric is the main quantitative evidence for the claim of improved action-following. Its computation requires manual marking of 'key scene structures' and a frame-trimming rule based on dense point tracking; the paper provides no inter-annotator agreement, no sensitivity analysis of the manual selection, and no release of evaluation annotations or scripts. The trimming rule may remove different frames for different methods, and SAM2 tracking accuracy on generated (possibly distorted) videos is only shown qualitatively. Since the headline 'up to 65%' improvement over NWM is computed with SCS, this is load-bearing. Please add metric-reliability evidence (e.g., annotator agreement, robustness to object selection, comparison with a fully automatic point-trajectory variant, or synthetic validation with known actions) and release the exact evaluation protocol.
  2. [Abstract vs. §1 and Table 1] The Abstract states 'improves SCS by up to 65%' and says the method 'applies seamlessly to three different video diffusion model architectures,' while the Introduction states 'up to 80%' and the implementation section describes two backbones (SVD and Cosmos). Table 1 supports at most a 65% relative improvement (EgoWM/SVD at frame 16: (55.2−33.4)/33.4 = 65.3%); no 80% result appears anywhere. The 'three architectures' claim is unsupported by the experiments. Please correct these figures and either demonstrate a third backbone or soften the claim to match the evidence.
  3. [Section 5.1 and Table 3] The comparison with NWM is described as 'for a fair comparison,' but Table 3 lists NWM as trained on RECON, SCAND, Tartan-Drive, and Huron, while the proposed method uses only the first three. It is unclear whether the released NWM checkpoint (trained with additional data) was evaluated directly, and what exactly makes the comparison 'fair.' If the authors intend to compare against the publicly available checkpoint, this should be stated explicitly and the wording adjusted. This does not necessarily change the conclusion, but the current description is ambiguous.
minor comments (5)
  1. [Section 4] The text says SCS 'automatically identify[ies] stable scene structures' but a few sentences later says 'we manually mark key scene structures.' Please reconcile these statements.
  2. [Figure 5 / Table 3] Latency values are reported without standard deviations or a precise hardware configuration (beyond 'a single A100'). Adding this information would improve reproducibility.
  3. [References] Reference [1] ('1X World Model Challenge, 2024') lacks a URL and access date; please complete the entry.
  4. [Appendix C] The statement 'We believe... can be solved as the base models get better' is speculative; it would be clearer to separate observed limitations from hypotheses.
  5. [General] No code or evaluation-script release is mentioned. For a paper introducing a new metric, releasing the SCS evaluation pipeline and annotations would substantially aid reproducibility.

Circularity Check

0 steps flagged

No significant circularity: EgoWM's action-conditioning is trained with a standard denoising objective, and its headline gains are not forced by SCS or by self-citation.

full rationale

The central derivation is empirical: EgoWM injects action embeddings into the timestep-conditioning pathway of a pre-trained video diffusion model (Eqs. 4-6) and fine-tunes with the standard denoising objective (Eq. 3). Nothing in this objective is fitted to SCS, and SCS (Eq. 7) is a tracking-based mask-IoU computed on predicted versus ground-truth videos; it contains no EgoWM parameters and is not optimized during training. A static or action-ignoring baseline would not score perfectly, so SCS is not satisfied by construction. The manual selection of 'key scene structures' and the trimming rule are validity/reproducibility concerns about the metric, not evidence that the metric reduces to the model's outputs. The paper also reports improvements on the external, established LPIPS and DreamSim metrics, so the central comparison does not rest solely on the self-proposed metric. The only overlapping-author citation is AllTracker (Harley et al. [16]), used to detect when points leave the field of view; this is a published external tracking tool, not a load-bearing self-citation justifying the method. The controlled SVD-versus-SVD-from-scratch comparison further supports the pretraining claim independently of SCS. Appendix C candidly documents failure modes, which is inconsistent with a derivation rigged to its own evaluation. No specific reduction of a claimed result to its inputs—by equation or by self-citation chain—could be identified, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The ledger is clean of invented physical entities. The method's load-bearing assumptions are (i) that internet-prior video models are sufficient substrates for action-conditioned world models and (ii) that the SCS metric is a valid, unbiased measure of action-following. The first is partially supported by the from-scratch ablation; the second is not externally validated.

free parameters (3)
  • SCS trimming threshold / manual object selection
    In Section 4, frames where all initially-visible points leave the FOV are trimmed using AllTracker, and 'key scene structures' are manually chosen; the exact selection and threshold are not specified numerically, shaping the resulting SCS.
  • Action projection MLP architecture
    Section 3.2 and Implementation details: the action is embedded with sine-cosine features followed by an MLP (SVD) or a more expressive MLP (Cosmos), with 1D conv downsampling to temporal latent resolution; details like layer count/width are hand-chosen.
  • Learning-rate multiplier for action layers = 10x
    Section 5 Impl. details: action projection layers use 10x the base model's learning rate, chosen by hand.
axioms (4)
  • domain assumption Pre-trained video diffusion models contain usable world priors (physics, scene structure) from internet-scale data that can be repurposed for robot action prediction with fine-tuning.
    This is the central premise of the paper, stated in Section 1 and implicitly everywhere; without it, the method would not outperform training from scratch (the scratch baseline in Table 2 partially supports it, but the premise remains an assumption about the base models).
  • ad hoc to paper Adding action embeddings to the timestep-conditioning pathway (Eq. 5) is sufficient to make a video diffusion model action-conditioned without corrupting the pre-trained representation.
    Section 3.2, Eq. (5): the design reuses the timestep modulation blocks; this is a design choice, not a derived guarantee, and is only empirically validated on two backbones.
  • ad hoc to paper The Structural Consistency Score, computed via mask IoU on manually-selected stable structures after frame trimming, faithfully measures action-following independent of appearance.
    Section 4 defines SCS and argues it correlates with action-following via Figure 3; there is no external validation or established benchmark showing SCS is unbiased.
  • standard math Standard mathematics of latent video diffusion and VAEs, including the denoising objective in Eq. (3).
    Section 3.1 relies on known diffusion-model math (Gaussian denoising, latent VAE) as background.

pith-pipeline@v1.3.0-alltime-deepseek · 13673 in / 12422 out tokens · 110996 ms · 2026-08-03T08:51:24.106695+00:00 · methodology

0 comments
read the original abstract

What if a video generation model could not only imagine a plausible future, but the correct one -- accurately reflecting how the world changes with each action? We answer this by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pre-trained video diffusion model into an action-conditioned world model, enabling precisely controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models by injecting appropriately compressed motor commands through lightweight conditioning layers. This allows our model to follow actions faithfully while preserving generalization and realism. Our approach scales naturally across embodiments and action spaces -- from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation, requiring only modest fine-tuning. To evaluate physical correctness independent of appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. Our method improves SCS by up to 65\% over the prior state of the art, Navigation World Models; applies seamlessly to three different video diffusion model architectures; and effectively utilizes Internet priors to generalize to unseen environments, including navigation and manipulation inside paintings. Finally, we demonstrate the applicability of EgoWM to robotic planning.

Figures

Figures reproduced from arXiv: 2601.15284 by Anurag Bagchi, Homanga Bharadhwaj, Martial Hebert, Pavel Tokmakov, Yu-Xiong Wang, Zhipeng Bao.

Figure 1
Figure 1. Figure 1: Our framework generates future frame predictions (shown in blue) that accurately follow the provided robot actions (ground-truth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our method embeds action sequences or arbitrary dimensionality into a universal feature space and injects these embeddings [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Each column shows two generated rollouts compared to the corresponding ground-truth frame, with their LPIPS, DreamSim, and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on ego-centric navigation. Both variants of our model generate realistic, temporally coherent sequences that [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The inference latency of autoregressive NWM is much [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Our action-conditioned video world model enables a humanoid agent to navigate and execute long-horizon, contact-rich manip [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Our model generalizes to unrealistic visual domains, such as navigating inside paintings, while still accurately following action [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Our method generalizes to real-world scenes captured in our campus, while following 25-DoF navigation commands. Here we [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: This is an illustration of our SCS metric computation. The first two rows show the difference between predicted locations of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Failure modes of our world model. In the first two rows we show shape inconsistencies in our manipulation results, in the third [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

    cs.CV 2026-05 unverdicted novelty 7.0

    ACWM-Phys is a controllable simulator benchmark with in- and out-of-distribution protocols for evaluating action-conditioned world models across rigid, kinematic, deformable, and particle dynamics.

  2. How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction

    cs.CV 2026-05 unverdicted novelty 6.0

    TrajPilot predicts candidate future trajectories from egocentric context and uses them to condition action prediction in an embedding space, outperforming VLM and planner baselines on Ego-Exo4D, Ego4D, and other datas...

  3. ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

    cs.CV 2026-05 unverdicted novelty 6.0

    ACWM-Phys benchmark shows action-conditioned world models generalize on simple geometric interactions but drop sharply on deformable contacts, high-dimensional control, and complex articulated motion, indicating relia...

  4. Human Cognition in Machines: A Unified Perspective of World Models

    cs.RO 2026-04 unverdicted novelty 6.0

    The paper introduces a unified framework for world models that fully incorporates all cognitive functions from Cognitive Architecture Theory, highlights under-researched areas in motivation and meta-cognition, and pro...

  5. Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints

    cs.CV 2026-03 unverdicted novelty 6.0

    A new occlusion-aware control module generates high-fidelity egocentric videos from sparse 3D hand joints, supported by a million-clip dataset and cross-embodiment benchmark.

  6. Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints

    cs.CV 2026-03 unverdicted novelty 6.0

    Sparse 3D hand joints plus an occlusion-aware control module produce higher-fidelity, 3D-consistent egocentric hand-object videos than dense-2D or implicit-pose baselines.

  7. WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL

    cs.RO 2026-02 conditional novelty 6.0

    WoVR shows that reinforcement learning can improve VLA robot policies through imagined rollouts in a video world model, reporting +29.3 points on LIBERO and +30.0 points on real Franka tasks.

  8. Drift-Resistant Navigation World Model with Anchored Epipolar Guidance

    cs.CV 2026-05 unverdicted novelty 5.0

    A generative navigation world model that uses sparse anchored rollout with epipolar constraints to reduce perceptual and geometric drift.

  9. Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

    cs.CV 2026-05 unverdicted novelty 2.0

    This survey reviews trends, challenges, benchmarks, and future directions in action-conditioned interactive world modeling for video and 3D generation.

Reference graph

Works this paper leans on

55 extracted references · 12 linked inside Pith · cited by 7 Pith papers

  1. [1]

    1X World Model Challenge, 2024

    1X Technologies. 1X World Model Challenge, 2024. 4, 6

    2024

  2. [2]

    Cosmos world foun- dation model platform for physical AI.arXiv preprint arXiv:2501.03575, 2025

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foun- dation model platform for physical AI.arXiv preprint arXiv:2501.03575, 2025. 2, 3, 6, 13

    Pith/arXiv arXiv 2025

  3. [3]

    Diffusion for world modeling: Visual details matter in Atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos J Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in Atari. NeurIPS, 2024. 2, 3

    2024

  4. [4]

    Stochastic variational video prediction.arXiv preprint arXiv:1710.11252, 2017

    Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction.arXiv preprint arXiv:1710.11252, 2017. 3

    Pith/arXiv arXiv 2017

  5. [5]

    ReferEverything: Towards seg- menting everything we can speak of in videos

    Anurag Bagchi, Zhipeng Bao, Yu-Xiong Wang, Pavel Tok- makov, and Martial Hebert. ReferEverything: Towards seg- menting everything we can speak of in videos. InICCV,

  6. [6]

    ReCamMaster: Camera-controlled generative rendering from a single video

    Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. ReCamMaster: Camera-controlled generative rendering from a single video. InICCV, 2025. 3 9 Groundtruth 25-DoF NavigationInit Frame Real World ScenePredicted 25-DoF Navigation Groundtruth 25-DoF NavigationInit Frame Real World Scene...

    2025

  7. [7]

    Navigation world models

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. InCVPR, 2025. 2, 3, 5, 6, 13

    2025

  8. [8]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 3, 6, 13

    Pith/arXiv arXiv 2023

  9. [9]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Jo ˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. InCVPR,

  10. [10]

    VideoCrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffusion models. InCVPR, 2024. 3

    2024

  11. [11]

    Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. 10 arXiv preprint arXiv:1812.00568, 2018. 3

    Pith/arXiv arXiv 2018

  12. [12]

    The PASCAL visual ob- ject classes (VOC) challenge.International Journal of Com- puter Vision, 88(2):303–338, 2010

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual ob- ject classes (VOC) challenge.International Journal of Com- puter Vision, 88(2):303–338, 2010. 5

    2010

  13. [13]

    Dream- Sim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- Sim: Learning new dimensions of human visual similarity using synthetic data. InNeurIPS, 2023. 5, 6

    2023

  14. [14]

    Ctrl-world: A controllable generative world model for robot manipulation, 2025

    Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation, 2025. 2, 3, 4

    2025

  15. [15]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. In NeurIPS, 2018. 2, 3

    2018

  16. [16]

    Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Am- brus, Katerina Fragkiadaki, and Leonidas J. Guibas. All- Tracker: Efficient dense point tracking at high resolution. In ICCV, 2025. 5

    2025

  17. [17]

    Grndctrl: Grounding world models via self-supervised reward alignment, 2025

    Haoyang He, Jay Patrikar, Dong-Ki Kim, Max Smith, Daniel McGann, Ali akbar Agha-mohammadi, Shayegan Omid- shafiei, and Sebastian Scherer. Grndctrl: Grounding world models via self-supervised reward alignment, 2025. 2, 3

    2025

  18. [18]

    ImaGen Video: High definition video generation with diffusion mod- els.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. ImaGen Video: High definition video generation with diffusion mod- els.arXiv preprint arXiv:2210.02303, 2022. 3

    Pith/arXiv arXiv 2022

  19. [19]

    Video prediction policy: A generalist robot policy with predictive visual representations

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. InICML,

  20. [20]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation.IEEE Robotics and Automation Letters, 7 (4):11807–11814, 2022

    Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett War- nell, S ¨oren Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, and Peter Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation.IEEE Robotics and Automation Letters, 7 (4):11807–11814, 2022. 6, 13

    2022

  21. [21]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. InCVPR, 2024. 3

    2024

  22. [22]

    Learning to simulate dynamic envi- ronments with GameGAN

    Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Tor- ralba, and Sanja Fidler. Learning to simulate dynamic envi- ronments with GameGAN. InCVPR, 2020. 3

    2020

  23. [23]

    Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013. 4

    Pith/arXiv arXiv 2013

  24. [24]

    Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025

    Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sud- hakar, Paarth Shah, Rares Ambrus, and Carl V ondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025. 3

    Pith/arXiv arXiv 2025

  25. [25]

    Zero-1-to-3: Zero-shot one image to 3D object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3D object. InICCV, 2023. 3

    2023

  26. [26]

    EvalCrafter: Benchmarking and eval- uating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. EvalCrafter: Benchmarking and eval- uating large video generation models. InCVPR, 2024. 2

    2024

  27. [27]

    Mannos and David J

    James L. Mannos and David J. Sakrison. The effects of a visual fidelity criterion on the encoding of images.IEEE Trans. Information Theory, 20(4):525–536, 1974. 3

    1974

  28. [28]

    Internal models in biological control.Annual review of control, robotics, and autonomous systems, 2(1):339–364, 2019

    Daniel McNamee and Daniel M Wolpert. Internal models in biological control.Annual review of control, robotics, and autonomous systems, 2(1):339–364, 2019. 2

    2019

  29. [29]

    Action-conditional video prediction us- ing deep networks in Atari games.NeurIPS, 2015

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction us- ing deep networks in Atari games.NeurIPS, 2015. 3

    2015

  30. [30]

    pix2gestalt: Amodal segmentation by synthesizing wholes

    Ege Ozguroglu, Ruoshi Liu, Dıdac Surıs, Dian Chen, Achal Dave, Pavel Tokmakov, and Carl V ondrick. pix2gestalt: Amodal segmentation by synthesizing wholes. InCVPR,

  31. [31]

    Learning view-invariant world models for vi- sual robotic manipulation

    Jing-Cheng Pang, Nan Tang, Kaiyuan Li, Yuting Tang, Xin- Qiang Cai, Zhen-Yu Zhang, Gang Niu, Masashi Sugiyama, and Yang Yu. Learning view-invariant world models for vi- sual robotic manipulation. InICLR, 2025. 3

    2025

  32. [32]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, 2023. 2, 3, 4

    2023

  33. [33]

    Active inference as a theory of sentient behavior.Biological Psy- chology, 186:108741, 2024

    Giovanni Pezzulo, Thomas Parr, and Karl Friston. Active inference as a theory of sentient behavior.Biological Psy- chology, 186:108741, 2024. 2

    2024

  34. [34]

    SAM 2: Seg- ment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Seg- ment anything in images and videos. InICLR, 2025. 5

    2025

  35. [35]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 4

    2022

  36. [36]

    U- Net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- Net: Convolutional networks for biomedical image segmen- tation. InMICCAI, 2015. 2, 3, 4

    2015

  37. [37]

    Masked world models for visual control

    Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. InCoRL, 2023. 2

    2023

  38. [38]

    Rapid exploration for open- world navigation with latent goal models

    Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open- world navigation with latent goal models. InCoRL, 2021. 6

    2021

  39. [39]

    Tar- tanDrive: A large-scale dataset for learning off-road dynam- ics models

    Samuel Triest, Matthew Sivaprakasam, Sean J Wang, Wen- shan Wang, Aaron M Johnson, and Sebastian Scherer. Tar- tanDrive: A large-scale dataset for learning off-road dynam- ics models. InICRA, 2022. 6, 13

    2022

  40. [40]

    Towards accurate generative models of video: A new metric and challenges

    Alexander Unterthiner, Sjoerd van Steenkiste, Daniel Key- sers, Thomas Kipf, Alexander D’Amour, Peter Sorrenson, and Olivier Bousquet. Towards accurate generative models of video: A new metric and challenges. InNeurIPS, 2018. 2, 3

    2018

  41. [41]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2, 3

    Pith/arXiv arXiv 2024

  42. [42]

    Generative camera dolly: Ex- treme monocular dynamic novel view synthesis

    Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sar- gent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi 11 Zheng, and Carl V ondrick. Generative camera dolly: Ex- treme monocular dynamic novel view synthesis. InECCV,

  43. [43]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025. 3

    Pith/arXiv arXiv 2025

  44. [44]

    ModelScope text-to-video technical report.arXiv preprint arXiv:2308.06571, 2023

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope text-to-video technical report.arXiv preprint arXiv:2308.06571, 2023. 3

    Pith/arXiv arXiv 2023

  45. [45]

    Drivedreamer: Towards real-world- drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- drive world models for autonomous driving. InECCV, 2024. 2

    2024

  46. [46]

    PredRNN: A recurrent neural network for spatiotemporal predictive learn- ing.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2208–2225, 2022

    Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jian- min Wang, Philip S Yu, and Mingsheng Long. PredRNN: A recurrent neural network for spatiotemporal predictive learn- ing.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2208–2225, 2022. 3

    2022

  47. [47]

    Bovik, Hamid R

    Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE Trans. Image Processing, 13 (4):600–612, 2004. 3

    2004

  48. [48]

    Benchmarking image similarity metrics for novel view syn- thesis applications.arXiv preprint arXiv:2506.12563, 2025

    Charith Wickrema, Sara Leary, Shivangi Sarkar, Mark Giglio, Eric Bianchi, Eliza Mace, and Michael Twardowski. Benchmarking image similarity metrics for novel view syn- thesis applications.arXiv preprint arXiv:2506.12563, 2025. 2

    Pith/arXiv arXiv 2025

  49. [49]

    Video models are zero-shot learn- ers and reasoners.arXiv preprint arXiv:2509.20328, 2025

    Thadd ¨aus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learn- ers and reasoners.arXiv preprint arXiv:2509.20328, 2025. 2

    Pith/arXiv arXiv 2025

  50. [50]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. InICLR, 2025. 3

    2025

  51. [51]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InCVPR, 2018. 2, 3, 5, 6

    2018

  52. [52]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. InICCV, 2023. 3

    2023

  53. [53]

    Unified world mod- els: Coupling video and action diffusion for pretraining on large robotic datasets

    Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burch- fiel, Paarth Shah, and Abhishek Gupta. Unified world mod- els: Coupling video and action diffusion for pretraining on large robotic datasets. InRSS, 2025. 2, 3

    2025

  54. [54]

    IRASim: A fine-grained world model for robot manipulation

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. InICCV, 2025. 2, 3, 4

    2025

  55. [55]

    Exploring pre-trained text- to-video diffusion models for referring video object segmen- tation

    Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text- to-video diffusion models for referring video object segmen- tation. InECCV, 2024. 3 12 Appendix In this appendix, we report additional results, details, and visualizations. We begin by discussing our Structural Con- sistency Metric in Section A. N...

    2024