Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Back to the Features: DINO as a Foundation for Video World Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DINO-world predicts future frames inside a frozen DINOv2 feature space and outperforms prior world models on dense forecasting.

desk verdict Solid scaling result in latent-space world models; the headline forecast numbers need a motion-based check to rule out blur-tolerance artifacts. read the letter →

arxiv 2507.19468 v1 pith:4SP77SIT submitted 2025-07-25 cs.CV

Federico Baldassarre , Marc Szafraniec , Basile Terver , Vasil Khalidov , Francisco Massa , Yann LeCun , Patrick Labatut , Maximilian Seitzer
show 1 more author
Piotr Bojanowski
This is my paper
classification cs.CV
keywords videoworldmodelslatent-spacepredictionDINOv2self-supervisedvisualfeaturesfutureframeforecastingsegmentationintuitivephysicsaction-conditionedplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Future video modeling can be done without generating pixels. This paper claims that a world model can predict the temporal evolution of a scene directly in the frozen feature space of a self-supervised image encoder, DINOv2, and that this suffices to forecast downstream quantities such as semantic segmentation and depth. The resulting model, DINO-world, is trained on roughly 66 million uncurated web videos with a cross-attention predictor of up to 1.1 billion parameters, and the paper reports that it outperforms pixel-based generative world models such as COSMOS and jointly trained latent models such as V-JEPA on dense forecasting benchmarks, while showing comparable intuitive-physics understanding. The same unconditional model can be fine-tuned for action-conditioned planning by adding small zero-initialized modules. If these claims hold, generalist world models become substantially cheaper to train and easier to adapt to control.

What carries the argument

The machine at the heart of the method is a transformer future-predictor built from pre-norm cross-attention blocks. For every future timestamp and spatial location, a learnable query token cross-attends to all past patch tokens and then passes through an MLP, with rotary position embeddings (RoPE) applied along three axes: absolute time in seconds, and normalized horizontal and vertical coordinates. A block-triangular attention mask lets all queries for all frames be trained in parallel while preserving causality. The predictor is trained with a smooth L1 loss on next-frame prediction, with time deltas sampled uniformly so that the model learns to respond to arbitrary future timestamps. For action conditioning, zero-initialized MLP blocks update each query with the corresponding action, preserving the pretrained video model and allowing the action blocks alone to be trained when action-labelled data is scarce.

What would settle it

On a benchmark with large camera motion, occlusion, or fast object deformation, compute forecasting accuracy with a copy-last-frame baseline as the comparison point: if predicted features at a 0.5-second horizon do not beat that baseline under the same linear heads, the claim that the frozen feature space carries sufficient dynamics collapses.

Watch

Extended reading notes

Core claim

The central discovery is that the last-layer patch tokens of a frozen DINOv2 encoder form a state space rich enough for a generalist future predictor. The model is trained to map past feature tokens and timestamps, together with a future coordinate, to the feature token at that coordinate, using a smooth L1 loss on next-frame predictions with uniform sampling of time deltas. At evaluation, linear heads trained on present-day features are applied directly to predicted features, and on VSPW the model reaches 6.3 percentage points higher mIoU than the second-best baseline at 0.5-second forecasting; it also improves depth forecasting on KITTI and matches much larger models on physics plausibility. A second discovery is that adding zero-initialized action blocks after each cross-attention block lets the pretrained unconditional model be fine-tuned on small observation-action datasets, and planning by rolling out candidate action sequences in latent space outperforms training the same predictor from scratch.

Load-bearing premise

The load-bearing premise is that the last-layer patch tokens of a frozen DINOv2 encoder preserve enough visual and motion information that future states can be predicted in that space and read out by linear heads trained on present frames.

Editorial extensions

If this is right

  • A generalist video world model can be trained with under one billion predictor parameters and a frozen encoder, in contrast to pixel-space generative world models that require billions of parameters and orders of magnitude more training compute.
  • Linear heads trained on present-day DINOv2 features transfer to predicted features, so downstream tasks such as segmentation and depth can be forecast without retraining task decoders on future data.
  • The same unconditional model can be post-trained for control by adding small action modules to each block, and this yields better planning success rates than training a predictor from scratch on the same offline trajectories.
  • Because the model takes explicit timestamps, it can predict at arbitrary frame rates and extrapolate either directly or autoregressively, with direct prediction preferred at short horizons and rollout preferred at long horizons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is whether the same frozen-feature recipe transfers to other self-supervised image encoders or to spatiotemporal encoders; the paper's own ablation with SigLIP2 suggests representation quality, not just model scale, determines forecasting accuracy.
  • The explicit-timestamp design points toward irregularly sampled sensor streams rather than uniform video only, because the query carries absolute time and the context can have arbitrary spacing.
  • The planning results are demonstrated on simple 2D environments; a natural next test is whether the fine-tuning recipe holds on high-dimensional robotic manipulation or driving, where offline action data is scarce and large-scale pretraining should matter more.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DINO-world, a latent-space video world model that predicts future frames in the frozen feature space of DINOv2. The predictor is a cross-attention transformer with 3-axial rotary position encodings, trained with a smooth L1 next-frame objective on a private corpus of about 66M uncurated videos. The manuscript evaluates the model on dense forecasting of segmentation and depth, on three intuitive-physics benchmarks, and on action-conditioned planning after fine-tuning with zero-initialized action blocks. The main claims are that frozen DINOv2 features form a suitable state space for world modeling, that large-scale uncurated video pretraining yields a generalist predictor, and that the resulting model can be adapted for planning.

Significance. If the central claims hold, this is a practically important result: a latent-space predictor on frozen self-supervised features offers a much cheaper alternative to pixel-space generative world models and still supports downstream dense-prediction and planning tasks. The paper is strong in scope and explicitness: it documents the architecture, training schedule, and ablation protocols in detail; it compares across latent-space and pixel-space paradigms; and it tests the model on multiple task families. The ablations of predictor size, training data, and encoder choice are useful and well motivated. However, the headline evidence that the model 'learns the temporal dynamics of diverse scenes' rests on dense-forecasting and physics-surprise proxies that do not isolate motion modeling from the blurring behavior of a deterministic L1-predictor, and the comparative claims are made without error bars. These issues need to be addressed before the central contribution can be considered fully supported.

major comments (3)
  1. [§4.1, Table 1] The dense-forecasting protocol applies linear heads trained on present-time encoder features to predicted future features. Because the predictor is trained deterministically with a smooth L1 loss (Eq. 6) and the paper itself reports that rollouts become blurry at longer horizons (Sec. 4.2, Fig. 3; Sec. E.1), the predicted features will tend toward a conditional-mean feature. Such a blurred feature can improve linear-probe metrics over the sharp last observed frame by overlapping future object locations or by remaining on the correct side of decision boundaries without modeling object motion. The reported gains over the 'Copy Last' baseline in Table 1 therefore do not by themselves establish that temporal dynamics are learned. Please add a direct feature-fidelity measure (e.g., cosine similarity or L1 distance between predicted features and the actual future encoder features), a motion-sensitive evaluation (e.g., optical flow, point/instance tracking, or a metric that rewards correct displacement), and a 'blurred last-feature' or conditional-mean baseline. These additions are needed to support the abstract claim that 'DINO-world learns the temporal dynamics of diverse scenes.'
  2. [§4, Tables 1–4] All headline results are single runs with no error bars, confidence intervals, or significance statements. For example, the KITTI mid-term RMSE difference between DINO-world (4.268) and COSMOS-12B (4.617), and the PointMaze success-rate difference between Fine-tuned (68.7) and Action-only (61.6), are modest relative to the likely variation. The planning results in Table 4 are averaged over 512 episodes, so standard errors could be reported with little extra compute. Please provide seed variance (at least three seeds for the main comparisons, or an explicit statement where that is infeasible) and standard errors for the planning results, so that the claimed outperformance is statistically grounded.
  3. [§4.1, Table 2 / Appendix E.2] The intuitive-physics evaluation is currently presented as a 'sanity check' rather than a benchmark, yet the abstract claims 'strong understanding of intuitive physics.' The mapping from prediction error to the reported 'mean relative accuracy' is not fully specified, and some per-category results are degenerate (e.g., DINO-world and DINO-Foresight both score 0.0 on GRASP Collision in Table 9b). Please describe the exact surprise-to-accuracy procedure used to compute Table 2, report the full category-level protocol, and either fix the metric or temper the physics claim in the abstract so that it matches the evidence provided.
minor comments (5)
  1. [Abstract vs. Appendix C.1] The abstract states that VSPW segmentation forecasting is evaluated at 0.5 seconds, but the appendix protocol for VSPW mid-term forecasting targets frame 19 from frames [1, 4, 7, 10], which is 600 ms later; please align the stated prediction horizons.
  2. [Figure 3] The y-axis and the exact meaning of the numbers '55' and '65' in the caption are unclear; please label the axis and state the metric explicitly.
  3. [Appendix B] The text refers to 'IntPhys [21]' in the description of Figure 2, but the IntPhys dataset is reference [64] in the bibliography; please correct the citation.
  4. [§3.2] The phrase 'Considering the input space, we use RoPE periods in the range [10^{-2}, 10^{2}]' is ambiguous; it should state that this range applies to the temporal and spatial coordinate encodings, as clarified in Eq. (5).
  5. [§4.3 / Table 4] The planning section compares to the DINO-WM setup but reduces the CEM 'top actions' parameter K from 30 to 10 without an ablation; please justify this choice or report its sensitivity, since it can affect the planning success rate.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: external benchmarks and separate linear probes; self-citations are non-load-bearing.

full rationale

The paper's central derivation chain is not circular. DINO-world trains a future predictor with a smooth-L1 regression loss (Eq. 4, Eq. 6) on future DINOv2 features; the reported benchmarks are external: dense forecasting uses linear heads trained separately on present-time ground-truth labels (Sec. 4.1, C.3) and applied to predicted features, planning success is measured by executing actions in the true environments (Sec. 4.3, D.3), and intuitive physics uses direct MAE between predicted and actual features (Sec. 4.1). None of these evaluation targets appear in the predictor's training objective, and no fitted parameter is renamed as a prediction. DINOv2 is an externally released, independently pretrained encoder; its use as the frozen state space is a design choice validated by the encoder ablation (Table 3, right) rather than an imported conclusion. The several co-authored references (DINOv2 [23], V-JEPA [24], DINO-WM [18], IntPhys protocol [21]) are used as baselines, protocols, or pretrained components, not as unverified uniqueness or derivation premises, so they do not make the argument circular. The paper itself flags its own limitations—rollouts become blurry at long horizons ('the longer the temporal horizon, the higher the uncertainty, and the more the predictions become blurry', Sec. 4.2, Fig. 2) and forecasting degrades near 1 s ('all predictions become inaccurate as the forecasting interval approaches 1 second', Fig. 3)—and treats the physics benchmarks as a 'sanity check rather than a benchmark' (Sec. 4.1). These are correctness/scope concerns about whether the dense-forecasting gains reflect true learned dynamics rather than blur-tolerant probes; they are not instances of a claim reducing to its inputs by construction. Accordingly, no circular step is identified; the score of 1 reflects the presence of overlapping self-citations that are not load-bearing.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method rests on a few domain assumptions about the DINOv2 state space and the training objective, all standard for empirical representation learning. The main risk is the unstated bounds on the time-delta sampling and the private pre-training dataset, which are free choices not tied to external evidence.

free parameters (5)
  • Smooth L1 loss beta = 0.1
    Huber threshold in Eq. 6, chosen by hand; standard for robust feature regression.
  • RoPE angular periods = 10 periods logarithmically spaced from 1e-2 to 1e2
    Chosen to cover temporal positions in seconds and normalized spatial positions in [-1,1]; affects relative position encoding.
  • Time-delta sampling bounds [Delta_tau_min, Delta_tau_max] = not reported
    Central to variable-FPS training; the paper defines the range in Section 3.2 but never gives the values, a reproducibility gap.
  • Planning CEM hyperparameters (N, H, f, K, J) = N=300, H=25, f=5, K=10, J=30
    Adopted from DINO-WM with K lowered from 30 to 10; directly affects planning success rates.
  • Fine-tuning epochs and frameskip = 25 epochs, frameskip=5
    Chosen for action-conditioned fine-tuning; frameskip is a modeling choice affecting temporal resolution.
assumptions (6)
  • domain assumption DINOv2 last-layer patch features form a sufficient and linearly decodable state representation for future prediction.
    Invoked in Section 3.1 and Section 4.1, where linear heads trained on present features are applied to predicted features. If this fails, the method and the evaluation both collapse.
  • domain assumption A smooth L1 loss on predicted versus actual DINOv2 features is a good surrogate for downstream task performance.
    Equation 6 defines the only training signal; there is no theory connecting feature-space L1 to semantic segmentation or depth accuracy.
  • domain assumption Prediction error in latent space is a valid proxy for physical plausibility.
    Section 4.1 defines surprise as mean absolute error between predicted and actual features; the authors themselves call the physics benchmarks a sanity check, so this is a heuristic.
  • domain assumption Large-scale uncurated web video diversity, not curation, drives generalization.
    Section 4.2 compares Cityscapes and SSv2 with the 66M-video set; the difference could also be due to pre-training compute or domain shift.
  • domain assumption Teacher-forced next-frame prediction with uniformly sampled time deltas generalizes to arbitrary future timestamps.
    Section 3.2; Figure 3 shows direct prediction degrades at long horizons, so this assumption holds only within a limited range.
  • ad hoc to paper Zero-initialized action blocks preserve pre-trained video knowledge during fine-tuning.
    Section 3.3 introduces this design; the ablation in Table 4 suggests it works, but it is a specific architectural assumption, not a general principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Back to the Features: DINO as a Foundation for Video World Models." pith.science (2026). https://pith.science/paper/4SP77SIT

@misc{pith2026250719468,
  author       = {Pith},
  title        = {Pith review of: Back to the Features: DINO as a Foundation for Video World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SP77SIT}},
  note         = {Machine review of arXiv:2507.19468}
}
read the original abstract

We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.

Figures

Figures reproduced from arXiv: 2507.19468 by the authors.

Figure 1
Figure 1. Latent video world model architecture. A frozen DINOv2 encoder maps video frames to patch tokens in latent space. The predictor is a stack of cross-attention blocks trained to predict the patch token xˆt+1,i′ ,j′ at a future timestamp τt+1 and location (i ′ , j′ ) from all past tokens x1:t,·,· and timestamps T 1:t. Spatiotemporal coordinates are injected into each token via a 3-axial RoPE. The block-triangular atten… view at source ↗
Figure 2
Figure 2. Autoregressive predictions. For each video, from top to bottom: frames with timestamps, encoder features, autoregressive predictions in latent space. The predictor has access to “ground-truth” encoder features for the first 8–10 frames (mid row, above the gray squares), and then to its own predictions (bottom row, past the gray squares). Latent features are visualized through a PCA projection computed on the encoder… view at source ↗
Figure 3
Figure 3. How far can the model predict? Cityscapes segmentation forecasting performance as context frames are progressively shifted back in time, further away from the target frame 19. We compare direct prediction vs. 1- or 2-step autoregressive prediction. Model PushT Wall PointMaze Scratch 46.9 87.1 59.4 Action-only 49.4 91.1 61.6 Fine-tuned 59.4 93.8 68.7 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pre-training dataset statistics. For our 66M video dataset, we report the joint histogram of height vs. width with highlighted aspect ratios 16:9, 1:1, and 9:16, as well as the distribution of number of frames vs. duration with highlighted 10, 30, 60 FPS. Figures and a…
Figure 5
Figure 5. Figure 5: Unconditional autoregressive rollouts in latent space. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Visualization of cross attentions. We visualize the cross-attention of a single query, to all patches of all previous frames, for two intermediate blocks of the predictor. The selected blocks 6 and 16, zero-indexed, are those where the attention maps are the most infor…

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DF3 forecasts next-frame semantic features entirely inside a frozen ViT via injected queries and motion fusion, then probes those features for segmentation without a conventional decoder.

  2. Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling

    cs.RO 2026-07 conditional novelty 6.0 of 10

    An action-conditioned flow-matching world model detects surgical execution failures by scoring how well an observed 8-step visual outcome transports back to Gaussian noise under nominal dynamics.

  3. Self-Supervised Learning of Structured Dynamics from Videos

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A two-token 'primary/residual' future-feature predictor separates camera from object motion, outpacing frozen-feature baselines and matching larger supervised models on several probes.

  4. Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.

  5. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

Reference graph

Works this paper leans on

70 extracted references · 52 canonical work pages · cited by 5 Pith papers

  1. [1]

    Recurrent world models facilitate policy evolution

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018. 1, 2.1, 5

    work page 2018

  2. [2]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. 1, 2.1, 2.2

    arXiv 2023

  3. [3]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In ICLR, 2023. 1, 2.1, 2.2

    work page 2023

  4. [4]

    Video gen- eration models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video gen- eration models as world simulators. Blog post, 2024. URL https://openai.com/research/ video-generation-models-as-world-simulators . 1, 2.2, 3.1, 5

    work page 2024

  5. [5]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In ICML, 2024. 1, 2.1

    work page 2024

  6. [6]

    Genie 2: A large- scale foundation world model

    Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang, Lei Zhang, Frederic Besse, Tim Harley, Anna ...

    work page 2024

  7. [7]

    Vavim and vavam: Autonomous driving through video generative modeling

    Florent Bartoccioni, Elias Ramzi, Victor Besnier, Shashanka Venkataramanan, Tuan-Hung Vu, Yihong Xu, Loick Chambon, Spyros Gidaris, Serkan Odabas, David Hurych, Renaud Marlet, Alexandre Boulch, Mickael Chen, Éloi Zablocki, Andrei Bursuc, Eduardo Valle, and Matthieu Cord. Vavim and vavam: Autonomous driving through video generative modeling. arXiv preprint...

    arXiv 2025

  8. [8]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. 1, 2.1, 2.2, 3.1, 4, 5

    arXiv 2025

Show all 70 references
  1. [9]

    GAIA-2: A controllable multi-view generative world model for autonomous driving

    Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. GAIA-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025. URL https://arxiv.org/abs/2503.20523. 1, 2.1

    2025 arXiv

  2. [10]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Lu...

    arXiv

  3. [11]

    Wan: Open and advanced large-scale video generative models

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. 1, 5

    2025 arXiv

  4. [12]

    A path towards autonomous machine intelligence, 2022

    Yann LeCun. A path towards autonomous machine intelligence, 2022. 1, 2.2

    2022

  5. [13]

    OpenEQA: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra,...

    2024

  6. [14]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pages 9568–9578, 2024. doi: 10.1109/ CVPR52733.2024.00914. 1

    2024

  7. [15]

    Mastering diverse control tasks through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1–7, 2025. 1, 2.1, 2.2

    2025

  8. [16]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. In NeurIPS, 2025. 1, 2.1, 2.2

    2025

  9. [17]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In ICLR, 2023. 1, 2.1, 2.2

    2023

  10. [18]

    DINO-WM: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024. 1, 2.1, 2.2, 3.3, 4.3, B, D.1, D.1, D.2, D.3, D.3

    2024 arXiv

  11. [19]

    Vlad Sobal, Wancong Zhang, Kynghyun Cho, Randall Balestriero, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. arXiv preprint arXiv:2502.14819, 2025. 1

    2025

  12. [20]

    DINO-Foresight: Looking into the future with dino

    Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. DINO-Foresight: Looking into the future with dino. arXiv preprint arXiv:2412.11673, 2024. 1, 2.1, 2.2, 3.1, 4, 4.1, 5

    2024

  13. [21]

    Intuitive physics understanding emerges from self-supervised pretraining on natural videos

    Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos. arXiv preprint arXiv:2502.11831, 2025. 1, 4.1, 5, B, E.2

    2025 arXiv

  14. [22]

    Do generative video models understand physical principles? arXiv preprint arXiv:2501.09038, 2025

    Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles? arXiv preprint arXiv:2501.09038, 2025. 1, 2.1

    2025 arXiv

  15. [23]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 1, 2.2, 3.1, 4, A.1

    2024

  16. [24]

    Revisiting feature prediction for learning visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. TMLR, 2024. 1, 2.2, 3.1, 4

    2024

  17. [25]

    Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2 (4), 1991. 2.1

    1991

  18. [26]

    Making the world differentiable: on using self supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments

    Jürgen Schmidhuber. Making the world differentiable: on using self supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. F orschungs- berichte, TU Munich, FKI 126 90, 1990

    1990

  19. [27]

    Goodwin and K.S

    G.C. Goodwin and K.S. Sin. Adaptive Filtering Prediction and Control. Information and Systems Sciences Series. Prentice-Hall, 1984. ISBN 9780130040695. 2.1

    1984

  20. [28]

    Lillicrap, Jimmy Ba, and Mohammad Norouzi

    Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2019. 2.1, 2.2, 5

    2019

  21. [29]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In ICLR, 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb. 2.1, 2.2

    2023

  22. [30]

    Lillicrap, Ian S

    Danijar Hafner, Timothy P. Lillicrap, Ian S. Fischer, Ruben Villegas, David R Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, 2019. 2.1, 2.2

    2019

  23. [31]

    Navigation world models, 2024

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, 2024. URL https://arxiv.org/abs/2412.03572. 2.1

    2024 arXiv

  24. [32]

    Video pixel networks

    Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In ICML, 2016. 2.1

    2016

  25. [33]

    Campbell, and Sergey Levine

    Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018

    2018

  26. [34]

    Stochastic video generation with a learned prior

    Remi Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, 2018. 11

    2018

  27. [35]

    VideoGPT: Video generation using vq-vae and transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021

    2021 arXiv

  28. [36]

    Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked generative video transformer. In CVPR, 2023. 2.1, 2.2

    2023

  29. [37]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In ECCV, 2023. 2.1

    2023

  30. [38]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023. URL https://openre...

    2023

  31. [39]

    D. Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David C. Minnen,...

    2023

  32. [40]

    Action-conditional video prediction using deep networks in atari games

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NeurIPS, 2015. 2.1

    2015

  33. [41]

    Recurrent environment simulators

    Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In ICLR, 2017. 2.1

    2017

  34. [42]

    How far is video generation from world model: A physical law perspective

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. arXiv preprint arXiv:2411.02385,

    arXiv

  35. [43]

    Predicting deeper into the future of semantic segmentation

    Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In ICCV, 2017. 2.1, 4.1, 5

    2017

  36. [44]

    Segmenting the future

    Hsu kuang Chiu, Ehsan Adeli, and Juan Carlos Niebles. Segmenting the future. In IROS, 2019. 2.1

    2019

  37. [45]

    Predicting future instance segmentation by forecasting convolutional features

    Pauline Luc, Camille Couprie, Yann LeCun, and Jakob Verbeek. Predicting future instance segmentation by forecasting convolutional features. In ECCV, 2018. 2.1

    2018

  38. [46]

    Anticipating visual representations from unlabeled video

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016. 2.1

    2016

  39. [47]

    Anticipative video transformer

    Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In ICCV, 2021

    2021

  40. [48]

    Anticipative feature fusion transformer for multi-modal action anticipation

    Zeyun Zhong, David Schneider, Michael V oit, Rainer Stiefelhagen, and Jürgen Beyerer. Anticipative feature fusion transformer for multi-modal action anticipation. In WACV, 2022. 2.1

    2022

  41. [49]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2022. 2.2

    2022

  42. [50]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 2.2

    2017

  43. [51]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025

  44. [52]

    Is sora a world simulator? a comprehensive survey on general world models and beyond

    Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, Yang You, Zhaoxiang Zhang, Dawei Zhao, Liang Xiao, Jian Zhao, Jiwen Lu, and Guan Huang. Is sora a world simulator? a comprehensive survey on general world model...

    2024

  45. [53]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3.2

    2017

  46. [54]

    Rethinking patch dependence for masked autoencoders

    Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, XuDong Wang, Adam Yala, Trevor Darrell, Alexei A Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. In NeurIPS 2024 Workshop: Self-Supervised Learning-Theory and Practice , 2024. 3.2 12

    2024

  47. [55]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 3.2, A.1

    2024

  48. [56]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. TMLR, 2023. 4

    2023

  49. [57]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 4, A.2

    2019

  50. [58]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 4, 4.1, B

    2016

  51. [59]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In IC...

    2017

  52. [60]

    HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630–2640, 2019. 4

    2019

  53. [61]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 4

    2017 arXiv

  54. [62]

    Vspw: A large-scale dataset for video scene parsing in the wild

    Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In CVPR, 2021. 4.1

    2021

  55. [63]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231–1237, 2013. 4.1

    2013

  56. [64]

    Intphys 2019: A benchmark for visual intuitive physics understanding

    Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys 2019: A benchmark for visual intuitive physics understanding. IEEE TPAMI, 44(9):5016–5025, 2021. 4.1

    2019

  57. [65]

    Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models

    Serwan Jassim, Mario Holubar, Annika Richter, Cornelius Wolff, Xenia Ohmer, and Elia Bruni. Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. In IJCAI, 2024. 4.1, E.2

    2024

  58. [66]

    Benchmarking progress to infant-level physical reasoning in AI

    Luca Weihs, Amanda Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in AI. TMLR, 2022. 4.1, E.2

    2022

  59. [67]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024

  60. [68]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2023. 4.3, B, D.1

    2023

  61. [69]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. 4.3, B, D.1 13 A Implementation and training A.1 Architecture Video encoder. For our main experiments, we ...

    2004 arXiv

  62. [70]

    frameskip

    to the tokens to inject the positional information in the attention operation. Specifically, since we need to encode positional information along three axis, we split the 60 dimensions of the head in three 20-dimensional chunks, to which we apply RoPE rotations derived from 10...

    2000

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.