Pith. sign in

REVIEW 4 major objections 6 minor 56 references

TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single autoregressive world model, trained on a graph of street panoramas with coordinates and move commands, can localize images to within 10 meters, generate the next view under instruction, and choose its own legal driving actions.

desk verdict The dataset is a real contribution; the model claims are not supported by the reported evaluation—georeferencing leaks the answer through the move command, and the FID/SSIM numbers are missing from the experiments. read the letter →

arxiv 2506.11302 v3 pith:3SUGRIRT submitted 2025-06-12 cs.CV cs.AI

Héctor Carrión , Yutong Bai , Víctor A. Hernández Castro , Kishan Panaganti , Ayush Zenith , Matthew Trang , Tony Zhang , Pietro Perona
show 1 more author
Jitendra Malik
This is my paper · ORCID
classification cs.CVcs.AI
keywords STRIDETARDISworldmodelgeoreferencingspatiotemporaldatasetautoregressivetransformercontrollableimagegenerationself-control
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that real-world driving environments can be modeled as a unified autoregressive sequence of observations, states, and actions across both space and time, and that a transformer trained on such sequences gains a range of embodied abilities. To show this, it builds STRIDE, a dataset of 82B tokens derived from 131k Google StreetView panoramas in San Mateo, permuted into 6.3M visual sentences by stitching panoramic views into graph-structured drivable routes with coordinates, months, years, and movement commands. It trains TARDIS, a 1B-parameter generative world model, on this dataset. TARDIS reportedly georeferences held-out images to within 10m error 60% of the time, generates photorealistic next views under spatial and temporal instruction, and autonomously generates movement commands that stay on a road 77.4% of the time at a 4m lane width. If true, this is evidence that a single sequence-prediction framework can fuse perception, localization, and control in dynamic real-world environments.

What carries the argument

The load-bearing object is the spatiotemporal token sequence V = {s_start, N_1, ..., N_n, s_end}, where each node N_i packages an egocentric observation O (1024 VQGAN tokens of a 360° panorama projected toward travel direction), spatial state S (latitude, longitude), temporal state S (month, year), and action A (displacement in meters, heading in degrees, Δmonth, Δyear). This representation turns the physical road network into a graph the model can traverse, and the paper formalizes the traversal as five coupled functions (fss, fts, fsa, fta, fO) that map observation to state, state to action, and state-plus-action to the next observation. The Markovian structure, validated by OpenStreetMap-grounded transitions, is what lets a single transformer trained as a next-token predictor perform georeferencing, generation, and self-control without task-specific heads.

What would settle it

Run TARDIS on the georeferencing benchmark with image tokens replaced by a constant placeholder image while keeping the true state and action tokens; if coordinate error stays near its current 60%-within-10m level, the images are not doing the localizing work and the result is odometry-plus-memory. A complementary check is a no-image baseline that receives only the previous coordinates and the move command and predicts the next coordinate.

Watch

Extended reading notes

Core claim

The central claim is that spatial and temporal navigation of the real world reduce to one auto-regressive prediction problem. On STRIDE, each step of a journey is a token block N_i = {image, latitude, longitude, month, year, displacement, heading, Δmonth, Δyear}, and the model is trained to predict the next block from the previous ones. The paper argues that because node transitions are grounded in the OpenStreetMap road network, the process is Markovian: the next observation depends only on the current observation, state, and action. From this single formulation TARDIS derives four capabilities: coordinate prediction (georeferencing), conditional photorealistic image generation, self-generated navigation actions, and explicit temporal reasoning, each demonstrated on data held out in space or time. The result, as the authors state it, is a promising direction toward generalist agents that understand and manipulate the spatial and temporal aspects of their material environments.

Load-bearing premise

The georeferencing result stands or falls on whether the true move command (distance and heading) given to the model does not effectively reveal the next location, since the evaluation measures coordinate error by masking the coordinate tokens but leaving the action tokens intact.

Editorial extensions

If this is right

  • A single model trained on observation-state-action sequences can output meter-level geographic coordinates from a street image alone, outperforming a dedicated aerial-matching geolocalization method given only a 50m search space.
  • The same model can be instructed to move and watch the world change accordingly, including temporal commands that alter seasons or infrastructure, which is a step toward controllable simulation of real environments.
  • The model's self-generated actions keep it on legal road segments 77.4% of the time at 4m lane width on held-out areas, suggesting that action generation emerges from the world-model objective rather than a hand-designed policy.
  • Because time is an explicit token dimension, the model can estimate position and generate views across years, with image dissimilarity scaling linearly (R²=0.94) over 5-year intervals.
  • The dataset construction method itself—permuting a small set of panoramas into 6.3M visual sentences with 27x augmentation—provides a recipe for building large spatiotemporal world-model datasets from modest image collections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the georeferencing is genuinely image-driven, the imagery itself encodes absolute position at meter level in an area the model never saw, which implies that a world model trained on dense street-view permutations could serve as a learned visual compass usable when GPS is unavailable.
  • The same observation-state-action tokenization could transfer to other embodied settings—indoor navigation, drone flight, or robot manipulation—wherever a graph of traversable nodes with sensor observations and motion commands can be assembled.
  • A natural stress test the paper does not run is to train on one city and georeference in a second, unseen city; the fixed 16K context and single-region training leave open whether the learned notion of space generalizes or is memorized road-network topology.
  • The temporal control dimension is effectively a learned simulator that can hallucinate the same place at different dates; such a simulator could be used as data augmentation for downstream perception models that must be robust to seasonal and infrastructural change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces STRIDE, a large-scale dataset that reorganizes Google Street View panoramas and OpenStreetMap data into 6.3M 'visual sentences' of observation-state-action tokens spanning 82B tokens, and TARDIS, a 1B-parameter transformer-based autoregressive world model trained on this dataset. The authors claim state-of-the-art georeferencing (60% of predictions within 10m error), controllable photorealistic image generation (with a 41% FID improvement over Chameleon7B), autonomous self-control (77.4% road adherence at 4m lane width), and temporal reasoning (linear SSIM decay with R²=0.94). The dataset and code are released.

Significance. If the claims were substantiated, STRIDE would be a valuable new resource for spatiotemporal world modeling, and TARDIS would demonstrate a promising unified architecture for navigation, generation, and temporal reasoning. The paper is commendable for releasing training code, model checkpoints, and a sizable dataset, and the graph-based composition of street-view imagery into navigable state-action trajectories is a creative and potentially reusable idea. However, as we argue below, the experimental protocol used to support the headline claims is flawed in several load-bearing ways, and some quantitative claims appear to be unsupported by the reported measurements. The significance of the contribution therefore rests on corrections that have not yet been demonstrated.

major comments (4)
  1. [Section 4.2] The georeferencing evaluation leaks the answer. In Eq. (2), the action tokens include displacement and heading derived from the ground-truth path. Section 4.2 states that TARDIS is prompted with these spatial commands and that 'self-masking of tokens which fall outside the expected radius defined by the move command' is used. Given the current coordinates in the prompt and the true move command, the next coordinate is determined up to tokenization error; the visual observation is not needed. The 60% within 10m result is therefore an expected consequence of dead reckoning plus in-city memorization, not image-based georeferencing. The comparison to SVG is unfair because SVG is restricted to a 50m search radius, does not receive move commands, and was not trained on the same road network. Please provide an evaluation that masks or abandons the true move command, and that fairly matches the baseline.
  2. [Section 1 and Section 4.1] The claims of '41% FID improvement over Chameleon7B' and 'linear SSIM decay R2=0.94 over 5-year intervals' are not supported by any reported measurement. The cited figures (Fig. 3, Fig. 9, Figs. 20-22) show qualitative images and perplexity plots, not FID or SSIM values. Either report the actual quantitative measurements with evaluation details, or remove these claims from the abstract and contributions list.
  3. [Section 2.3] The spatiotemporal holdout is not a true out-of-distribution test. The 'bottom 10% of the training area' is embedded within a 9.2x7.5 km city where the remaining ~130k training nodes lie on the same road network, often immediately adjacent to the test nodes. Dense training coverage on the same streets enables memorization and interpolation, so the claimed generalization to 'unseen areas' is overstated. The temporal holdout (2023-2024) is likewise within the same city. To support the generalization claim, evaluate on a geographically disjoint area or a different city with a separate training set.
  4. [Section 4.3] The self-control evaluation lacks any baseline or random policy comparison. The metric (road adherence) is computed for actions generated by TARDIS, but without comparing to, e.g., a random heading/distance policy, a nearest-neighbor policy, or an action-only model that ignores the image, it is not possible to determine whether the image observation contributes to the result. Since TARDIS is trained on road-legal transitions, the 77.4% road-adherence rate may reflect memorized road layout rather than visual understanding. Add at least one non-trivial, non-oracle baseline.
minor comments (6)
  1. [Section 2.2 and Table 1] The number of panoramic images is reported as 135k in Section 2.2 but Table 1 lists 130k (total) or 105k (training); please reconcile these inconsistencies.
  2. [Section 4.1] The use of perplexity as a quantitative proxy for image quality should be justified; the paper itself notes that visual metrics are often not representative, and perplexity of a language model over image tokens is a weak proxy for photorealism or control.
  3. [Section 4.2] The phrase 'we dynamically allow self-masking of tokens which fall outside the expected radius defined by the move command' needs a precise definition of 'expected radius' and of how the masking is applied; as written, it strengthens the impression of answer leakage.
  4. [Appendix D] The tokenization table lists 'Year' with possible range 2000-2030, but the temporal distribution in Figure 19 only goes to 2024; clarify whether the model is required to predict beyond 2030 and how the token range was chosen.
  5. [Appendix E] The phrase '734 days of wall time, or about 16 hours on our 128 VM cluster' is confusing; clarify the relationship between total CPU-time, wall-time, and cluster size.
  6. [Appendix J.2] The dataset is released under CC-BY-NC-4.0, which may restrict commercial and downstream use; this should be stated explicitly in the main text, as it could affect the dataset's utility for the autonomous-driving community.

Circularity Check

1 steps flagged · score 7.0 of 10

Georeferencing evaluation reduces to dead reckoning: the true move command is supplied and coordinate tokens are masked to the expected radius, so the reported 60%-within-10m result is forced by the input rather than by image-based localization.

  1. self definitional [Section 2.1 Eq. (2); Section 4.2 Georeferencing]
    "Ni ={Oimage, Slatitude, Slongitude, Smonth, Syear, Adisplacement, Arotation, A∆month, A∆year}. ... Since TARDIS is able to input spatial commands which relate to the following spatial location, we dynamically allow self-masking of tokens which fall outside the expected radius defined by the move command."

    The STRIDE sample already contains current lat/lon and the next action (displacement plus heading). Given S_n and A_n, the next coordinate is determined by dead reckoning; the image observation is unnecessary. In the georeferencing evaluation, TARDIS is prompted with the true move command, and coordinate tokens are then masked to the 'expected radius defined by the move command'—i.e., to the neighborhood of the dead-reckoned location. The reported 60% of predictions within 10m and 90% within 20m therefore measure whether the model can apply supplied odometry and choose among road-constrained candidates, not whether it can geolocalize from an image. The SVG baseline receives no move command and is only restricted to a 50m search around the true location, so the comparison is unmatched.

full rationale

The dataset construction and the image-generation, self-control, and temporal experiments are not circular in the same way: they compare generated images or actions against held-out ground truth without feeding the answer into the prompt in a manner that determines the output. The self-citation to prior work for hyperparameter design is not load-bearing. However, the headline georeferencing result is a load-bearing circular evaluation: TARDIS is given the true spatial move command, and the coordinate search is masked to the radius defined by that command, so the meter-level precision is largely an odometry-application check rather than an image-based localization benchmark. The spatiotemporal holdout (bottom 10% of a 9.2x7.5 km city with dense training nodes on the same road network) is also not a strong out-of-distribution test, but that is an external-validity concern rather than a circularity. Overall, one of the four main contributions reduces by construction, so the circularity score is substantial but not total.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on dataset construction assumptions (accurate GSV metadata, OSM road validity) and on evaluation choices (holdout layout, command-based masking). No free parameters are fit in the mathematical sense; the listed items are hand-chosen ranges, thresholds, and protocol choices that directly affect the reported metrics.

free parameters (4)
  • Tokenization ranges and precisions for state/action modalities = lat 1e-5 over 37.50555-37.57277, lon 1e-5 over -122.34916 to -122.249168, distance 0.1m over 0-50m, heading 0.1deg…
    Chosen by hand in Table 2; these define the coordinate/displacement resolution and the maximum move action, shaping georeferencing error and action statistics.
  • Maximum move action and associated search radius = 50 meters
    Used to restrict SVG baseline to 50m search space and to self-mask TARDIS coordinate tokens during georeferencing; this bounds the evaluation and favors the proposed model.
  • DFS path generation limits = Not reported
    Configurable parameters in Section 2.2 that control sequence composition; their values are not stated, so the effective data graph and potential memorization surface are unspecified.
  • Road-adherence lane width threshold = 4 meters, with 1-10m sweep
    Metric for self-control quality; the headline 77.4% is at 4m threshold chosen post hoc, with no random or baseline policy compared.
assumptions (5)
  • domain assumption Every GSV panorama and its metadata (lat/lon, month/year) are accurate and align with the OSM road network.
    Relied on throughout Section 2; if metadata are noisy, the ground truth for georeferencing and temporal evaluation is corrupted.
  • domain assumption Transitions along DFS paths over the OSM-derived graph cover all permissible steps and satisfy P(S_{t+1}|S_t,A_t)>0 iff a valid road path exists.
    Stated in Section 3; this Markovian grounding is foundational to the dataset and world model, but OSM tags and GSV node spacing may not capture driveability or traffic constraints.
  • domain assumption Projecting a panorama into four 90-degree views and tokenizing to 512x512 VQGAN codes preserves enough visual information for navigation and geolocation.
    Underlies the 27x expansion and all image-based tasks; panoramic stitching artifacts are acknowledged in Appendix A.1 but not quantified for downstream accuracy.
  • domain assumption Standard transformer and LLaMA scaling assumptions apply, and a 1B-parameter model with 16K context is sufficient to learn the spatiotemporal mapping.
    Architecture choices in Section D follow [30] and [56]; the paper admits training was likely insufficient for convergence in Appendix E.
  • domain assumption Spatial (bottom 10%) and temporal (2023-2024) holdouts provide a genuine out-of-distribution test of georeferencing and generation.
    Assumed in Section 2.3 and used for all test claims; however, the held-out area is adjacent to dense training nodes in the same city, so visual and road-network context is in-distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy." pith.science (2026). https://pith.science/paper/3SUGRIRT

@misc{pith2026250611302,
  author       = {Pith},
  title        = {Pith review of: TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3SUGRIRT}},
  note         = {Machine review of arXiv:2506.11302}
}
read the original abstract

World models aim to simulate environments and enable effective agent behavior. However, modeling real-world environments presents unique challenges as they dynamically change across both space and, crucially, time. To capture these composed dynamics, we introduce a Spatio-Temporal Road Image Dataset for Exploration (STRIDE) permuting 360-degree panoramic imagery into rich interconnected observation, state and action nodes. Leveraging this structure, we can simultaneously model the relationship between egocentric views, positional coordinates, and movement commands across both space and time. We benchmark this dataset via TARDIS, a transformer-based generative world model that integrates spatial and temporal dynamics through a unified autoregressive framework trained on STRIDE. We demonstrate robust performance across a range of agentic tasks such as controllable photorealistic image synthesis, instruction following, autonomous self-control, and state-of-the-art georeferencing. These results suggest a promising direction towards sophisticated generalist agents--capable of understanding and manipulating the spatial and temporal aspects of their material environments--with enhanced embodied reasoning capabilities. Training code, datasets, and model checkpoints are made available at https://huggingface.co/datasets/Tera-AI/STRIDE.

Figures

Figures reproduced from arXiv: 2506.11302 by the authors.

Figure 1
Figure 1. Data structure (STRIDE) and modeling process (TARDIS) overview. TARDIS inputs Observation O0 which conditions State S0, both in space (coordinates) and time (month, year). Following this, O0 and S0 condition action A0 spatially as a move distance in meters with heading in degrees and temporally via month, year offsets. Finally, fO : (O0, S0, A0) → O1, and the auto￾regressive cycle repeats. At any point within this c… view at source ↗
Figure 2
Figure 2. STRIDE Geographical Distribution. We define a 9.2km x 7.5 km area covering San Mateo City, the resulting 135k data distribution is visualized onto the map spatiotemporally. The bottom 10 percent of the map is kept out of the training set (spatiotemporal hold-out) and all observations in 2023 or 2024 are also reserved for testing (temporal hold-out). 2.3 Training & Testing Dataset Composition We held-out testing data… view at source ↗
Figure 3
Figure 3. TARDIS vs. Chameleon Image Generation. TARDIS appears to more closely follow the image pattern and instructions. In addition, Chameleon smaller context length (4k vs 16k) impacts performance after 5 frames. 4.1 Controllable Image Generation Generating photorealistic images in a physically controllable way is an under-explored task. Existing generative models either output a single image given a text description or a… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: (Left) TARDIS Georeferencing. Blue signifies ground-truth locations, Red signifies TARDIS coordinate predictions. (Right) CDF of TARDIS vs SVG with 50 Meter Search Space. Describes the fraction of predictions within some distance threshold. In particular, TARDIS model …
Figure 6
Figure 6. Figure 6: (Left) Qualitative Analysis of Self-Control. Arrowheads represent heading action, line length represents distance action generated by TARDIS. We observe the majority of actions keep within the lane or onto driveways. (Right) Temporal Reasoning. Blue represents a prompt…
Figure 5
Figure 5. Figure 5: Self-Control Quality for Different Lane Widths. . For qualitative analysis we plot heading (ar￾rowhead) and distance (line length) on a satel￾lite map of the spatiotemporally held-out area, shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: General overview of the architecture for the distributed data aggregator. We deployed a simple Kubernetes application where we independently scaled each worker to accommodate the desired throughput for Google StreetView images through the use of message brokers and que…
Figure 8
Figure 8. Figure 8: we prompt TARDIS with five real image samples, then allow the model to extrapolate freely the next five observations. Figures 10,11,12,13 and 14 contain results where we generate the n-th image by prompting with real data up to step n − 1, similar to what the model mig…
Figure 9
Figure 9. Figure 9: Qualitative TARDIS Samples. As mentioned on the data section, STRIDE lacks quality data around overpasses, which leads to degraded performance on row 3. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Additional Qualitative TARDIS Samples #1 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Additional Qualitative TARDIS Samples #2 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Additional Qualitative TARDIS Samples #3 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Additional Qualitative TARDIS Samples #4 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Additional Qualitative TARDIS Samples #5 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 16
Figure 16. Figure 16 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: CDF TARDIS vs SVG: This describes the fraction of predictions within some distance threshold, we find TARDIS error to be within 20m 90% of the time, while SVG is within this error at about a 10% rate. H Additional Self-Control Results We observe the displacement actio…
Figure 18
Figure 18. Figure 18: TARDIS Self-Control Action Magnitude CDF. Self-generated actions actions move at a magnitude of 5m or above 60% of the time. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: ). We do not observe a significant error trend increase when generating images into our fully held-out years. This is visualized on [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Perplexity vs Year-Month Dates. Across a range of dates average perplexity remains mostly consistent. 15 10 5 0 5 10 GT Temporal Action 100 200 300 400 500 600 700 800 900 Perplexity Perplexity over GT Temporal Action Year and Month GT Temporal Action Year GT Temporal…
Figure 21
Figure 21. Figure 21: Perplexity over GT Temporal Action Year and Month. Non-extreme temporal instruc￾tions show a consistent perplexity score. J Code and Data Release Documentation J.1 Code Usage Instructions The code is thoroughly documented and released at https://github.com/tera-ai/tar…
Figure 22
Figure 22. Figure 22: Perplexity vs. Testing Action Move. The majority of temporal instructions show a consistent perplexity score. J.2 Dataset Format and Access The dataset can be downloaded and inspected (using the Dataset Viewer feature on HuggingFace) by following the instructions at h…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 30 canonical work pages

  1. [1]

    Statewide visual geolocalization in the wild

    Florian Fervers, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelhagen. Statewide visual geolocalization in the wild. InECCV, 2024. 9

    work page 2024

  2. [2]

    Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization.Advances in Neural Information Processing Systems, 36, 2024

    Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah. Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization.Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    Pigeon: Predicting image geolocations

    Lukas Haas, Michal Skreta, Silas Alberti, and Chelsea Finn. Pigeon: Predicting image geolocations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12893– 12902, 2024

    work page 2024

  4. [4]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024

    2024

  5. [5]

    Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation.arXiv preprint arXiv:2312.14125, 2023

    arXiv 2023

  6. [6]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    arXiv 2022

  7. [7]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563– 22575, 2023

    work page 2023

  8. [8]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023

    work page 2023

Show all 56 references
  1. [9]

    Navigation world models.arXiv preprint arXiv:2412.03572, 2024

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models.arXiv preprint arXiv:2412.03572, 2024

    2024 arXiv

  2. [10]

    Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  3. [11]

    Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    2024 arXiv

  4. [12]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations, 2020

    2020

  5. [13]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  6. [14]

    Planet dump retrieved from https://planet.osm.org

    OpenStreetMap contributors. Planet dump retrieved from https://planet.osm.org. https://www. openstreetmap.org, 2017

    2017

  7. [15]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  8. [16]

    Video pixel networks

    Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings...

    2017

  9. [17]

    Efficient video generation on complex datasets.CoRR, abs/1907.06571, 2019

    Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets.CoRR, abs/1907.06571, 2019. URLhttp://arxiv.org/abs/1907.06571

    1907 arXiv

  10. [18]

    Unsupervised learning for physical interaction through video prediction

    Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. InProceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 64–72, Red Hook, NY , USA, 2016. Curran Associat...

    2016

  11. [19]

    Transformation-based adversarial video prediction on large-scale data.CoRR, abs/2003.04035, 2020

    Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-based adversarial video prediction on large-scale data.CoRR, abs/2003.04035, 2020

    2003 arXiv

  12. [20]

    Deep predictive coding networks for video prediction and unsupervised learning

    William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. InInternational Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1ewdt9xe. 10

    2017

  13. [21]

    Videogpt: Video generation using vq-vae and transformers, 2021

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021

    2021

  14. [22]

    Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis

    A. Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575, 2023

    2023

  15. [23]

    Predicting video with VQV AE, 2021

    Jacob C Walker, Ali Razavi, and Aaron van den Oord. Predicting video with VQV AE, 2021

    2021

  16. [24]

    Ccvs: Context-aware controllable video synthesis

    Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Ccvs: Context-aware controllable video synthesis. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 14042–14055. Curra...

    2021

  17. [25]

    Diffusion models for video prediction and infilling.Transactions on Machine Learning Research, 2022

    Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling.Transactions on Machine Learning Research, 2022. ISSN 2835-8856

    2022

  18. [26]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. InThe Eleventh International Conference on Lear...

    2023

  19. [27]

    Kingma, Ben Poole, Mohammad Norouzi, David J

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022

    2022

  20. [28]

    Video diffusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 8633–8646....

    2022

  21. [29]

    L. Yu, Y . Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M. Yang, Y . Hao, I. Essa, and L. Jiang. Magvit: Masked generative video transformer. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10459–10469, Los Alamitos, CA, U...

    2023

  22. [30]

    Sequential modeling enables scalable learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  23. [31]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5820–5829, 2021

    2021

  24. [32]

    Scalability in per- ception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang,...

    2020

  25. [33]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InCVPR, 2020

    2020

  26. [34]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and fo...

    2021

  27. [35]

    Recurrent world models facilitate policy evolution

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. InProceedings of the 32Nd International Conference on Neural Information Processing Systems, NeurIPS’18, pages 2455–2467, 2018. 11

    2018

  28. [36]

    Action-conditional video prediction using deep networks in atari games

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. InProceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, page 2863–2871, Ca...

  29. [37]

    Action-conditioned benchmarking of robotic video prediction models: a comparative study

    Manuel Serra Nunes, Atabak Dehban, Plinio Moreno, and José Santos-Victor. Action-conditioned benchmarking of robotic video prediction models: a comparative study. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8316–8322, 2020. doi: 10.1109/ICRA40...

    2020

  30. [38]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. InInternational Conference on Learning Representations, 2021

    2021

  31. [39]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  32. [40]

    Transformer-based world models are happy with 100k interactions

    Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  33. [41]

    Learning to simulate dynamic environments with gamegan

    Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  34. [42]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5820–5829, June 2021

    2021

  35. [43]

    Chris Bamford and Simon M. Lucas. Neural game engine: Accurate learning ofgeneralizable forward models from pixels. InConference on Games, 2020

    2020

  36. [44]

    Recurrent environment simulators

    Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. InInternational Conference on Learning Representations, 2017

    2017

  37. [45]

    Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models

    Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volu...

    2022

  38. [46]

    S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chlo...

    2018 doi

  39. [47]

    Drivedreamer: Towards real-world-driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    2023 arXiv

  40. [48]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024

    2024 arXiv

  41. [49]

    Pixel recurrent neural networks

    Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747–1756. PMLR, 2016

    2016

  42. [50]

    Video pixel networks

    Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. InInternational Conference on Machine Learning, pages 1771–1779. PMLR, 2017

    2017

  43. [51]

    Maskvit: Masked visual pre-training for video prediction

    Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=QAV2CcLEDh

    2023

  44. [52]

    Autoregressive video generation without vector quantization

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openre...

    2025

  45. [53]

    Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025

    Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction.arXiv preprint arXiv:2503.19325, 2025

    2025 arXiv

  46. [54]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

    2020

  47. [55]

    Kesten, M

    R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V . Shet. Woven planet perception dataset 2020.https://woven.toyota/en/perception-data...

    2020

  48. [56]

    look around

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 13 A Limitations W...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.