Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A driving world model can predict long, coherent futures by separating large scene changes from small continuous motion, and distilling fine-scale consistency into the coarse-scale predictor.

desk verdict The 27% FVD claim over Vista is not supported: the test-time text prompt is never specified and likely encodes future frames, and the paper's own Table 1 shows it loses to Vista on the 25-frame task. read the letter →

arxiv 2506.01546 v1 pith:W5OF76CR submitted 2025-06-02 cs.CV

Xiaodong Wang , Zhirong Wu , Peixi Peng This is my paper
classification cs.CV
keywords drivingworldmodellong-termvideogenerationdiffusiontransformercross-granularitydistillationhierarchicalmotiondecouplingpredictiontrajectorycontrolNuScenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a driving world model can generate long, temporally coherent video by splitting prediction into two granularities instead of rolling out a short-clip model repeatedly. A coarse Diffusion Transformer predicts large scene changes between keyframes sampled at low frame rate, while a fine Diffusion Transformer handles smooth continuous motion at high frame rate and can interpolate between keyframes. A distillation step uses the fine model's one-step denoising outputs as self-supervised targets for the coarse model, aligning the two scales and reducing scene mutations. On NuScenes, the authors report a 27% improvement in FVD, a standard video-quality metric, over the state-of-the-art front-view model Vista, and an 85% reduction in inference time when generating 113 frames. If correct, this makes long-horizon prediction practical enough for driving simulation and counterfactual 'what if' reasoning.

What carries the argument

The load-bearing object is cross-granularity flow distillation between two Diffusion Transformers (DiTs, transformer-based denoisers for video latents). A 'flow' here is a one-step denoising output at a fixed noise level; the fine-grained flows produced by the frozen Fine DiT are treated as self-supervised teaching signals for the coarse-grained flows of the trainable Coarse DiT, matching them at coarse frame positions with an $L_2$ loss. This is supported by the hierarchical decoupling: the Coarse DiT encodes each keyframe independently (no temporal compression) so it can express large scene changes, while the Fine DiT uses temporally compressed tokens and learns both forward prediction and backward interpolation, so it is a reliable source of fine temporal priors. The warp-guided control module additionally injects depth-free warped future frames into the Fine DiT to make trajectory-conditioned generation track a given route.

What would settle it

Inspect the validation-set captions generated by the MLLM: if any caption names a critical object, road condition, or scene change that is not visible in the first frame, future information leaks into the prompt. A decisive check is to rerun the 113-frame FVD evaluation with prompts ablated to first-frame-only captions or a fixed generic caption; if the gap to Vista shrinks or disappears, the headline 27% gain is not purely a result of the proposed architecture.

Watch

Extended reading notes

Core claim

The central claim is that the training-inference gap that makes long driving videos blurry or inconsistent is caused by training on short high-fps clips, and it can be closed by decoupling world-model learning into large-motion learning and bidirectional continuous-motion learning. The Coarse DiT is trained on 13 keyframes at 1 fps to predict large dynamics from the first frame, with a frequency-domain structure-preservation loss to keep individual frames sharp. The Fine DiT is trained on 13 continuous frames at 10 fps for both causal prediction and interpolation. After both are trained, the frozen Fine DiT denoises 12 fine-grained segments in one step, and the Coarse DiT denoises the corresponding coarse keyframes with the same noise and timestep; an $L_2$ loss between the two flows at coarse positions transfers the fine model's temporal consistency into the coarse model. The paper claims this architecture achieves state-of-the-art FVD on NuScenes across short and long horizons, with a 27% FVD improvement and 85% faster inference than Vista at 113 frames.

Load-bearing premise

The reported gains assume that the text prompt given to the model at test time is derived from the first frame only; if the prompt is actually built from all eight keyframes, it may leak future scene content, and the FVD comparison against models without such text would no longer be fair.

Editorial extensions

If this is right

  • Long videos of 113 frames can be generated with substantially less error accumulation than autoregressive rollouts, because coarse keyframes are predicted once and fine frames are filled in between them.
  • Inference time is no longer proportional to video length: 145 frames cost roughly one coarse rollout plus parallel interpolation, about 70 seconds on a single A100.
  • Trajectory-conditioned generation becomes more accurate, since warped future frames give the model explicit geometric hints without needing depth maps.
  • The method builds on a standard DiT backbone (CogVideoX) and adds only coarse-frame sampling, fine training, and distillation, so it does not require a new generative architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to rerun the 113-frame FVD comparison with the MLLM prompt removed or fixed to a generic caption; if the 27% gain persists, the gain is architectural rather than textual.
  • The decoupling principle suggests that other long-horizon simulators, such as multi-view driving, 3D occupancy, or general video, could adopt a 'predict keyframes, distill from a fine interpolator' scheme without retraining a purpose-built model.
  • The inference-time advantage depends on the fine segments being interpolated in parallel; on hardware or implementations where parallelism is limited, the 85% figure should shrink and the comparison should be re-measured.
  • The structure-preservation loss and flow distillation together imply a trade-off between per-frame fidelity and temporal consistency; ablating distillation steps, as the appendix does, shows where that trade-off sits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes LongDWM, a hierarchical driving world model for long-term video prediction, with three main components: a Coarse Diffusion Transformer (CDiT) trained on low-fps keyframes for large-motion prediction, a Fine Diffusion Transformer (FDiT) trained on high-fps continuous frames for causal/bidirectional motion learning, and a cross-granularity distillation stage in which frozen fine-flow outputs are used as self-supervised targets for coarse-flow prediction. A warp-guided controllable prediction module is also introduced. On NuScenes, the paper claims a 27% FVD improvement over Vista for generating 110+ frames and an 85% inference-time reduction. The main quantitative evidence is in Tables 1 and 2, supplemented by ablations and human evaluations.

Significance. If the claimed results hold, the hierarchical decoupling of large and continuous motion, the cross-granularity distillation idea, and the parallel interpolation scheme would be useful contributions to long-term driving world models. The 85% inference-time reduction appears robust because it stems from an architectural property (parallel interpolation across coarse frames) rather than from the contested evaluation protocol. The quality claim, however, is not yet established: the reported FVD advantage over Vista on long videos is vulnerable to a text-promp leak, and the paper's own short-video results contradict its stated claim of state-of-the-art performance in all settings. The method is interesting and potentially sound, but the current evaluation does not support the headline quality claim.

major comments (3)
  1. [Sec. 3.1, Eq. (3); Appendix B; Appendix D] The evaluation of the central quality claim is compromised by an unspecified inference-time text prompt. The model conditions on a text embedding p (Eq. 3) produced by an MLLM, and Appendix B states that LLaVA-NeXT-Video-7B annotates 'each video sequence of 8 keyframes' for 'all samples in the training set and validation set.' Appendix D describes the long-video evaluation only as 'We input the first frame to all models' and never states that the text prompt is regenerated from that first frame at inference. If the validation prompts are the precomputed full-clip annotations, they encode future content (Weather, Road environment, Critical objects) that the baselines in Table 2 do not receive, making the FVD comparison circular. The paper must specify and implement a first-frame-only prompt protocol, or the 27% FVD improvement claim over Vista cannot be accepted.
  2. [Table 1; Sec. 4.1; Abstract] The paper claims in the abstract and contributions that it achieves 'state-of-the-art performances on the FVD metric in all settings,' and Sec. 4.1 states that compared with SOTA models using extra driving videos (including Vista) 'our model also outperforms them on video fidelity.' This is contradicted by Table 1, where Ours has FVD 102.9 and FID 12.3, while Vista has FVD 89.4 and FID 6.9 on the same 25-frame task. The claim should be corrected to restrict state-of-the-art status to the long-video regime, and the discrepancy should be acknowledged.
  3. [Sec. 3.3, Eq. (8)] The distillation loss is underspecified regarding the alignment of coarse and fine latents. Coarse latents are per-frame encodings (Sec. 3.1) while fine latents are temporally compressed by the 3D-VAE (Sec. 3.2); the paper does not define how 'corresponding positions' [c1,...,cK] are identified across these two different temporal resolutions, nor how 'the same timestep and noise' is shared when the latent shapes differ. Equation (8) is therefore not reproducible as written, and this is load-bearing for the proposed distillation method.
minor comments (7)
  1. [Abstract; Table 2] The abstract's 'improves FVD by 27%' should specify that this result is on a 750-sample subset of the NuScenes validation set at 113 frames, not on the full validation set.
  2. [Appendix D] The sentence 'We input the first frame to all models' is incomplete for LongDWM, which also requires a text prompt; the prompt used for each model in Table 2 should be explicitly stated.
  3. [Table 1] The 'Reconstruction' row is not a model and should be labeled as a lower-bound reference (e.g., 'CogVideoX VAE reconstruction') to avoid confusion.
  4. [Appendix D; Table 11] The long-video evaluation reports FVD at 25, 68, and 113 frames, but Table 11 says LongDWM generates 145 frames in two rollouts; the discrepancy and how 113 frames are selected should be clarified.
  5. [Sec. 3.1, Eq. (4)] The structure-preservation loss H is described only as a '2D high-pass filter of frequency domain in latent space'; the implementation (e.g., FFT-based, per-channel) should be specified for reproducibility.
  6. [Sec. 3.2] The symbol K is reused for the number of coarse frames in Sec. 3.1 and the number of continuous frames in Sec. 3.2, which is confusing; use distinct notation.
  7. [Abstract] The claim that distillation improves 'the coherence of infinite video generation' is not supported by quantitative experiments beyond 145 frames; the evaluation covers at most 113 frames, so the claim should be tempered.

Circularity Check

1 steps flagged · score 6.0 of 10

Long-video FVD gain rests on a text prompt that, by the only documented protocol, is generated from an 8-keyframe validation clip containing the future frames being scored.

  1. self definitional [App. B; Sec. 3.1 Eq. (3); App. D 'Long Video prediction']
    "We use LLaVA-NeXT-Video-7B [39] to annotate each video sequence of 8 keyframes. We obtain annotations for all samples in the training set and validation set of NuScenes. ... Given a detailed video prompt, we utilize T5 to encode the prompt to a text embedding p. ... L_diffusion = E_{x^c,p,ε,t} ||D^c_θ(x^c_t, t, p, x^c_1) − x^c||^2_2. ... Long Video prediction. We input the first frame to all models, and rollout multiple times to generate long videos."

    The Coarse DiT is conditioned on text embedding p (Eq. 3). Appendix B is the only annotation protocol described, and it uses an 8-keyframe video, not a single first frame, so p can encode weather, road environment, and critical objects visible in the upcoming validation frames. Appendix D specifies only that the first frame is input to all models; it never states that p is recomputed from that single frame. Consequently, the 113-frame FVD in Table 2—the basis of the headline 27% improvement—may be generated from a p that summarizes the target clip, while baselines such as Vista receive no such text. The reported gain is then partly a target-conditioned reconstruction, not an independent prediction, under the documented protocol.

full rationale

The paper's method—hierarchical coarse/fine DiT and fine-to-coarse flow distillation—is not circular internally: Eq. (8) is a standard teacher–student loss with a frozen Fine DiT teacher, and no parameter is fitted to the metric it later reports. The self-citations ([29], [41], [34]) are not load-bearing for the central claim. The one substantive circularity is in the evaluation chain: the validation text annotation (App. B) is derived from an 8-keyframe sequence that includes future frames, and the long-video evaluation (App. D) does not state that the prompt is regenerated from the first frame only. Since the model's loss and inference condition on this prompt (Eq. 3), the Table 2 FVD scores—the source of the headline 27% gain—can be a function of the target video they are scored against. The 85% inference-time reduction is unaffected, but the central quality claim is compromised under the stated protocol. A score of 6 reflects partial circularity of the benchmark, not of the core derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a training recipe with several hand-chosen hyperparameters and on assumptions about the distillation signal, the VAE encoding, and the evaluation protocol. The most problematic is the unstated assumption about the test-time text prompt, which could invalidate the headline comparison if false.

free parameters (4)
  • beta_s structure loss weight = not reported
    Controls the balance between diffusion loss and the high-pass structure preservation loss in Eq. 5; chosen by hand, no ablation over its value is reported.
  • number of coarse frames K = 13
    Sequence length for both CDiT and FDiT training; a design choice that determines the granularity of the two-stage generation.
  • fps for coarse and fine sampling = 1 and 10
    Coarse frames are sampled at 1 fps, fine frames at 10 fps; these rates define the 'large motion' vs 'continuous motion' split and are chosen by the authors.
  • distillation steps = 100 (default)
    Paper reports a trade-off in Table 10 and defaults to 100 steps; this is a hyperparameter selected by the authors, not fitted to external data.
assumptions (4)
  • ad hoc to paper One-step denoising flows from the frozen Fine DiT are valid self-supervised targets for the Coarse DiT at corresponding latent positions.
    The distillation loss in Eq. 8 assumes that fine-grained flows encode the temporal consistency that coarse flows lack, and that matching latents at coarse positions is meaningful; this is the core mechanism and is not independently validated.
  • ad hoc to paper Sharing the same timestep and noise across different granularities (coarse per-frame latents and fine temporally compressed latents) is well-defined.
    Eq. 8 applies the same timestep t and noise epsilon to both branches, but the latent shapes and temporal compression differ; the paper does not specify how coarse positions map into the compressed fine latent timeline.
  • domain assumption The 3D-VAE encodes each coarse frame independently without losing structure needed for large motion prediction.
    CDiT uses per-frame VAE encoding to avoid temporal compression, relying on the VAE to preserve frame-level detail; no analysis is given for this assumption.
  • ad hoc to paper The text prompt used at inference is generated from the first frame only and does not encode future content.
    Appendix B describes annotating each video sequence of 8 keyframes; the paper does not state that inference-time prompts are generated from the first frame only, so the evaluation may be contaminated by future-derived text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model." pith.science (2026). https://pith.science/paper/W5OF76CR

@misc{pith2026250601546,
  author       = {Pith},
  title        = {Pith review of: LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W5OF76CR}},
  note         = {Machine review of arXiv:2506.01546}
}
abstract

Driving world models are used to simulate futures by video generation based on the condition of the current state and actions. However, current models often suffer serious error accumulations when predicting the long-term future, which limits the practical application. Recent studies utilize the Diffusion Transformer (DiT) as the backbone of driving world models to improve learning flexibility. However, these models are always trained on short video clips (high fps and short duration), and multiple roll-out generations struggle to produce consistent and reasonable long videos due to the training-inference gap. To this end, we propose several solutions to build a simple yet effective long-term driving world model. First, we hierarchically decouple world model learning into large motion learning and bidirectional continuous motion learning. Then, considering the continuity of driving scenes, we propose a simple distillation method where fine-grained video flows are self-supervised signals for coarse-grained flows. The distillation is designed to improve the coherence of infinite video generation. The coarse-grained and fine-grained modules are coordinated to generate long-term and temporally coherent videos. In the public benchmark NuScenes, compared with the state-of-the-art front-view model, our model improves FVD by $27\%$ and reduces inference time by $85\%$ for the video task of generating 110+ frames. More videos (including 90s duration) are available at https://Wang-Xiaodong1899.github.io/longdwm/.

Figures

Figures reproduced from arXiv: 2506.01546 by the authors.

Figure 1
Figure 1. Long video generation comparison in the autonomous driving scenario. The results include Vista [12], CogVideoX [15], CogVideoXsft and ours, the input first frame (f=1) is from the validation set of NuScenes [16]. The result of Vista shows extremely blurred frames (e.g., yellow stripe), and the results of CogVideoX and variant show unrealistic motion, while our model generates a realistic long-term future. on 25-fram… view at source ↗
Figure 2
Figure 2. The overall distillation framework. First, we decouple the long-term world model learning into a large motion learning and a small continuous motion learning by designing a Coarse DiT and a Fine DiT to adapt to different granularities. Then, we propose a novel flow distillation method between different granularities, i.e., using fine flows’ better priors to distill coarse flows, which prompts the Coarse DiT to produ… view at source ↗
Figure 3
Figure 3. Warp-guided trajectory controllable video training. Given an input trajectory, we first transform it into camera poses and utilize a depth￾free image warping to obtain warped subsequent frames. These frames are fed into FDiT along with a simple prompt to predict the clean frames. Inspired by this, we propose a novel trajec￾tory control method that leverages 3D informa￾tion without the need for reconstruction or ad￾d… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Short video prediction comparison. Compared to our model’s prediction with Vista, our model can produce more detailed future frames and generate reasonable and realistic motion. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Long-term video prediction comparison. Compared with the state-of-the-art Vista [12] which faces severe error accumulation, our model can predict higher-quality scenarios for the long￾term future and generalize to diverse scenarios. Input Vista Ours [PITH_FULL_IMAGE:f…
Figure 6
Figure 6. Figure 6: Trajectory controllability comparison. For different input images and trajectory points, we present the generation results (the 12th and 24th frames) of Vista and our model. Our model not only generates more realistic scenes but also accurately predicts the future unde…
Figure 7
Figure 7. Figure 7: Ablation study. The first row shows CDiT predictions including lots of distortions, the second row adds the structure preservation term, and the last row shows the best. Quantitative Results Tab. 6 shows the effect of the three main modules used in our method, includin…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenLongTail: Generative Scaling of Long-Tail Driving Data

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pose-informed diffusion with Plücker rays, depth warps, and cross-view memory converts monocular long-tail videos into multi-view assets that improve closed-loop driving robustness nearly to ground-truth multi-view levels.

  2. WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A five-aspect, 24-metric benchmark, a 26K human-annotated dataset, and an AI evaluator show that today's driving world models cannot simultaneously look real, respect geometry, and behave safely.

  3. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

Reference graph

Works this paper leans on

59 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    arXiv 2018

  2. [2]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1), 2022

    work page 2022

  3. [3]

    Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    arXiv 1912

  4. [4]

    Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    arXiv 2010

  5. [5]

    Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    arXiv 2023

  6. [6]

    Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.arXiv preprint arXiv:1812.00568, 2018

    arXiv 2018

  7. [7]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. InConference on robot learning, pages 1–16. PMLR, 2017

    2017

  8. [8]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    arXiv 2018

Show all 59 references
  1. [9]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  2. [10]

    Drivedreamer: Towards real-world-driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving.arXiv preprint arXiv:2309.09777, 2023

    2023 arXiv

  3. [11]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.arXiv preprint arXiv:2311.17918, 2023

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.arXiv preprint arXiv:2311.17918, 2023

    2023 arXiv

  4. [12]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  5. [13]

    Sora technical report

    OpenAI. Sora technical report. 2024

    2024

  6. [14]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  7. [15]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  8. [16]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern re...

    2020

  9. [17]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  10. [18]

    Riflex: A free lunch for length extrapolation in video diffusion transformers.arXiv preprint arXiv:2502.15894, 2025

    Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. Riflex: A free lunch for length extrapolation in video diffusion transformers.arXiv preprint arXiv:2502.15894, 2025. 10

    2025 arXiv

  11. [19]

    Moviedreamer: Hierarchical generation for coherent long visual sequence

    Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407.16655, 2024

    2024

  12. [20]

    Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023

    Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation.arXiv preprint arXiv:2303.12346, 2023

    2023 arXiv

  13. [21]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  14. [22]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2023

  15. [23]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  16. [24]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  17. [25]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.arXiv preprint arXiv:2405.04233, 2024

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.arXiv preprint arXiv:2405.04233, 2024

    2024 arXiv

  18. [26]

    Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis.Advances in Neural Information Processing Systems, 35:15420–15432, 2022

    Jian Liang, Chenfei Wu, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis.Advances in Neural Information Processing Systems, 35:15420–15432, 2022

    2022

  19. [27]

    Streamingt2v: Consistent, dynamic, and extendable long video generation from text.arXiv preprint arXiv:2403.14773, 2024

    Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text.arXiv preprint arXiv:2403.14773, 2024

    2024 arXiv

  20. [28]

    Adriver-i: A general world model for autonomous driving.arXiv preprint arXiv:2311.13549, 2023

    Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving.arXiv preprint arXiv:2311.13549, 2023

    2023 arXiv

  21. [29]

    Prophetdwm: A driving world model for rolling out future actions and videos.arXiv preprint arXiv:2505.18650, 2025

    Xiaodong Wang and Peixi Peng. Prophetdwm: A driving world model for rolling out future actions and videos.arXiv preprint arXiv:2505.18650, 2025

    2025 arXiv

  22. [30]

    Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    2023 arXiv

  23. [31]

    Occworld: Learning a 3d occupancy world model for autonomous driving

    Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. InEuropean Conference on Computer Vision, pages 55–72. Springer, 2025

    2025

  24. [32]

    Temporal triplane transformers as occupancy world models.arXiv preprint arXiv:2503.07338, 2025

    Haoran Xu, Peixi Peng, Guang Tan, Yiqian Chang, Yisen Zhao, and Yonghong Tian. Temporal triplane transformers as occupancy world models.arXiv preprint arXiv:2503.07338, 2025

    2025

  25. [33]

    Doe-1: Closed-loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627, 2024

    Wenzhao Zheng, Zetian Xia, Yuanhui Huang, Sicheng Zuo, Jie Zhou, and Jiwen Lu. Doe-1: Closed-loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627, 2024

    2024 arXiv

  26. [34]

    Holodrive: Holistic 2d-3d multi-modal street scene generation for autonomous driving.arXiv preprint arXiv:2412.01407, 2024

    Zehuan Wu, Jingcheng Ni, Xiaodong Wang, Yuxin Guo, Rui Chen, Lewei Lu, Jifeng Dai, and Yuwen Xiong. Holodrive: Holistic 2d-3d multi-modal street scene generation for autonomous driving.arXiv preprint arXiv:2412.01407, 2024. 11

    2024 arXiv

  27. [35]

    Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024

    2024 arXiv

  28. [36]

    Dive: Dit-based video generation with enhanced control

    Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, et al. Dive: Dit-based video generation with enhanced control. arXiv preprint arXiv:2409.01595, 2024

    2024 arXiv

  29. [37]

    Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation.arXiv preprint arXiv:2503.15208, 2025

    Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, et al. Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation.arXiv preprint arXiv:2503.15208, 2025

    2025 arXiv

  30. [38]

    Uniscene: Unified occupancy-centric driving scene generation.arXiv preprint arXiv:2412.05435, 2024

    Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation.arXiv preprint arXiv:2412.05435, 2024

    2024 arXiv

  31. [39]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  32. [40]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  33. [41]

    Learning 3d photography videos via self-supervised diffusion on single images

    Xiaodong Wang, Chenfei Wu, Shengming Yin, Minheng Ni, Jianfeng Wang, Linjie Li, Zhengyuan Yang, Fan Yang, Lijuan Wang, Zicheng Liu, et al. Learning 3d photography videos via self-supervised diffusion on single images. InProceedings of the Thirty-Second International Joint Conf...

    2023

  34. [42]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6613–6623, 2024

    2024

  35. [43]

    Improved distribution matching distillation for fast image synthesis

    Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867, 2024

    2024 arXiv

  36. [44]

    From slow bidirectional to fast causal video generators.arXiv preprint arXiv:2412.07772, 2024

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators.arXiv preprint arXiv:2412.07772, 2024

    2024

  37. [45]

    Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    2024 arXiv

  38. [46]

    Vd3d: Taming large video diffusion transformers for 3d camera control.arXiv preprint arXiv:2407.12781, 2024

    Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control.arXiv preprint arXiv:2407.12781, 2024

    2024 arXiv

  39. [47]

    Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation.arXiv preprint arXiv:2406.10126, 2024

    2024 arXiv

  40. [48]

    Gs-dit: Advancing video generation with pseudo 4d gaussian fields through efficient dense 3d point tracking.arXiv preprint arXiv:2501.02690, 2025

    Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yijin Li, Fu-Yun Wang, and Hongsheng Li. Gs-dit: Advancing video generation with pseudo 4d gaussian fields through efficient dense 3d point tracking.arXiv preprint arXiv:2501.02690, 2025

    2025 arXiv

  41. [49]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5820–5829, 2021

    2021

  42. [50]

    Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation

    Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. InEuropean Conference on Computer Vision, pages 329–345. Springer, 2025. 12

    2025

  43. [51]

    Generalized predictive model for autonomous driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Generalized predictive model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14662–14672, 2024

    2024

  44. [52]

    Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving.arXiv preprint arXiv:2412.04842, 2024

    Rui Chen, Zehuan Wu, Yichen Liu, Yuxin Guo, Jingcheng Ni, Haifeng Xia, and Siyu Xia. Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving.arXiv preprint arXiv:2412.04842, 2024

    2024 arXiv

  45. [53]

    Dynamicrafter: Animating open-domain images with video diffusion priors

    Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. InEuropean Conference on Computer Vision, pages 399–417. Springer, 2025

    2025

  46. [54]

    I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.arXiv preprint arXiv:2311.04145, 2023

    Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.arXiv preprint arXiv:2311.04145, 2023

    2023 arXiv

  47. [55]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  48. [56]

    Seine: Short-to-long video diffusion model for generative transition and prediction

    Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  49. [57]

    Framer: Interactive frame interpolation

    Wen Wang, Qiuyu Wang, Kecheng Zheng, Hao Ouyang, Zhekai Chen, Biao Gong, Hao Chen, Yujun Shen, and Chunhua Shen. Framer: Interactive frame interpolation. InThe Thirteenth International Conference on Learning Representations

  50. [58]

    Navier-stokes, fluid dynamics, and image and video inpainting

    Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. Navier-stokes, fluid dynamics, and image and video inpainting. InProceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–I. IEEE, 2001. 13 A De...

    2001

  51. [59]

    Distillation.After obtaining the well-trained Coarse DiT and Fine DiT, we establish the distillation training

    The Fine DiT is trained on all continuous frames of the training set of NuScenes, with a total batch size of 8, and trained with 10k steps on 4×A100 GPUs. Distillation.After obtaining the well-trained Coarse DiT and Fine DiT, we establish the distillation training. We sample 1...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.