Pith. sign in

REVIEW 3 major objections 6 minor 65 references

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-training future video and ego trajectories inside one shared Diffusion Transformer transfers world dynamics to planning more effectively than separate video and action models.

desk verdict Solid systems paper: shared DiT + future-token mask is a real, useful design for WAMs; the shared-vs-Two-DiT gap is directionally right but not capacity-matched. read the letter →

arxiv 2607.05133 v1 pith:D4N4OO53 submitted 2026-07-06 cs.CV

Mengmeng Liu , Diankun Zhang , Jiuming Liu , Jianfeng Cui , Hongwei Xie , Guang Chen , Hangjun Ye , Francesco Nex
show 2 more authors
Hao Cheng Michael Ying Yang
This is my paper
classification cs.CV
keywords unifiedvideo-actionmodeldiffusiontransformerautonomousdrivingworldactionmodelsmodalitydecouplingtrajectoryplanningzero-shotgeneralizationflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

World Action Models try to make driving planners generalize better by using future video prediction as dense supervision of scene dynamics and temporal causality. This paper claims that those video-learned priors transfer much more strongly into trajectory generation when future video latents and ego-trajectory tokens are denoised by the same shared Diffusion Transformer parameters, rather than by cascaded or dual-DiT branches that only regularize planning indirectly. UNIVERSE co-trains both modalities in one mask-modulated DiT and adds a Modality-Decoupling Visibility Mask that shares historical context while blocking mutual attention between future video and trajectory tokens. That design keeps training causally valid and lets the model drop future-video denoising at test time, giving roughly 4.3× faster trajectory-only inference with comparable accuracy. On NAVSIM the unified model reaches 91.0 PDMS versus 89.6 for a Two-DiT control, and it shows strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning.

What carries the argument

Modality-Decoupling Visibility Mask on a single shared Diffusion Transformer: future video and trajectory tokens may attend to the same historical context and to their own modality, but not to each other, yielding coupling-by-parameters and decoupling-by-visibility so one model supports joint, video-only, and trajectory-only rollouts.

What would settle it

Train a matched dual-DiT system with equal total parameters, the same pretrained backbone, and the same training budget on NAVSIM; if its zero-shot L2 and collision rates on nuScenes and Bench2Drive match or beat the single shared DiT, the claim that shared generative parameters are required for the transfer fails.

Watch

Extended reading notes

Core claim

Architecture determines how well future-video supervision helps planning: dense video dynamics improve action generalization most when future video latents and ego-trajectory tokens update one shared generative backbone. UNIVERSE places both inside a single Diffusion Transformer under a Modality-Decoupling Visibility Mask that shares history but blocks future video–trajectory attention, so video loss directly shapes trajectory denoising without future-target leakage, and trajectory-only inference remains valid and efficient.

Load-bearing premise

The Two-DiT and no-mask variants are treated as fair controls, so gains are attributed to shared parameters and the visibility mask rather than to capacity, optimization, or the large pretrained video backbone.

Editorial extensions

If this is right

  • Trajectory-only deployment can keep video co-training’s generalization benefit without paying multi-frame video diffusion cost at test time.
  • Cascaded or dual-DiT world–action designs leave more room for dataset-specific action overfitting than a shared backbone does.
  • The same trained weights can switch among trajectory-only planning, video-only simulation, and joint interpretable rollout by changing the mask and instantiated token groups.
  • Future-video prediction is more than auxiliary visualization: removing it measurably hurts closed-loop PDMS and zero-shot safety.
  • Ablations that isolate video loss, the visibility mask, and parameter sharing are sufficient to attribute which factor drives transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same coupling-by-parameters and decoupling-by-visibility pattern could apply to other multi-modal generative policies where one modality is expensive at test time, such as robot manipulation or navigation.
  • If training cost is dominated by the large video backbone, distilling the shared DiT into a trajectory-only student may preserve most of the generalization at lower train cost.
  • External odometry consistency between generated video and predicted trajectory is a reusable diagnostic for world–action models beyond this paper’s ablations.
  • Extending the mask to multi-view, map, or LiDAR tokens would test whether shared-backbone gains grow or shrink once richer history is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. UNIVERSE proposes a unified video–action world model for autonomous driving built on a single mask-modulated Diffusion Transformer that co-trains future video latents and ego-trajectory tokens under shared generative parameters. A Modality-Decoupling Visibility Mask shares historical context while blocking mutual attention between future video and trajectory tokens, enabling joint, video-only, and trajectory-only inference; trajectory-only rollout removes future-video denoising and yields a reported 4.3× speedup. The central claim is that coupling-by-parameters transfers video-learned dynamics to planning more effectively than cascaded or dual-DiT designs, while decoupling-by-visibility preserves causal validity and efficient deployment. Evidence includes NAVSIM closed-loop PDMS (91.0), controlled variants (no video co-train, no mask, Two-DiT), zero-shot transfer to nuScenes and Bench2Drive, FID/FVD video quality, DPVO-based video–action consistency, and efficiency tables.

Significance. If the architectural thesis holds, the paper offers a clear and practically useful design principle for driving WAMs: dense video supervision should update the same DiT parameters used for trajectory denoising, while future modalities remain causally separated at the token level. The combination of strong zero-shot transfer, flexible inference modes, and efficiency gains over coupled rollout is valuable for planning-oriented world models. Strengths include matched ablations isolating video co-training, the visibility mask, and shared vs. separated DiTs; external-benchmark evaluation; and explicit DPVO consistency checks. These make the work more than a pure systems bake-off and give the community a falsifiable comparison between shared-parameter and dual-DiT video–action designs.

major comments (3)
  1. [Sec. 3.5, 4.5; Table 5] Sec. 3.5 and Sec. 4.5 treat UNIVERSE Two-DiT as the fair control that preserves video co-training, conditions, losses, tokenization, and schedule while removing shared parameters. Table 5 shows substantially higher training peak memory (88.3 GB vs 52.1 GB) and higher inference memory/latency for Two-DiT, indicating two full DiT stacks rather than a parameter- or FLOP-matched dual backbone. With a large pretrained Wan2.2 backbone and joint flow matching (Eq. 6), the PDMS/zero-shot gap (91.0 vs 89.6; weaker Two-DiT transfer in Table 2) may partly reflect optimization/capacity allocation rather than pure coupling-by-parameters. A width/depth- or parameter-matched dual-DiT control (or explicit parameter/FLOP accounting with multi-seed variance) is needed to isolate the architectural claim.
  2. [Tables 1–2, 4–5; Sec. 4.5] All headline numbers (NAVSIM PDMS, zero-shot L2/collision, efficiency) appear to be single-run. The shared-vs-Two-DiT and video-co-train gaps are modest in absolute terms on NAVSIM (91.0 vs 89.6 / 88.2) but are used to support the central generalization thesis. Without seed variance, confidence intervals, or repeated training, it is hard to judge whether the architectural ranking is stable. At minimum, multi-seed means/std for the main variants on NAVSIM and one zero-shot setting would make the load-bearing comparisons credible.
  3. [Sec. 3.2–3.3, 4.5; Tables 1–2, 5] The no-mask variant matches in-domain PDMS (90.9 vs 91.0) but is worse under domain shift and much slower because coupled rollout is required (Tables 1–2, 5). The paper argues this shows the mask’s value for causal validity and flexible deployment. That is plausible, but the train–test dependency argument would be stronger with a direct measurement of train–test mismatch when future video is dropped for a model trained without the mask (e.g., attention-dependency diagnostics or forced trajectory-only evaluation of the no-mask model). Currently the safety/latency costs are shown, but the claimed causal-leakage mechanism is only indirectly supported.
minor comments (6)
  1. [Abstract, Sec. 1–2] Throughout the manuscript, spacing artifacts appear in acronyms and names (e.g., “W AMs”, “NA VSIM”, “BEV ,”, “WoV oGen”). These should be cleaned for camera-ready readability.
  2. [Sec. 3.1, Eq. (2)] Eq. (2) reuses Ts for both the text-encoder output and the earlier text-instruction symbol T; a distinct symbol for encoded text tokens would avoid confusion with the raw instruction.
  3. [Fig. 1–2] Fig. 1 and Fig. 2 are dense; enlarging the mask schematic and explicitly labeling which attention edges are blocked would help readers who skip Sec. 3.2.
  4. [Table 6; Sec. 4.6] Table 6 shows 12 future frames underperform 8 frames; a short discussion of why longer horizons hurt (noise accumulation, optimization trade-off, or resolution/compute) would help practitioners choose N.
  5. [Sec. 4.1; Limitations] Limitations correctly note front-view-only input and training cost of the video backbone; stating whether multi-view or map/LiDAR fusion is architecturally compatible with the same mask would clarify the path to more interactive scenes.
  6. [Sec. 2.2; Fig. 1] Related work is thorough; a short explicit comparison table of cascaded vs dual-DiT vs unified designs (inputs, shared parameters, test-time video required) would make the positioning in Fig. 1 easier to cite.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical architecture paper evaluated on external closed-loop and zero-shot benchmarks; claims are not forced by definition or self-citation.

full rationale

UNIVERSE is a systems paper whose load-bearing claims are empirical comparisons (NAVSIM PDMS, nuScenes/Bench2Drive L2 and collision rates, latency/memory) against external benchmarks and controlled ablations (w.o. video co-train, w.o. Mask, Two-DiT). The shared-DiT objective (Eq. 6), Modality-Decoupling Visibility Mask (Sec. 3.2 / Eq. 7), and flexible inference modes do not define the reported metrics in terms of fitted free parameters, nor do they rename a known result as a first-principles derivation. Self-citations (e.g., DriveVA, Fast-WAM positioning) are background, not uniqueness theorems that force the architecture. Capacity-matching concerns about Two-DiT vs shared DiT are experimental-design risks, not circular reductions of prediction to input. Score 0 is appropriate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central claim rests on empirical ML assumptions and design choices rather than formal axioms. Load-bearing ingredients are: flow-matching co-training of video and trajectory tokens; the hand-designed visibility mask; shared DiT parameterization; a large pretrained video backbone; and hyperparameter choices (horizon, sampling steps, loss weight). Invented entities are architectural constructs validated only by the paper’s ablations, not independent theory.

free parameters (5)
  • lambda_vid (video loss weight)
    Balances trajectory and video flow-matching losses; default >0 is required for the co-training claim, but the paper does not report a sensitivity sweep of the exact value.
  • future video horizon N
    Default N=8 frames chosen via ablation (4/8/12); PDMS depends on this choice (Table 6).
  • flow sampling steps at inference
    Default 2 steps; 1 step collapses performance, 3 steps match 2 (Table 6). Central efficiency/accuracy claims use this setting.
  • learning rate / batch / training schedule
    AdamW 1e-4, staged batch 80 then 640, 20k+10k steps, warmup—standard but claim-sensitive optimization knobs not derived from theory.
  • history length l and resolution
    l=4 history frames at 832×480 are fixed design choices that define the observation interface for all reported results.
assumptions (5)
  • domain assumption Dense future-video prediction provides useful world-dynamics supervision for ego trajectory generation under domain shift.
    Core WAM premise stated in Introduction and tested by the no-video-co-train ablation; not proven generally, only supported empirically here.
  • domain assumption Flow matching on latent video and trajectory tokens is a valid generative training objective for planning metrics.
    Sec. 3.4 adopts flow matching without deriving that velocity matching optimizes closed-loop PDMS/collision rates.
  • ad hoc to paper Blocking mutual attention between future video and trajectory tokens preserves causal validity while shared parameters still transfer dynamics.
    The Modality-Decoupling Visibility Mask (Sec. 3.2, App. B.1) is a design postulate validated by ablations, not a theorem.
  • domain assumption NAVSIM-trained front-view policies evaluated zero-shot on nuScenes/Bench2Drive measure transferable action priors rather than benchmark artifacts.
    Cross-domain claims in Sec. 4.3 depend on this evaluation interpretation.
  • standard math Standard transformer self-attention and DiT blocks can share modalities via token concatenation plus masks.
    Uses ordinary attention masking and shared parameters; mathematical machinery is standard.
invented entities (3)
  • UNIVERSE shared mask-modulated video-action DiT
    purpose: Single generative backbone that co-denoises future video latents and ego-trajectory tokens under shared parameters.
    Architectural construct introduced to transfer video dynamics directly into trajectory denoising; evidence is internal ablations and benchmarks only.
  • Modality-Decoupling Visibility Mask
    purpose: Share historical context across modalities while blocking future video–trajectory attention to prevent leakage and enable modality-flexible inference.
    Hand-designed binary attention pattern (Eq. 7 in supplement); falsifiable only via the paper’s own no-mask comparison, not an external physical entity.
  • Flexible Inference Controller (trajectory-only / video-only / joint modes)
    purpose: Instantiate selected future token groups at test time without retraining.
    Operational interface enabled by the mask; validated by latency and accuracy tables inside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation." pith.science (2026). https://pith.science/paper/D4N4OO53

@misc{pith2026260705133,
  author       = {Pith},
  title        = {Pith review of: UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4N4OO53}},
  note         = {Machine review of arXiv:2607.05133}
}
abstract

World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.

Figures

Figures reproduced from arXiv: 2607.05133 by the authors.

Figure 1
Figure 1. Comparison with previous video action models in autonomous driving. Prior cascaded or dual￾DiT designs separate video imagination from trajectory generation. UNIVERSE co-trains future video latents and ego trajectories within a single mask-modulated DiT, while enabling efficient trajectory-only inference by removing future-video denoising at test time. Our key finding is that architecture mat￾ters: video co-training… view at source ↗
Figure 2
Figure 2. Overall pipeline of UNIVERSE. Given history observations, ego velocity (vx, vy), and language instructions, UNIVERSE encodes conditional tokens with a text encoder and video VAE [53]. A unified DiT [23] jointly trains future video latents and action tokens, while the Modality￾Decoupling Visibility Mask blocks future video-action leakage but preserves shared historical context. The model supports joint, video-only, a… view at source ↗
Figure 3
Figure 3. Qualitative future video and trajectory generation. The predicted trajectories are consistent with the generated future video frames, showing that UNIVERSE learns aligned visual dynamics and ego-motion within one unified generation framework. Generated videos by UNIVERSER 1s 2s 3s Predicted Trajectories by UNIVERSER Generated videos by UNIVERSER Two DIT Ground Truth videos Predicted Trajectories by UNIVERSER Two DIT… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Zero-shot driving-intention comparison. UNIVERSE is evaluated on Bench2Drive without fine-tuning. Red and green curves denote predicted and ground-truth trajectories, respectively. When a stationary vehicle blocks the lane ahead, UNIVERSE predicts a bypass trajectory, …
Figure 5
Figure 5. Figure 5: Zero-shot video-action consistency across diverse driving scenarios. UNIVERSE is directly evaluated on unseen nuScenes and Bench2Drive scenes without fine-tuning, covering lead-vehicle following, urban lane change, and curved-road planning. For nuScenes, DPVO is used t…
Figure 6
Figure 6. Figure 6: Zero-shot qualitative results on nuScenes and Bench2Drive. UNIVERSE is trained on NAVSIM and directly evaluated on unseen target datasets without fine-tuning. The generated future videos remain consistent with the predicted ego trajectories across both real-world and s…
Figure 7
Figure 7. Figure 7: Zero-shot video generation and trajectory consistency on Bench2Drive. UNIVERSE is directly evaluated on unseen Bench2Drive scenes without fine-tuning, including challenging nighttime and foggy driving conditions. The generated future videos remain visually consistent w…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 30 linked inside Pith

  1. [1]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023

    2023

  2. [2]

    Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.IEEE transactions on pattern analysis and machine intelligence, 45(11):12878–12895, 2022

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.IEEE transactions on pattern analysis and machine intelligence, 45(11):12878–12895, 2022

    2022

  3. [3]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving

    Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12037–12047, 2025

    2025

  4. [4]

    World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    arXiv 2026

  5. [5]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.Open Review, 62(1):1–62, 2022

    2022

  6. [6]

    World models.arXiv preprint arXiv:1803.10122, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    arXiv 2018

  7. [8]

    Genad: Generative end-to-end autonomous driving

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. InEuropean Conference on Computer Vision, pages 87–104. Springer, 2024

    2024

  8. [10]

    Dreamgen: Unlocking generalization in robot learning through video world models

    Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loïc Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zh...

    2025

Show all 65 references
  1. [11]

    Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control.arXiv preprint arXiv:2601.21998, 2026

    2026 arXiv

  2. [12]

    Unified video action model.arXiv preprint arXiv:2503.00200, 2025

    Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model.arXiv preprint arXiv:2503.00200, 2025. 10

    2025 arXiv

  3. [13]

    Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025

    Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025

    2025 arXiv

  4. [14]

    Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    2026 arXiv

  5. [15]

    Imagidrive: A unified imagination-and-planning framework for autonomous driving.arXiv preprint arXiv:2508.11428, 2025

    Jingyu Li, Bozhou Zhang, Xin Jin, Jiankang Deng, Xiatian Zhu, and Li Zhang. Imagidrive: A unified imagination-and-planning framework for autonomous driving.arXiv preprint arXiv:2508.11428, 2025

    2025

  6. [16]

    Driveva: Video action models are zero-shot drivers

    Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Michael Ying Yang, Francesco Nex, and Hao Cheng. Driveva: Video action models are zero-shot drivers. arXiv preprint arXiv:2604.04198, 2026

    2026 arXiv

  7. [17]

    Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026

    Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, Min Cao, Peng Li, Qiuping Deng, Wenjun Mei, Xiaofeng Wang, Xinze Chen, Xinyu Zhou, Yang Wang, Yifan Chang, Yifan Li, Yukun Zhou, Yun Ye, Zhichao Liu, and Zheng Zhu. ...

    2026

  8. [18]

    Prophetdwm: A driving world model for rolling out future actions and videos.arXiv preprint arXiv:2505.18650, 2025

    Xiaodong Wang and Peixi Peng. Prophetdwm: A driving world model for rolling out future actions and videos.arXiv preprint arXiv:2505.18650, 2025

    2025 arXiv

  9. [19]

    mimic- video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025

    Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic- video: Video-action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025

    2025 arXiv

  10. [20]

    Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025

    Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sudhakar, Paarth Shah, Rares Ambrus, and Carl V ondrick. Video generators are robot policies.arXiv preprint arXiv:2508.00795, 2025

    2025 arXiv

  11. [21]

    Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163, 2026

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, et al. Cosmos policy: Fine-tuning video models for visuomotor control and planning.arXiv preprint arXiv:2601.16163, 2026

    2026 arXiv

  12. [22]

    Large video planner enables generalizable robot control

    Boyuan Chen, Tianyuan Zhang, Haoran Geng, Kiwhan Song, Caiyi Zhang, Peihao Li, William T Freeman, Jitendra Malik, Pieter Abbeel, Russ Tedrake, et al. Large video planner enables generalizable robot control. arXiv preprint arXiv:2512.15840, 2025

    2025 arXiv

  13. [23]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  14. [24]

    Dual-stream diffusion for world-model augmented vision-language-action model.arXiv preprint arXiv:2510.27607, 2025

    John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, and Jinwoo Shin. Dual-stream diffusion for world-model augmented vision-language-action model.arXiv preprint arXiv:2510.27607, 2025

    2025 arXiv

  15. [25]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5820–5829, 2021

    2021

  16. [26]

    Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    2023 arXiv

  17. [27]

    Drivedreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. InEuropean conference on computer vision, pages 55–72. Springer, 2024

    2024

  18. [28]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10412–10420, 2025

    2025

  19. [29]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  20. [30]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. Advances in Neural Information Processing Systems, 37:91560–91596, 2024

    2024

  21. [31]

    Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023. 11

    2023 arXiv

  22. [32]

    Gaia-2: A controllable multi-view generative world model for autonomous driving.arXiv preprint arXiv:2503.20523, 2025

    Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving.arXiv preprint arXiv:2503.20523, 2025

    2025 arXiv

  23. [33]

    Magicdrive-v2: High- resolution long video generation for autonomous driving with adaptive control

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive-v2: High- resolution long video generation for autonomous driving with adaptive control. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 28135–28144, 2025

    2025

  24. [34]

    Mila: Multi-view intensive-fidelity long-term video generation world model for autonomous driving

    Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, and Bing Wang. Mila: Multi-view intensive-fidelity long-term video generation world model for autonomous driving. arXiv preprint arXiv:2503.15875, 2025

    2025 arXiv

  25. [35]

    Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving.arXiv preprint arXiv:2503.22231, 2025

    Yishen Ji, Ziyue Zhu, Zhenxin Zhu, Kaixin Xiong, Ming Lu, Zhiqi Li, Lijun Zhou, Haiyang Sun, Bing Wang, and Tong Lu. Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving.arXiv preprint arXiv:2503.22231, 2025

    2025 arXiv

  26. [36]

    Genesis: Multimodal driving scene generation with spatio-temporal and cross-modal consistency.arXiv preprint arXiv:2506.07497, 2025

    Xiangyu Guo, Zhanqian Wu, Kaixin Xiong, Ziyang Xu, Lijun Zhou, Gangwei Xu, Shaoqing Xu, Haiyang Sun, Bing Wang, Guang Chen, et al. Genesis: Multimodal driving scene generation with spatio-temporal and cross-modal consistency.arXiv preprint arXiv:2506.07497, 2025

    2025 arXiv

  27. [37]

    Omninwm: Omniscient driving navigation world models.arXiv preprint arXiv:2510.18313, 2025

    Bohan Li, Zhuang Ma, Dalong Du, Baorui Peng, Zhujin Liang, Zhenqiang Liu, Chao Ma, Yueming Jin, Hao Zhao, Wenjun Zeng, et al. Omninwm: Omniscient driving navigation world models.arXiv preprint arXiv:2510.18313, 2025

    2025 arXiv

  28. [38]

    Occworld: Learning a 3d occupancy world model for autonomous driving

    Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. InEuropean conference on computer vision, pages 55–72. Springer, 2024

    2024

  29. [39]

    Driveworld: 4d pre-trained scene understanding via world models for autonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and patte...

    2024

  30. [40]

    Drivingworld: Constructing world model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024

    Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Drivingworld: Constructing world model for autonomous driving via video gpt.arXiv preprint arXiv:2412.19505, 2024

    2024 arXiv

  31. [41]

    Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation

    Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. InEuropean Conference on Computer Vision, pages 329–345. Springer, 2024

    2024

  32. [42]

    Uniscene: Unified occupancy-centric driving scene generation

    Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 11971–11981, 2025

    2025

  33. [43]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1...

    2024

  34. [44]

    Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers

    Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 26890–26900, 2025

    2025

  35. [45]

    Enhancing end-to-end autonomous driving with latent world model.arXiv preprint arXiv:2406.08481, 2024

    Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model.arXiv preprint arXiv:2406.08481, 2024

    2024 arXiv

  36. [46]

    Doe-1: Closed-loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627, 2024

    Wenzhao Zheng, Zetian Xia, Yuanhui Huang, Sicheng Zuo, Jie Zhou, and Jiwen Lu. Doe-1: Closed-loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627, 2024

    2024 arXiv

  37. [47]

    Resim: Reliable world simulation for autonomous driving

    Jiazhi Yang, Kashyap Chitta, Shenyuan Gao, Long Chen, Yuqian Shao, Xiaosong Jia, Hongyang Li, Andreas Geiger, Xiangyu Yue, and Li Chen. Resim: Reliable world simulation for autonomous driving. arXiv preprint arXiv:2506.09981, 2025

    2025 arXiv

  38. [48]

    End-to-end driving with online trajectory evaluation via bev world model.arXiv preprint arXiv:2504.01941, 2025

    Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model.arXiv preprint arXiv:2504.01941, 2025

    2025 arXiv

  39. [49]

    Drivelaw: Unifying planning and video generation in a latent driving world.arXiv preprint arXiv:2512.23421, 2025

    Tianze Xia, Yongkang Li, Lijun Zhou, Jingfeng Yao, Kaixin Xiong, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, et al. Drivelaw: Unifying planning and video generation in a latent driving world.arXiv preprint arXiv:2512.23421, 2025. 12

    2025 arXiv

  40. [50]

    Epona: Autoregressive diffusion world model for autonomous driving

    Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, et al. Epona: Autoregressive diffusion world model for autonomous driving. arXiv preprint arXiv:2506.24113, 2025

    2025 arXiv

  41. [51]

    From forecasting to planning: Policy world model for collaborative state-action prediction.arXiv preprint arXiv:2510.19654, 2025

    Zhida Zhao, Talas Fu, Yifan Wang, Lijun Wang, and Huchuan Lu. From forecasting to planning: Policy world model for collaborative state-action prediction.arXiv preprint arXiv:2510.19654, 2025

    2025

  42. [52]

    Drivevla-w0: World models amplify data scaling law in autonomous driving.arXiv preprint arXiv:2510.12796, 2025

    Yingyan Li, Shuyao Shang, Weisong Liu, Bing Zhan, Haochen Wang, Yuqi Wang, Yuntao Chen, Xiaoman Wang, Yasong An, Chufeng Tang, et al. Drivevla-w0: World models amplify data scaling law in autonomous driving.arXiv preprint arXiv:2510.12796, 2025

    2025 arXiv

  43. [53]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  44. [54]

    Videovla: Video generators can be generalizable robot manipulators.arXiv preprint arXiv:2512.06963, 2025

    Yichao Shen, Fangyun Wei, Zhiying Du, Yaobo Liang, Yan Lu, Jiaolong Yang, Nanning Zheng, and Baining Guo. Videovla: Video generators can be generalizable robot manipulators.arXiv preprint arXiv:2512.06963, 2025

    2025

  45. [55]

    Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing Systems, 37:28706– 28719, 2024

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.Advances in Neural Information Processing ...

    2024

  46. [56]

    Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving

    OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving. InProceedings of the Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, pages 18–22, 2023

    2023

  47. [57]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern r...

    2020

  48. [58]

    Vad: Vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, pag...

    2023

  49. [59]

    Bench2drive: Towards multi- ability benchmarking of closed-loop end-to-end autonomous driving.Advances in Neural Information Processing Systems, 37:819–844, 2024

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi- ability benchmarking of closed-loop end-to-end autonomous driving.Advances in Neural Information Processing Systems, 37:819–844, 2024

    2024

  50. [60]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. InConference on robot learning, pages 1–16. PMLR, 2017

    2017

  51. [61]

    How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence.IEEE Transactions on Intelligent Vehicles, 9(1):593–612, 2023

    Xuemin Hu, Shen Li, Tingyu Huang, Bo Tang, Rouxing Huai, and Long Chen. How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence.IEEE Transactions on Intelligent Vehicles, 9(1):593–612, 2023

    2023

  52. [62]

    V ADv2: End-to-end autonomous driving via probabilistic planning

    Bo Jiang, Shaoyu Chen, Hao Gao, Bencheng Liao, Qian Zhang, Wenyu Liu, and Xinggang Wang. V ADv2: End-to-end autonomous driving via probabilistic planning. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  53. [63]

    Para-drive: Parallelized architecture for real-time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449–15458, 2024

    2024

  54. [64]

    Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving.arXiv preprint arXiv:2506.08052, 2025

    Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, et al. Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving.arXiv preprint arXiv:2506.08052, 2025

    2025 arXiv

  55. [65]

    Deep patch visual odometry.Advances in Neural Information Processing Systems, 36:39033–39051, 2023

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry.Advances in Neural Information Processing Systems, 36:39033–39051, 2023

    2023

  56. [66]

    St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via spatial-temporal feature learning. InEuropean Conference on Computer Vision, pages 533–549. Springer, 2022

    2022

  57. [67]

    Scene as occupancy

    Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8406–8415, 2023. 13 A Supplementary Material This supplement...

    2023

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.