Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

Physical Informed Driving World Model

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DrivePhysica claims a diffusion-based driving world model can generate multi-view videos that obey motion, temporal, occlusion, and cross-view consistency, reporting SOTA FID/FVD and perception gains.

desk verdict DrivePhysica is a coherent engineering effort with a credible data-augmentation result, but the SOTA FID/FVD claim appears to rely on an unstated (T+I)2V protocol that is easier than the baselines' tasks. read the letter →

arxiv 2412.08410 v2 pith:G7VQQDVG submitted 2024-12-11 cs.CV

Zhuoran Yang , Xi Guo , Chenjing Ding , Chiyu Wang , Wei Wu This is my paper · ORCID
classification cs.CV
keywords drivingworldmodelmulti-viewvideogenerationdiffusion3DinstanceflowocclusionhierarchynuScenescontrollableautonomous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DrivePhysica is a diffusion-based driving world model that generates multi-view street videos designed to obey the physics of driving: moving objects should have the correct absolute and ego-relative motion, object colors and shapes should stay stable across frames, nearer objects should occlude farther ones, and the same vehicle should look the same across cameras. The paper argues that previous world models fail these physical constraints, and that encoding them explicitly as control conditions is what lets a world model produce videos that are both realistic and useful for training perception systems. On the nuScenes validation set the model reports an FID of 3.96 and FVD of 38.06, and generated videos reach 86.38% of the real-data NDS when scored by a pretrained video-based detector. When generated frames are mixed into real training data, the detector's NDS rises by 3.6 points over training on real data alone, and training only on generated data recovers over 90% of the real-data score. If these numbers hold, world models become a practical way to manufacture labeled driving video, including rare safety-critical events, without new annotation effort.

What carries the argument

The central object is a three-part control stack fused through a ControlNet into a spatial-temporal diffusion transformer. The Coordinate System Aligner embeds camera intrinsics and extrinsics with Fourier features, then uses cross-attention to fuse vehicle-coordinate conditions (3D bounding-box projections, road-map projections, and box-coordinate embeddings) with a world-coordinate condition (the instance-flow map), so the model can interpret relative motion against absolute motion. The Instance Flow Guidance module converts the 3D position offsets of each instance between consecutive frames into an RGB trajectory map, encodes it with the video autoencoder, and uses it to propagate object attributes such as color and texture across time. The Box Coordinate Guidance module Fourier-embeds the eight corners of each 3D bounding box together with its class label, giving the model explicit depth cues for occlusion hierarchy. Cross-view consistency is handled by reshaping the multi-view input so all views are concatenated along the width dimension, creating a parameter-free spatial view-inflated attention.

What would settle it

A controlled rerun of the Table 1 baselines under identical resolution, frame count, camera rig, full validation set, and several seeds, showing FID, FVD, or NDS equal to or better than 3.96, 38.06, and 86.38%, would refute the state-of-the-art claim.

Watch

Extended reading notes

Core claim

DrivePhysica establishes that physically grounded control conditions, rather than more data or a larger generator, are what let a driving world model produce coherent multi-view video. The model aligns conditions expressed in the ego-vehicle coordinate system with conditions expressed in the absolute world coordinate system, uses per-instance 3D motion offsets to keep object attributes stable over time, and encodes 3D bounding-box coordinates so the generator can resolve depth ordering and occlusion. On the nuScenes validation set this yields an FID of 3.96 and FVD of 38.06, outperforming the compared image-based and video-based baselines, and the generated videos score 86.38% relative NDS with a pretrained perception model. Training a detector on generated data alone reaches 92.69% of the mAP and 90.41% of the NDS of training on real data alone, while mixing generated with real data adds 3.6 NDS points over real-only training.

Load-bearing premise

The headline comparison assumes every prior method in Table 1 was evaluated under the same protocol—same resolution, frame count, camera rig, and the full validation set without sample selection—because the paper reports the baselines' published numbers without rerunning them.

Editorial extensions

If this is right

  • If DrivePhysica's numbers reproduce, generated nuScenes videos can serve as a drop-in substitute for real training data: a detector trained only on generated data reaches 92.69% of the mAP and 90.41% of the NDS of one trained only on real data.
  • Mixing generated and real data gives a concrete, immediately usable gain: +3.6 NDS over real-only training, which is larger than the +2.3 NDS reported for the Panacea baseline under the same recipe.
  • Because the generator is conditioned on layouts, users can manufacture long-tail events such as lane cutting and sudden braking from simulator-provided layouts, producing videos of those events on demand without collecting rare real footage.
  • Editing only the text prompt changes weather and time of day while geometry and motion conditions stay fixed, which is a cheap way to diversify training data for perception models.
  • The reported FID/FVD gains support the claim that a physically informed condition design, rather than a larger backbone, is the bottleneck for coherent driving-video generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the 3D instance-flow condition is what drives temporal consistency, then a similar offset-map conditioning could transfer to other layout-conditioned video generators, such as simulation-to-real dashcam translation, without retraining the whole backbone.
  • Editorial inference: The reported state-of-the-art margins should be read as upper bounds until the baselines are rerun under an identical protocol with matched resolution, frame count, camera rig, and multiple seeds; a controlled rerun could narrow or invert the margin.
  • Editorial inference: Combining generated and real data at scale could shift the cost curve of autonomous-driving perception by letting one label boxes in a simulator and synthesize unlimited variations, though the paper does not test whether the +3.6 NDS gain saturates as the synthetic fraction grows.
  • Editorial inference: The ablation tables suggest the Instance Flow module matters most when the first frame is not provided, so pure text-to-video generation is the regime where temporal consistency is still the fragile part and deserves targeted stress tests on diverse weather and night scenes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DrivePhysica, a multi-view driving video world model built on OpenSora's ST-DiT, augmented with three modules: a Coordinate System Aligner (CSA) that fuses ego- and world-coordinate conditions via camera pose, an Instance Flow Guidance (IFG) module that supplies per-instance 3D motion offsets, and a Box Coordinate Guidance (BCG) module that encodes 3D bounding box coordinates. The authors claim that these modules enforce physical principles such as relative/absolute motion, temporal consistency, occlusion hierarchy, and cross-view consistency, and report state-of-the-art video generation quality on nuScenes (FID 3.96, FVD 38.06) together with downstream perception gains: 86.38% relative NDS with a pretrained StreamPETR and a +3.6 NDS improvement when generated data are mixed with real training data. The paper also includes ablations in both (T+I)2V (first-frame-conditioned) and T2V (text/layout-to-video) modes, qualitative comparisons against Panacea, and demonstrations of CARLA-based long-tail scenario generation.

Significance. If the quantitative claims are supported under a fair protocol, the paper would make a useful contribution to controllable driving video generation: it introduces a compact set of 3D-grounded control signals (instance flow, box coordinates, camera-pose-aligned conditions), provides an interesting data-augmentation study with a stronger real-data baseline than Panacea (Table 3), and demonstrates scalable long-tail condition generation via CARLA. The ablations in Tables 4-6 are directionally consistent and support the usefulness of the proposed modules. The main weakness is that the headline SOTA comparison in Table 1 appears to compare different conditioning protocols, so the central claim of state-of-the-art generation quality is not currently established.

major comments (3)
  1. [Table 1, Section 3.3.1] Table 1 compares DrivePhysica against text/layout-to-video baselines without stating which generation mode is used, while Section 3.2 explicitly enables image-to-video generation by setting a 0.2 probability of no noise on the first frame. Tables 4 and 5 show that the same model obtains FVD 38.06 in (T+I)2V mode but FVD 107.50 in T2V mode. The reported Table 1 numbers (FVD 38.06, FID 3.96) therefore most likely correspond to the (T+I)2V protocol, in which the model receives a real first frame and, per Sections 2.2.2 and 2.2.3, ground-truth per-frame 3D boxes and instance flow. This is a materially easier conditional generation task than the protocol used for the baselines, and it explains why the reported margin over methods such as Panacea (139 FVD) and DriveDreamer2 (55.7 FVD) is much larger than the paper's own T2V ablation (107.50 FVD) would suggest. The headline SOTA claim is therefore not supported unless the protocol is stated and the baselines are rerun under the same conditions (or the comparison is made in T2V mode).
  2. [Tables 2-3, Section 3.3.1] The perception comparisons in Tables 2 and 3 have the same protocol-mismatch problem. DrivePhysica rows are explicitly labeled (T+I)2V, whereas Panacea is not rerun in that mode; Panacea's published numbers are from its own text/layout-conditioned protocol. The reported 86.38% relative NDS and the +3.6 NDS data-augmentation gain may therefore reflect the additional first-frame and per-frame 3D-condition information available to DrivePhysica rather than a model-level advantage. The authors should either rerun Panacea (and ideally other baselines) under the identical (T+I)2V protocol, or clearly qualify the comparison as cross-protocol.
  3. [Sections 2.2.2, 2.2.3, and 3.3.2] The evaluation of 'physical-law adherence' is partly circular with respect to the conditioning. The model is conditioned on ground-truth per-frame 3D box coordinates (BCG) and ground-truth instance flow offsets (IFG), and the qualitative demonstrations in Figures 4 and 7 show that the generated videos match these supplied conditions. That is evidence of control precision, but it is not independent evidence that the model has learned physical principles such as occlusion hierarchy or temporal consistency from unconstrained inputs. The paper should either reframe the physical-law claims as controllability results, or provide T2V-mode evaluations (with conditions withheld) showing that physical consistency degrades gracefully and is not merely copied from the input conditions. The T2V ablations in Table 5 are a step in this direction, but they are not linked to the physical-law metrics.
minor comments (6)
  1. [Section 3.2] The sentence 'we set a 0.2 probability of not adding noise to the first frame' should be expanded to state whether the reported main results in Table 1 and Tables 2-3 use this image-to-video capability, and to specify the total number of evaluation videos, the frame count, and the resolution used for FID/FVD computation.
  2. [Equation (2)] There is a typographical error in Equation (2): the line ends with a double comma ('MLPp(Fourier(bi_t)), ,').
  3. [Section 3.3.1] The phrase 'the entire validation set of the nuSences' contains a typo; it should be 'nuScenes'.
  4. [Section 2.2.1] The statement that '¯P contains values from sine and cosine functions' is confusing because K, R, and T are raw camera parameters; it is the Fourier embedding that produces sine and cosine values. This sentence should be reworded for clarity.
  5. [Section 2.2.3] In the box coordinate encoding, b_i^t is described as an 8x3 corner-point matrix, but Equation (2) applies Fourier embedding and an MLP to it as a single object; please clarify whether the embedding is applied per corner point and then pooled or concatenated.
  6. [Introduction and Section 2.2] The introduction mentions 'parameter-free spatial view-inflated attention,' but the method section only describes reshaping the input so that views are treated as part of the frame width. Please specify where and how view-inflated attention is implemented, or remove the term if it refers to this reshaping.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline FVD/FID and physical-law evaluations are conditioned, in (T+I)2V mode, on the exact GT boxes and instance flow used as the target; partial circularity, though T2V ablations and data-augmentation experiments remain independent.

  1. fitted input called prediction [Sec. 3.3.1, Tables 1, 4, 5; Sec. 3.2]
    "Table 1: 'DrivePhysica ✓ ✓ 38.06 3.96'. Table 4: 'Ablation study results in (T+I)2V scenarios on the generated nuScenes validation set ... DrivePhysica 38.06 3.96'. Table 5: 'Ablation study results in T2V scenarios ... DrivePhysica 107.50 12.91'. Sec. 3.2: 'we set a 0.2 probability of not adding noise to the first frame and assigned a timestep of 0 to the first frame, enabling the model to have image-to-video generation capability.'"

    The headline numbers in Table 1 are identical to the (T+I)2V row in Table 4, not to the T2V row in Table 5. In (T+I)2V mode the model receives the real first frame and per-frame GT 3D boxes and instance flow, so FID/FVD measure a conditional reconstruction task. The Table 1 baselines (BEVGen, BEVControl, WoVoGen, Drive-WM, DriveDreamer, Panacea, DriveDreamer2) are not evaluated in this image-to-video mode; the claimed SOTA margin is therefore driven by supplying the target's first frame and annotation sequence as input, i.e., the 'prediction' contains the target by construction.

  2. self definitional [Sec. 2.2.2 and Sec. 3.3.2]
    "Sec. 2.2.2: 'F i offset = { (x i j − x i−1 j , y i j − y i−1 j , z i j − z i−1 j ) } N j=0 , which serves as a basis to track and propagate attributes (such as color and texture) over frames...' Sec. 3.3.2: 'Our model accurately captures the relative motion of each instance, demonstrating a precise understanding of both the vehicle coordinate system and the world coordinate system.'"

    The 'relative motion' whose understanding is claimed is exactly the input condition F_offset, computed from ground-truth world positions and fed to the model as instance flow. The qualitative physical-law evaluation thus checks whether the video renders the already-supplied motion and depth order (also supplied via Box Coordinate Guidance), not whether the model independently predicted or derived the law. The 'physical law adherence' is defined as alignment to conditions that already encode the law, making the demonstration self-definitional.

full rationale

The strongest circularity burden is the (T+I)2V evaluation protocol: the headline FVD/FID (3.96/38.06) matches the (T+I)2V ablation row rather than the T2V row (12.91/107.50), and the perception-validation numbers measure how well a detector re-finds GT annotations that were fed back into the generator as conditions. These central 'physical-law' and 'SOTA quality' claims therefore reduce, in part, to conditioning on the target quantities. Not all claims are circular: the T2V ablations (Table 5), the mixed-data training experiment (Table 3, +3.6 NDS on real validation), and the CARLA long-tail simulation are external, falsifiable benchmarks that do not depend on using GT validation conditions as inputs. There is no load-bearing self-citation or imported uniqueness theorem; self-citations [8,9] appear only in related work. On balance the paper's core quantitative comparisons are partially circular via the conditioning protocol, while the data-augmentation results retain independent content.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on domain assumptions about annotation quality, detector-proxy validity, and benchmark comparability, not on mathematical axioms. The free-parameter and invented-entity ledgers are empty because the paper introduces no physical constants and no new postulated entities; its contributions are engineered conditioning modules.

assumptions (3)
  • domain assumption Ground-truth nuScenes 3D bounding boxes, camera poses, and road maps are accurate and available as conditioning for every generated validation video.
    Section 2.2 and Section 3.3.1 use these annotations to generate and evaluate the validation set; if they are noisy or are treated as privileged information, the controllability numbers are optimistic for real unlabeled deployment.
  • domain assumption StreamPETR detection quality on generated videos is a valid surrogate for physical-law adherence and control precision.
    Tables 2, 3, and 6 interpret NDS and mAP as evidence of layout and motion fidelity; if the detector is insensitive to occlusion, flow, or temporal errors, the physical-law conclusions do not follow.
  • domain assumption Reported FID and FVD baselines were computed under protocols comparable to the authors' setup.
    Table 1 mixes published numbers from different papers without rerunning them; the state-of-the-art claim depends on this comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Physical Informed Driving World Model." pith.science (2026). https://pith.science/paper/G7VQQDVG

@misc{pith2026241208410,
  author       = {Pith},
  title        = {Pith review of: Physical Informed Driving World Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7VQQDVG}},
  note         = {Machine review of arXiv:2412.08410}
}
read the original abstract

Autonomous driving requires robust perception models trained on high-quality, large-scale multi-view driving videos for tasks like 3D object detection, segmentation and trajectory prediction. While world models provide a cost-effective solution for generating realistic driving videos, challenges remain in ensuring these videos adhere to fundamental physical principles, such as relative and absolute motion, spatial relationship like occlusion and spatial consistency, and temporal consistency. To address these, we propose DrivePhysica, an innovative model designed to generate realistic multi-view driving videos that accurately adhere to essential physical principles through three key advancements: (1) a Coordinate System Aligner module that integrates relative and absolute motion features to enhance motion interpretation, (2) an Instance Flow Guidance module that ensures precise temporal consistency via efficient 3D flow extraction, and (3) a Box Coordinate Guidance module that improves spatial relationship understanding and accurately resolves occlusion hierarchies. Grounded in physical principles, we achieve state-of-the-art performance in driving video generation quality (3.96 FID and 38.06 FVD on the Nuscenes dataset) and downstream perception tasks. Our project homepage: https://metadrivescape.github.io/papers_project/DrivePhysica/page.html

Figures

Figures reproduced from arXiv: 2412.08410 by the authors.

Figure 1
Figure 1. Limitations of previous works in modeling physical laws within driving scenarios. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of DrivePhysica. (a) refers to Coordinate System Aligner module, which uses camera pose parameters to align the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the computation process for [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of videos generated by our model and Panacea, the current state-of-the-art method. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: DrivePhysica’s editing capability by adjusting the weather and time of day. By adding ”Rainy,” ”Sunny,” and ”Night” to the original text prompt, while keeping other conditions (such as camera pose, 3D bounding box coordinates, 3D bounding box projections, road map proj…
Figure 6
Figure 6. Figure 6: DrivePhysica’s ability to generate rare but critical driving scenarios based on layout conditions provided by the Carla simulator. The use of Carla-generated layouts addresses a critical limitation in real-world driving video datasets: the lack of diversity in scene ty…
Figure 7
Figure 7. Figure 7: Precise control mechanisms. We overlay the 3D bounding box projections onto the generated videos. The precision of control is [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Diverse videos using varying noise inputs and the same control conditions. By introducing stochastic noise while maintaining consistent control signals—such as 3D bounding box coordinates, lane line projections, and instance flow—our model can produce a variety of vide…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A five-aspect, 24-metric benchmark, a 26K human-annotated dataset, and an AI evaluator show that today's driving world models cannot simultaneously look real, respect geometry, and behave safely.

  2. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

  3. Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities

    cs.RO 2025-09 conditional novelty 4.0 of 10

    Foundation-model perception for autonomous driving is surveyed through four capability lenses: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding.

  4. A Survey of World Models for Autonomous Driving

    cs.RO 2025-01 conditional novelty 2.0 of 10

    A survey presenting a three-branch taxonomy of world models for autonomous driving, plus benchmark tables comparing representative generation and planning methods on nuScenes, Waymo, Occ3D, and CarlaSC.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages · cited by 4 Pith papers

  1. [1]

    Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion

    Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video genera- tion. arXiv preprint arXiv:2304.08477, 2023. 17

    arXiv 2023

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  3. [3]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. 2023. 17

    work page 2023

  4. [4]

    Vir- tual kitti 2

    Yohann Cabon, Naila Murray, and Martin Humenberger. Vir- tual kitti 2. arXiv preprint arXiv:2001.10773, 2020. 18

    arXiv 2001

  5. [5]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 7

    work page 2020

  6. [6]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 11, 16

    work page 2017

  7. [7]

    Magicdrive: Street view generation with diverse 3d geometry control

    Ruiyuan Gao, Kai Chen, Enze Xie, Hong Lanqing, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. In ICLR, 2024. 1, 17

    work page 2024

  8. [8]

    Gan-based virtual- to-real image translation for urban scene semantic segmen- tation

    Xi Guo, Zhicheng Wang, Qin Yang, Weifeng Lv, Xiang- long Liu, Qiong Wu, and Jian Huang. Gan-based virtual- to-real image translation for urban scene semantic segmen- tation. Neurocomputing, 394:127–135, 2020. 18

    work page 2020

Show all 51 references
  1. [9]

    Learning video rep- resentations of human motion from synthetic data

    Xi Guo, Wei Wu, Dongliang Wang, Jing Su, Haisheng Su, Weihao Gan, Jian Huang, and Qin Yang. Learning video rep- resentations of human motion from synthetic data. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20197–20207, 2022. 18

    2022

  2. [10]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 17

    2023 arXiv

  3. [11]

    Latent video diffusion models for high-fidelity video generation with arbitrary lengths

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221, 2022. 17

    2022 arXiv

  4. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Proceedings of the 31st International Conference on Neural Information Processing Systems , page 6...

    2017

  5. [13]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. 1

    2021

  6. [14]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 1

    2020

  7. [15]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 17

    arXiv

  8. [16]

    Adriver-i: A general world model for autonomous driving

    Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549, 2023. 1

    2023 arXiv

  9. [17]

    Drivingdiffu- sion: Layout-guided multi-view driving scene video gener- ation with latent diffusion model, 2023

    Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffu- sion: Layout-guided multi-view driving scene video gener- ation with latent diffusion model, 2023. 2, 18

    2023

  10. [18]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023. 6

    2023

  11. [19]

    Wovogen: World volume-aware diffusion for con- trollable multi-camera driving scene generation, 2024

    Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for con- trollable multi-camera driving scene generation, 2024. 7

    2024

  12. [20]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, pages 405–421. Springer, 2020. 4

    2020

  13. [21]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  14. [22]

    SDXL: improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952,

    arXiv

  15. [23]

    Exploring the limits of transfer learning with a uni- fied text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, et al. Exploring the limits of transfer learning with a uni- fied text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  16. [24]

    Hierarchical text-conditional im- age generation with clip latents, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional im- age generation with clip latents, 2022. URL https://arxiv. org/abs/2204.06125, 7, 2022. 17

    2022 arXiv

  17. [25]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  18. [26]

    The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes

    German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234–3243,

  19. [27]

    Uniad: A unified ad hoc data processing system.ACM Trans- actions on Database Systems (TODS), 42(1):1–42, 2016

    Xiaogang Shi, Bin Cui, Gillian Dobbie, and Beng Chin Ooi. Uniad: A unified ad hoc data processing system.ACM Trans- actions on Database Systems (TODS), 42(1):1–42, 2016. 1

    2016

  20. [28]

    Learning 12 from simulated and unsupervised images through adversarial training

    Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning 12 from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2107–2116, 2017. 18

    2017

  21. [29]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  22. [30]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. In ICLR, 2020. 1

    2020

  23. [31]

    Street- view image generation from a bird’s-eye view layout

    Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street- view image generation from a bird’s-eye view layout. arXiv preprint arXiv:2301.04634, 2023. 7, 17

    2023 arXiv

  24. [32]

    Mvdiffusion: Enabling holistic multi- view image generation with correspondence-aware diffusion

    Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi- view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097, 2023. 17

    2023 arXiv

  25. [33]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 2...

    2017

  26. [34]

    Consistent view synthesis with pose-guided diffusion models

    Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia- Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In CVPR, 2023. 17

    2023

  27. [35]

    To- wards accurate generative models of video: A new metric challenges

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. To- wards accurate generative models of video: A new metric challenges. arXiv:1812.01717, 2018. 7

    2018 arXiv

  28. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 17

    2017

  29. [37]

    Exploring object-centric temporal modeling for efficient multi-view 3d object detection

    Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xi- angyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In CVPR, pages 3621–3631, 2023. 7, 14

    2023

  30. [38]

    Videofactory: Swap at- tention in spatiotemporal diffusions for text-to-video gener- ation, 2023

    Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap at- tention in spatiotemporal diffusions for text-to-video gener- ation, 2023. 17

    2023

  31. [39]

    Drivedreamer: Towards real-world- driven world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023. 1, 7

    2023 arXiv

  32. [40]

    Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving, 2023

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving, 2023. 1, 7

    2023

  33. [41]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  34. [42]

    Bevcontrol: Accurately controlling street- view elements with multi-perspective consistency via bev sketch layout

    Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street- view elements with multi-perspective consistency via bev sketch layout. arXiv preprint arXiv:2308.01661 , 2023. 7, 17

    2023 arXiv

  35. [43]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. 2023. 17

    2023

  36. [44]

    Moonshot: To- wards controllable video generation and editing with multi- modal conditions, 2024

    David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sahoo. Moonshot: To- wards controllable video generation and editing with multi- modal conditions, 2024. 17

    2024

  37. [45]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 5

    2023

  38. [46]

    Collaborative and adversarial network for unsupervised do- main adaptation

    Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised do- main adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3801–3809,

  39. [47]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. arXiv preprint arXiv:2403.06845,

    arXiv

  40. [48]

    Genad: Generative end-to-end au- tonomous driving

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end au- tonomous driving. arXiv preprint arXiv:2402.11502, 2024. 1

    2024 arXiv

  41. [49]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3, 7

    2024

  42. [50]

    patchified

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023. 17 13 A. More Experimental Details We provide a webpage https://github.com/DrivePhysica for additional video results. B. More...

    2023

  43. [51]

    Simulation-to-Real Visual Translation

    adopts a multi-stage pipeline involving multiple mod- els and extensive post-processing, our approach simplifies the workflow through an efficient, end-to-end framework, ensuring both temporal coherence and computational effi- ciency. Simulation-to-Real Visual Translation. Rec...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.