Pith. sign in

REVIEW 4 major objections 8 minor 4 cited by

Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One model generates aligned driving video and LiDAR.

desk verdict A serious joint video+LiDAR generation system with a genuinely new coupling idea, but the ablation evidence for its central mechanism contradicts itself and the headline consistency claim is not directly measured. read the letter →

arxiv 2506.07497 v4 pith:5K2ADKD3 submitted 2025-06-09 cs.CV

Xiangyu Guo , Zhanqian Wu , Kaixin Xiong , Ziyang Xu , Lijun Zhou , Gangwei Xu , Shaoqing Xu , Haiyang Sun
show 5 more authors
Bing Wang Guang Chen Hangjun Ye Wenyu Liu Xinggang Wang
This is my paper
classification cs.CV
keywords jointgenerationmulti-viewvideoLiDARsequencecross-modalconsistencybird's-eye-viewfeaturesdiffusiontransformersdrivingscenesynthesisnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents Genesis, a unified framework for jointly generating multi-view driving videos and LiDAR point-cloud sequences from the same scene description, so that what the cameras see and what the LiDAR senses stay consistent in space and time. Previous methods generate each modality separately or rely on expensive occupancy labels to couple them. Genesis instead couples the branches through a shared latent space and through bird's-eye-view (BEV) features lifted from the generated images, and it adds structured scene- and instance-level captions produced by a vision-language model. On the nuScenes benchmark it reports state-of-the-art numbers for both modalities (video FVD 16.95, FID 4.24; LiDAR Chamfer distance 0.611 with ground-truth images and 0.634 with its own generated images), and it shows that the generated data improves downstream BEV segmentation and 3D detection. If correct, this makes it possible to synthesize unlimited paired camera-LiDAR training data for driving perception without manual annotation.

What carries the argument

The load-bearing mechanism is the shared-conditioning bridge through a bird's-eye-view latent space: the LiDAR generation branch receives, as a condition input, the camera branch's RGB video outputs lifted to BEV features by the LSS algorithm, concatenated with the scene-layout latent. This is what is supposed to make the point cloud agree with the multi-view video without requiring occupancy grids. The second essential device is DataCrafter, a vision-language-model-based captioning module that produces structured scene-level and instance-level descriptions injected as cross-attention conditions into both branches. The third device is the pair of autoencoders (a 3D-VAE for video, a BEV autoencoder with NeRF-based decoding for LiDAR) that compress both modalities into compact latents so that diffusion operates on them.

What would settle it

Run Genesis on validation scenes where the video branch is known to hallucinate, such as occluded or truncated vehicles, and measure a direct reprojection error between the generated LiDAR points and the generated image pixels. If that error stays low even when the object is removed from the layout, or if the LiDAR Chamfer distance does not degrade when the image-BEV condition is replaced by a deliberately misaligned image, then the cross-modal consistency is not actually enforced by the shared latent space.

Watch

Extended reading notes

Core claim

Genesis's central claim is that one model can jointly synthesize multi-view RGB video and LiDAR sequences with spatio-temporal and cross-modal consistency, using only shared scene captions and layouts as controlling conditions. The video branch is a spatiotemporal diffusion transformer with a 3D variational autoencoder; the LiDAR branch is a BEV-represented autoencoder with NeRF-style rendering, a spatiotemporal diffusion module, and a ControlNet-style conditioning path. The two branches are coupled by converting the camera branch's RGB output into BEV features with the LSS lifting algorithm and feeding those features, together with encoded 3D boxes and captions, into the LiDAR branch. The paper reports the best known video and LiDAR metrics on nuScenes and shows that jointly generated data yields the largest downstream gains in 3D detection (mAP from 66.87 to 67.78, NDS from 69.65 to 71.13 on BEVFusion), which it reads as evidence that the synthetic data is semantically and geometrically faithful.

Load-bearing premise

The entire cross-modal consistency story rests on the assumption that BEV features lifted from the generated camera images carry trustworthy geometry; if the video branch produces depth or layout hallucinations, the LiDAR generator will inherit them and the reported Chamfer numbers may reflect image-to-LiDAR mimicry rather than true spatial agreement.

Editorial extensions

If this is right

  • Generated paired video and LiDAR can serve as augmented training data for perception models, and the paper shows joint generation helps more than either modality alone (NDS +1.48 vs. +0.47 with video only and +0.93 with LiDAR only).
  • Editing the shared layout, such as shifting the ego trajectory by four meters, produces coherent camera and LiDAR output beyond the training set, enabling controlled scenario replay for testing planners.
  • Structured captions let users change global attributes such as time-of-day and weather while keeping the same underlying map and object layout, which is useful for robustness testing.
  • Long-horizon LiDAR prediction improves more than in prior baselines (Chamfer 0.633 at 3 s vs. 1.17 for the previous best), suggesting that dual-branch coupling stabilizes geometry over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the image-to-BEV lifting is the only geometric bridge between branches, the same coupling could be reused for other sensor pairs such as radar or thermal cameras by swapping the lifting operator, making Genesis a template for multimodal world models rather than a video-LiDAR-specific solution.
  • Editorial inference: the paper never measures cross-modal alignment directly, so a fair reader should treat the consistency claim as only indirectly supported until a geometric reprojection metric is computed on the generated pairs.
  • Editorial inference: the caption-quality gate built into DataCrafter may bias training toward simpler scenes; a testable extension is to break down generation quality by scene complexity and check whether complex urban intersections, where instance-level captions are harder, show disproportionate degradation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. Genesis proposes a two-branch framework for joint generation of multi-view driving videos and LiDAR sequences on nuScenes. The video branch is a DiT-based spatiotemporal diffusion model conditioned on projected BEV layouts, captions from a new DataCrafter module, and pedestrian pose keypoints; the LiDAR branch is a BEV-latent diffusion model that receives, in addition to layouts and captions, BEV features lifted from the (generated) camera images via LSS. The paper reports state-of-the-art FVD/FID for video and Chamfer distance for LiDAR, plus downstream gains for BEV segmentation and 3D detection. The central claim is that cross-modal consistency is achieved through the camera-to-LiDAR BEV conditioning, without explicit occupancy supervision.

Significance. If the central claim holds, Genesis would be a valuable contribution: a single model that synthesizes temporally coherent, semantically controllable, and mutually aligned multi-camera video and LiDAR sequences, with demonstrated utility for downstream perception. The paper contains extensive experiments, including qualitative controllability demonstrations, downstream task evaluations, and an ablation study, and it is generally clearly written. However, the pivotal evidence for the claimed cross-modal coupling mechanism is internally inconsistent: the LiDAR ablation in Table 4 contradicts the text in Section 4.6, and the missing comparison cell prevents a clean test of the image-BEV-to-LiDAR contribution. Since the cross-modal consistency claim is the paper's main novelty, the current manuscript is not yet ready for acceptance.

major comments (4)
  1. [Sec. 4.6 / Table 4] The text states that 'further excluding BEV latent features causes additional degradation,' but the only controlled comparison for this claim is between rows 1 and 2 of Table 4, where removing the image-BEV latent improves Chamfer@1s from 0.668 to 0.661 (and similarly at 2s/3s: 0.669 vs 0.672, 0.673 vs 0.677). This is the opposite of the stated conclusion, so the ablation does not support the claimed role of the BEV latent in preserving spatial structure.
  2. [Sec. 3.3 / Table 4] The experimental design lacks the critical cell 'w/o Img BEV Latent + w/ Ref. Frame.' Without this cell, the marginal contribution of the image-to-LiDAR bridge is never measured under the final system's operating condition (with first-frame conditioning). The available comparison without the reference frame indicates that adding the BEV latent slightly hurts performance, so the central claim that camera-derived BEV features improve cross-modal consistency is currently unsupported by controlled evidence.
  3. [Sec. 4.4 / Table 2] The claim that performance 'remains stable' when substituting ground-truth images with generated images is not supported by the reported numbers: Chamfer@1s degrades from 0.611 to 0.634, Chamfer@2s from 0.625 to 0.638, and Chamfer@3s from 0.633 to 0.641. The authors should either report confidence intervals or otherwise quantify these differences, or reinterpret them as evidence of a limitation in the image-to-LiDAR coupling.
  4. [Secs. 3.3 and 4.5] No direct quantitative metric for cross-modal alignment is reported; the consistency claim rests on qualitative figures. A concrete test, such as BEV occupancy IoU between the generated LiDAR and depth lifted from the generated images, or Chamfer distance between generated LiDAR and the corresponding image-derived BEV features, would make the central claim falsifiable and should be added.
minor comments (8)
  1. [Abstract] The phrase 'synthetical data' should be 'synthetic data.'
  2. [Sec. 3.3] The sentence 'the cross-attention operation cloud be formulated as' contains a typo: 'cloud' should be 'can.'
  3. [Eq. (4)] The notation z(t)_v and z(t-1)_v is not defined; clarify the time index and the recursion, since Equation (4) currently resembles a step in a Markov chain rather than a diffusion sampling step.
  4. [Table 1] The baseline rows lack sample-number and frame-number entries; include these so that the FVD/FID comparisons are apples-to-apples.
  5. [Sec. 3.1 / Eq. (1)] The weights lambda_1, lambda_2, lambda_3 are said to be fixed but are never specified; report their values or state explicitly that they were hand-tuned.
  6. [References] References [32] and [33] are the same paper; deduplicate them.
  7. [Abstract vs. Sec. 3.3] The abstract's claim that both modalities are 'directly coupled through a shared latent space' is not reflected in Section 3.3, where the coupling is through LSS-lifted BEV features concatenated with layout latents; align the terminology.
  8. [Sec. 4.7 / Table 6] Specify whether the generated camera and LiDAR data are paired (generated jointly from the same layout) and describe the data mixing ratio when fine-tuning BEVFusion; this is needed to interpret the +0.91 mAP / +1.48 NDS gains.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the SOTA claims are evaluated on held-out nuScenes validation and external baselines; the main caveat is an internal inconsistency in the LiDAR ablation (Table 4 vs Sec. 4.6), not a circular reduction.

full rationale

The paper's central claim—joint multi-view video and LiDAR generation with state-of-the-art FVD/FID/Chamfer numbers—rests on held-out nuScenes validation evaluations (Secs. 4.3, 4.4) and downstream BEV segmentation / 3D detection tasks (Sec. 4.7). No fitted parameter is renamed as a prediction: DataCrafter is a captioning and quality-filtering module (Sec. 3.1, Appendix A) whose threshold influences training-data selection but does not enter the reported metrics by construction. The video backbone is initialized from external pretrained weights (MagicDriveDiT, CogVideo-XL) and the LiDAR branch is trained from scratch (Appendix B), while Table 2 compares LiDAR generation against external HERMES, ViDAR, and 4D-Occ protocols. Some comparison baselines (DiVE [16], Cogen [15], MiLA [35]) come from the same Xiaomi EV group, but these are benchmark numbers rather than load-bearing derivations; the independent evidence from external baselines does not depend on them. One caveat should be flagged, though it is not circularity: Sec. 4.6 claims 'Further excluding BEV latent features causes additional degradation,' but Table 4 shows that with the reference frame removed, 'w/o Img BEV Latent + w/o Ref. Frame' gives Chamfer@1s 0.661 while 'w/ Img BEV Latent + w/o Ref. Frame' gives 0.668, i.e., the image-BEV condition slightly hurts in that pairwise comparison. The missing cell 'w/o Img BEV Latent + w/ Ref. Frame' also prevents a clean marginal test of the LSS-based BEV bridge in the final system, and no direct cross-modal alignment metric is reported. This is an internal-consistency / omitted-evidence issue, not a case where a prediction is equivalent by construction to its inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on standard diffusion-model assumptions plus domain assumptions about nuScenes annotations and LSS-based geometric bridging. The main ad hoc design assumption is that shared conditioning and image-derived BEV features are enough to align video and LiDAR without an explicit joint latent or occupancy representation. No invented physical entities are present.

free parameters (5)
  • DataCrafter quality threshold = not reported
    Training clips below a predefined quality threshold are discarded; the threshold is chosen by the authors and its value is not given (Appendix A(a)).
  • Quality score weights lambda_1, lambda_2, lambda_3 = not reported
    Fixed weights in Eq. (1) combining clarity, structure, and aesthetics scores; no values or sensitivity analysis are provided (Sec 3.1).
  • LiDAR occupancy post-processing threshold = not reported
    Points in grids with occupancy below a threshold are filtered after NeRF rendering; the threshold is chosen by hand and not specified (Sec 3.3).
  • Resolution and clip-length schedule = 144p/128 frames up to 900p/6 frames, then 360p/16 frames
    Multi-resolution pre-training schedule is a hand-designed curriculum that affects video quality (Appendix A(c)).
  • Training stage durations = 300/800/200 epochs
    Stage-wise epoch counts and learning rates are chosen without sensitivity analysis (Appendix A(c)).
assumptions (5)
  • standard math The learned diffusion denoiser accurately approximates the rectified-flow velocity field on nuScenes data.
    Standard assumption for all diffusion models; the paper does not verify convergence beyond reported metrics.
  • domain assumption LSS lifting from perspective images to BEV features preserves the geometric information needed for LiDAR conditioning.
    Sec 3.3 relies on LSS as the cross-modal bridge; no direct geometric alignment metric is reported.
  • domain assumption nuScenes camera intrinsics/extrinsics and annotations (boxes, lanes, keypoints) are accurate enough for layout projection and BEV lifting.
    Sec 4.1 and Fig 5 use calibrated intrinsics and extrinsics for projection; annotation errors would propagate to both generation branches.
  • domain assumption Interpolating semantic occupancy labels from 2 Hz to 12 Hz yields valid dense supervision.
    Sec 4.1 states occupancy labels are interpolated; interpolation can create inconsistent motion labels for fast-moving objects.
  • ad hoc to paper Shared captions and layout conditions are sufficient to enforce cross-modal consistency without explicit 3D occupancy or a joint latent variable.
    This is the paper's central design choice (abstract and Sec 3.3), and it is validated only indirectly via downstream tasks and qualitative examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency." pith.science (2026). https://pith.science/paper/5K2ADKD3

@misc{pith2026250607497,
  author       = {Pith},
  title        = {Pith review of: Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5K2ADKD3}},
  note         = {Machine review of arXiv:2506.07497}
}
read the original abstract

We present Genesis, a unified framework for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. Genesis employs a two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and a BEV-aware LiDAR generator with NeRF-based rendering and adaptive sampling. Both modalities are directly coupled through a shared latent space, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level supervision. Extensive experiments on the nuScenes benchmark demonstrate that Genesis achieves state-of-the-art performance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantic fidelity and practical utility of the generated data.

Figures

Figures reproduced from arXiv: 2506.07497 by the authors.

Figure 1
Figure 1. Comparison of Multimodal Scene Generation Pipelines. (a) Video-only generation, (b) LiDAR-only generation, (c) Occupancy-guided dual-branch generation, (d) Our Genesis. * Equal contribution, † Project leader, Corresponding author. Preprint. Under review. arXiv:2506.07497v4 [cs.CV] 20 Jun 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Genesis architecture for joint video and LiDAR generation. A dual￾branch design processes shared semantic conditions via camera and LiDAR pathways, using STDiT blocks for spatiotemporal generation and a BEV encoder for geometric alignment. LiDAR Point Cloud Generation. LiDAR point clouds form the perceptual cornerstone of au￾tonomous driving, enabling precise 3D understanding through sparse yet geome… view at source ↗
Figure 3
Figure 3. DataCrafter pipeline for structured multi-view captioning. Videos are segmented and filtered via a VLM-based quality checker (1–2), then per-view captions are generated and fused into coherent, structured descriptions (3–4). Steps 1–4 are used during training, while only 3–4 are used at inference. where Q terms represent VLM-derived subscores and λi are fixed weights. To ensure consistency across overlapping views, … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Left: STDiT-Block-C of Camera branch. Right: STDiT-Block-L of Lidar branch. 3.2 Video Generation Model To ensure coherent multi-view video generation, a DiT-based diffusion backbone is extended with 3D-aware latent encoding and conditioned on scene-level priors—includi…
Figure 5
Figure 5. Figure 5: Illustration of the scene layouts projection pipeline, which enables view-consistent and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of video generation. From top to bottom: (1) Ground-truth images, (2) Scene Layouts input, (3) Panacea [39], (4) MagicDrive [10], (5) Ours. Panacea suffers from hallucinated textures and geometric misalignment. MagicDrive shows vehicle distortion…
Figure 7
Figure 7. Figure 7: Joint generation of LiDAR and multi-view video. Our method generates spatially aligned LiDAR and camera views conditioned on a shared right turn T-junction layout [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparison with and without DataCrafter. Our method yields semantically rich, structurally consistent scenes, while the baseline lacks fine-grained details and geometric accuracy. Genesis consistently outperforms existing approaches across both short- and long-term hor…
Figure 9
Figure 9. Figure 9: Controllable generation across time-of-day. By altering scene-level conditions, our method produces consistent multi-view videos aligned with the same underlying map and object layout, while adapting appearance to represent daytime and nighttime settings [PITH_FULL_IM…
Figure 10
Figure 10. Figure 10: Qualitative comparison of video generation quality. Our method (second column) preserves accurate layout, object shapes, and background integrity. MagicDrive (third column) shows vehicle distortion and broken structures. Panacea (fourth column) often suffers from hall…
Figure 11
Figure 11. Figure 11: Trajectory-conditioned novel view synthesis. Given a ground-truth trajectory (middle), we modify the layout (top) by shifting the ego path 4 meters right (bottom). Our model generates plausible and consistent scenes across all views under these layout changes [PITH_F…
Figure 12
Figure 12. Figure 12: Joint generation of LiDAR and multi-view video. Our method generates spatially aligned LiDAR and camera views conditioned on a shared straight street layout [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Joint generation of LiDAR and multi-view video. Our method generates spatially aligned LiDAR and camera views conditioned on a Busy junction layout. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Long-term multi-view video generation over 3 seconds in an urban driving scene, condi [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Long-term multi-view video generation over 3 seconds in an urban driving scene, condi [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A video-generation world model that warps positional encodings of memory frames to target viewpoints achieves state-of-the-art long-term consistency and camera control.

  2. OmniNWM: Omniscient Driving Navigation World Models

    cs.CV 2025-10 conditional novelty 6.0 of 10

    OmniNWM jointly generates long panoramic multi-modal driving videos, controls them precisely via normalized Plücker ray-maps, and derives dense driving rewards from generated 3D occupancy.

  3. UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A single mask-modulated DiT that co-trains future video and trajectories yields stronger autonomous-driving action generalization and 4.3× faster trajectory-only inference than dual-DiT designs.

  4. A Survey on the Applications of Generative Artificial Intelligence in Automated Driving Systems Test Scenario Generation Methods

    cs.SE 2025-12 reject novelty 4.0 of 10

    A literature survey of scenario-generation methods for ADS testing that adds an unvalidated AII/RAS/OCS metric suite and ODD-difficulty schema, undermined by inconsistent calculations in the worked examples.

Reference graph

Works this paper leans on

50 extracted references · 14 canonical work pages · cited by 4 Pith papers

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  2. [2]

    Transfusion: Robust lidar-camera fusion for 3d object detection with transformers

    Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1090–1099, 2022

    work page 2022

  3. [3]

    Camera-lidar inte- gration: Probabilistic sensor fusion for semantic mapping.IEEE Transactions on Intelligent Transportation Systems, 23(7):7637–7652, 2021

    Julie Stephany Berrio, Mao Shan, Stewart Worrall, and Eduardo Nebot. Camera-lidar inte- gration: Probabilistic sensor fusion for semantic mapping.IEEE Transactions on Intelligent Transportation Systems, 23(7):7637–7652, 2021

    work page 2021

  4. [4]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020

    2020

  5. [5]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    2024

  6. [6]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

  7. [7]

    Trafficgen: Learning to generate diverse and realistic traffic scenarios

    Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In2023 IEEE international conference on robotics and automation (ICRA), pages 3567–3575. IEEE, 2023

    work page 2023

  8. [8]

    Magic- drive3d: Controllable 3d generation for any-view rendering in street scenes.arXiv preprint arXiv:2405.14475, 2024

    Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magic- drive3d: Controllable 3d generation for any-view rendering in street scenes.arXiv preprint arXiv:2405.14475, 2024

    arXiv 2024

Show all 50 references
  1. [9]

    Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024

    2024 arXiv

  2. [10]

    Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    2023 arXiv

  3. [11]

    Vista: A generalizable driving world model with high fidelity and versatile controllability.arXiv preprint arXiv:2405.17398, 2024

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability.arXiv preprint arXiv:2405.17398, 2024

    2024 arXiv

  4. [12]

    Vision meets robotics: The kitti dataset.The international journal of robotics research, 32(11):1231–1237, 2013

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset.The international journal of robotics research, 32(11):1231–1237, 2013

    2013

  5. [13]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  6. [14]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  7. [15]

    Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving.arXiv preprint arXiv:2503.22231, 2025

    Yishen Ji, Ziyue Zhu, Zhenxin Zhu, Kaixin Xiong, Ming Lu, Zhiqi Li, Lijun Zhou, Haiyang Sun, Bing Wang, and Tong Lu. Cogen: 3d consistent video generation via adaptive conditioning for autonomous driving.arXiv preprint arXiv:2503.22231, 2025. 10

    2025 arXiv

  8. [16]

    Dive: Dit-based video generation with enhanced control

    Junpeng Jiang, Gangyi Hong, Lijun Zhou, Enhui Ma, Hengtong Hu, Xia Zhou, Jie Xiang, Fan Liu, Kaicheng Yu, Haiyang Sun, et al. Dive: Dit-based video generation with enhanced control. arXiv preprint arXiv:2409.01595, 2024

    2024 arXiv

  9. [17]

    Point cloud forecasting as a proxy for 4d occupancy forecasting

    Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1116–1124, 2023

    2023

  10. [18]

    Uniscene: Unified occupancy-centric driving scene generation.arXiv preprint arXiv:2412.05435, 2024

    Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation.arXiv preprint arXiv:2412.05435, 2024

    2024 arXiv

  11. [19]

    Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection

    Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. InProceedings of the IEEE/CVF conference on computer vision and patter...

    2022

  12. [20]

    Bevfusion: A simple and robust lidar-camera fusion framework

    Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:10421–10434, 2022

    2022

  13. [21]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  15. [23]

    Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation

    Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Li Zhang. Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. InEuropean Conference on Computer Vision, pages 329–345. Springer, 2024

    2024

  16. [24]

    Unleashing generalization of end-to-end autonomous driving with controllable long video generation.arXiv preprint arXiv:2406.01349, 2024

    Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, et al. Unleashing generalization of end-to-end autonomous driving with controllable long video generation.arXiv preprint arXiv:2406.01349, 2024

    2024 arXiv

  17. [25]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021

    2021

  18. [26]

    Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d

    Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 194–210. Springer, 2020

    2020

  19. [27]

    Scenario diffusion: Controllable driving scenario generation with diffusion.Advances in Neural Information Processing Systems, 36:68873–68894, 2023

    Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Kai Wang, and Nick Roy. Scenario diffusion: Controllable driving scenario generation with diffusion.Advances in Neural Information Processing Systems, 36:68873–68894, 2023

    2023

  20. [28]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017

    2017

  21. [29]

    Towards realistic scene generation with lidar diffusion models

    Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14738–14748, 2024

    2024

  22. [30]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InProceedings of the IEEE/CVF conference on computer vis...

    2020

  23. [31]

    Drivescenegen: Generating diverse and realistic driving scenarios from scratch

    Shuo Sun, Zekai Gu, Tianchen Sun, Jiawei Sun, Chengran Yuan, Yuhang Han, Dongen Li, and Marcelo H Ang. Drivescenegen: Generating diverse and realistic driving scenarios from scratch. IEEE Robotics and Automation Letters, 2024. 11

    2024

  24. [33]

    Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024

    Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird’s-eye view layout.IEEE Robotics and Automation Letters, 2024

    2024

  25. [34]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  26. [35]

    Mila: Multi-view intensive-fidelity long-term video generation world model for autonomous driving.arXiv preprint arXiv:2503.15875, 2025

    Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, and Bing Wang. Mila: Multi-view intensive-fidelity long-term video generation world model for autonomous driving.arXiv preprint arXiv:2503.15875, 2025

    2025 arXiv

  27. [36]

    Drive- dreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-drive world models for autonomous driving. InEuropean Confer- ence on Computer Vision, pages 55–72. Springer, 2024

    2024

  28. [37]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1...

    2024

  29. [38]

    Lidenerf: Neural radiance field reconstruction with depth prior provided by lidar point cloud.ISPRS Journal of Photogrammetry and Remote Sensing, 208:296–307, 2024

    Pengcheng Wei, Li Yan, Hong Xie, Dashi Qiu, Changcheng Qiu, Hao Wu, Yinghao Zhao, Xiao Hu, and Ming Huang. Lidenerf: Neural radiance field reconstruction with depth prior provided by lidar point cloud.ISPRS Journal of Photogrammetry and Remote Sensing, 208:296–307, 2024

    2024

  30. [39]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  31. [40]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. InProceedings of the IEEE/CVF International Conference ...

    2023

  32. [41]

    Driver lane change intention recognition based on attention enhanced residual-mbi-lstm network.IEEE Access, 10:58050– 58061, 2022

    Zhanqian Wu, Kaichong Liang, Dengcheng Liu, and Zhiguo Zhao. Driver lane change intention recognition based on attention enhanced residual-mbi-lstm network.IEEE Access, 10:58050– 58061, 2022

    2022

  33. [42]

    Point-nerf: Point-based neural radiance fields

    Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438–5448, 2022

    2022

  34. [43]

    Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.arXiv preprint arXiv:2308.01661, 2023

    Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.arXiv preprint arXiv:2308.01661, 2023

    2023 arXiv

  35. [44]

    Visual point cloud forecasting enables scalable autonomous driving

    Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673–14684, 2024

    2024

  36. [45]

    Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion.arXiv preprint arXiv:2311.01017, 2023

    Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion.arXiv preprint arXiv:2311.01017, 2023

    2023 arXiv

  37. [46]

    Bevworld: A multimodal world model for autonomous driving via unified bev latent space.arXiv preprint arXiv:2407.05679, 2024

    Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for autonomous driving via unified bev latent space.arXiv preprint arXiv:2407.05679, 2024

    2024 arXiv

  38. [47]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10412–10420, 2025. 12

    2025

  39. [48]

    Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation.arXiv preprint arXiv:2501.14729, 2025

    Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation.arXiv preprint arXiv:2501.14729, 2025

    2025 arXiv

  40. [49]

    V oxelnet: End-to-end learning for point cloud based 3d object detection

    Yin Zhou and Oncel Tuzel. V oxelnet: End-to-end learning for point cloud based 3d object detection. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018

    2018

  41. [50]

    Lidardm: Generative lidar simulation in a generated world.arXiv preprint arXiv:2404.02903, 2024

    Vlas Zyrianov, Henry Che, Zhijian Liu, and Shenlong Wang. Lidardm: Generative lidar simulation in a generated world.arXiv preprint arXiv:2404.02903, 2024

    2024

  42. [51]

    Daytime",

    Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. InEuropean Conference on Computer Vision, pages 17–35. Springer, 2022. 13 A Technical Appendices and Supplementary Material (a) DataCrafter Setup To enable structured semantic super...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.