Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

The paper claims that scaling occupancy-centric generation to a 3.6-million-frame dataset lets one model jointly produce semantic occupancy, multi-view video, and LiDAR point clouds with the best reported fidelity on all three tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:01 UTC pith:TSWNDTJM

load-bearing objection Serious engineering with a useful new dataset, but label validation and an uneven video comparison temper the SOTA claims. the 3 major comments →

arxiv 2510.22973 v2 pith:TSWNDTJM submitted 2025-10-27 cs.CV

Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method

Bohan Li , Xin Jin , Hu Zhu , Hongsi Liu , Ruikai Li , Jiazhe Guo , Kaiwen Cai , Chao Ma
show 4 more authors
Yueming Jin Hao Zhao Xiaokang Yang Wenjun Zeng
This is my paper
classification cs.CV
keywords driving scene generationsemantic occupancy4D dynamic scene modelingmulti-view video generationLiDAR simulationdiffusion transformerGaussian splatting conditioningannotated driving datasets
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that the reason occupancy-centric driving scene generation underperforms is data scarcity, not architecture. To test that, the authors build Nuplan-Occ, a 3.6-million-frame semantic occupancy dataset, and train UniScenev2, a unified model that first generates 4D semantic occupancy, then uses it as a pivot to render multi-view video and LiDAR point clouds. They report that at this scale the model beats prior work on all three modalities: occupancy generation mIoU 32.22, video FVD 63.29, and LiDAR MMD 0.457. They also show the synthetic data transfers: downstream occupancy prediction and planning improve when trained or tested on the generated scenes. If correct, this makes large-scale, multi-sensor synthetic driving data practical for closed-loop evaluation.

Core claim

On the paper's own terms, the central discovery is that semantic occupancy can serve as a scalable intermediate representation when the dataset is large enough. From BEV layouts or pure noise, a spatio-temporally disentangled diffusion transformer generates a 4D occupancy volume; those volume grids are converted to Gaussian primitives and rendered into sparse semantic and depth maps that condition a video DiT; and the same occupancy guides a sensor-aware LiDAR decoder via occupancy priors, Plucker and histogram ray embeddings, and a sensor-specific embedding. With Nuplan-Occ's scale, the framework reports the best generation fidelity against existing methods on occupancy, video, and LiDAR, a

What carries the argument

The load-bearing mechanism is the occupancy-centric hierarchy combined with a spatio-temporal disentangled DiT: spatial expansion (ego-moving scenes) and temporal forecasting (ego-stationary scenes) are trained separately but share one architecture. Occupancy then becomes a common conditioning signal for video, via Gaussian splatting with unscented-transform calibration to render sparse point maps, and for LiDAR, via occupancy-guided sparse sampling with sensor embeddings, Plucker and histogram ray features, and ray smoothness regularization.

Load-bearing premise

The whole edifice rests on Nuplan-Occ's auto-generated labels being true occupancy: foreground voxels are filled from bounding boxes and background voxels from BEV-map projections, with no point-level segmentation or human audit to check against, so if those labels are systematically wrong or over-inflated, the reported scores measure fidelity to the labels, not to the real world.

What would settle it

Manually annotate a random sample of Nuplan-Occ frames with point-level semantic occupancy, then recompute the reported occupancy-generation mIoU and downstream perception/planning numbers on that human-verified ground truth; or ray-cast raw LiDAR to count voxels that are labeled as occupied inside bounding boxes but actually contain free space. If the numbers collapse under such an audit, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Occupancy generation mIoU rises to 32.22 from 17.52 (OccWorld) and 22.64 (UniScene), and the ablations attribute the gain to the 4D VAE's axial attention and the spatial/temporal DiT blocks.
  • Video generation FVD improves to 63.29 from 108.50 (Vista*) with ground-truth occupancy; the sparse Gaussian point maps with unscented-transform calibration are the decisive components.
  • LiDAR MMD drops to 0.457 from 0.999 (UniScene), with sensor-specific embedding, Plucker embedding, histogram embedding, and smoothness regularization each contributing large margins.
  • The synthetic data transfers: MonoScene/LMSCNet occupancy prediction reaches IoU 21.6 vs 19.5 for UniScene-C, and UniAD planning reaches NC 95.7 vs 97.8 for ground-truth scenes.
  • The data filtering strategy (ego-moving vs ego-stationary) lets one architecture perform both spatial expansion and temporal forecasting, enabling theoretically infinite scene expansion by iterative outpainting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the auto-generated Nuplan-Occ labels are roughly faithful, this removes the annotation bottleneck that limited occupancy-centric generation, making closed-loop AD evaluation on synthetic scenes feasible at scale.
  • The occupancy-centric hierarchy suggests occupancy as a shared currency between perception and planning; a natural extension is to test whether controllers trained entirely on UniScenev2 synthetic scenes transfer to real-world driving.
  • The sensor-specific LiDAR embedding could plausibly transfer to novel sensor rigs without retraining, a testable extension beyond the paper's fixed five-LiDAR setup.
  • The box-fill labeling in Eq. 6 means foreground occupancy is an over-approximation; a useful follow-up would be to refine the dataset with free-space evidence from raw LiDAR rays.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Nuplan-Occ, claimed to be the largest semantic-occupancy driving dataset to date, constructed from the Nuplan benchmark via a Foreground-Background Separate Aggregation (FBSA) pipeline that combines point-cloud aggregation, NKSR mesh reconstruction, and hybrid bounding-box/BEV-map semantic labeling. On top of this dataset, the authors propose UniScenev2, a unified framework that jointly generates 4D semantic occupancy, multi-view video, and LiDAR point clouds, with a spatio-temporally disentangled occupancy diffusion architecture, Gaussian-splatting sparse point-map conditioning for video, and sensor-specific embeddings for LiDAR. The paper reports state-of-the-art results across occupancy generation (mIoU 32.22 vs 17.52/22.64), video generation (FVD 63.29 with ground-truth occupancy), LiDAR generation (MMD 0.457 vs 0.999), and downstream occupancy prediction and planning, along with ablations supporting each proposed component.

Significance. If the dataset and evaluation are sound, this is a substantial contribution: it scales occupancy-centric driving generation by an order of magnitude in scenes and frames, proposes a unified multi-modal generation pipeline with several reusable architectural ideas, and releases code and data that could benefit the community. The internal ablations (Tables VIII–X) do show meaningful functional roles for the proposed components, and the headline margins are arithmetically consistent with the tables. The main risk is that the entire evaluation chain—occupancy generation, video FVD, LiDAR MMD, and downstream perception/planning—uses Nuplan-Occ auto-labels as ground truth without an external audit, and that the video/downstream comparisons give UniScenev2 oracle occupancy conditioning that baselines do not receive. These issues are fixable but load-bearing for the paper's central claims.

major comments (3)
  1. [Section III-C, Eqs. (6)–(7) and Fig. 3] The Nuplan-Occ labels are generated by an automated pipeline with no reported validation. Eq. (6) assigns any point inside an object's 3D bounding box the object class, which may systematically inflate occupied volume beyond observed surfaces; Eq. (7) propagates BEV-map labels to vertical columns, which can mislabel structures such as bridges over roads. NKSR mesh reconstruction can also hallucinate surfaces on sparse or noisy point clouds. Nuplan does not provide point-level segmentation, and the paper reports no human study, no inter-annotator agreement, and no cross-check against point-level annotations from Occ3D/SurroundOcc. Since every headline metric in Tables III–VII uses these auto-labels as the reference, all reported mIoU/FVD/MMD values may reflect fidelity to the auto-label distribution rather than to real geometry. This must be addressed before the dataset contribution and t
  2. [Section IV-B, Table IV] The video generation comparison is asymmetric. UniScenev2 is evaluated 'with ground truth occupancy'—the model receives dense semantic and depth sparse maps rendered from the exact target scene—while the baselines (BEVGen, DriveDreamer, MagicDrive, Vista) do not receive this oracle conditioning. The 63.29 FVD vs 108.50 for Vista* may therefore substantially reflect the extra conditioning signal rather than the proposed video architecture or rendering strategy. The same issue affects the downstream planning comparison in Table VII, which also uses 'ground truth occupancy conditions.' Please either provide baseline variants conditioned on the same GT occupancy-derived maps, evaluate UniScenev2 without GT occupancy (e.g., with predicted occupancy from its own generation model), or clearly separate the oracle-conditioned performance from the truly generative setting.
  3. [Section V-A, Tables VI–VII] The 'Generation Fidelity Evaluation' inherits the same oracle-conditioning concern. Table VI shows UniScenev2-C and UniScenev2-L improving downstream occupancy prediction, but these synthetic inputs are generated with GT occupancy as condition, while competing synthetic data (MagicDrive, Vista*, UniScene-C) are not. The claimed practical value for downstream tasks therefore conflates the quality of the generation model with the availability of ground-truth scene structure during generation. Please report a configuration where the occupancy condition is predicted or ablated, and clarify whether the downstream gains persist when the full UniScenev2 pipeline (occupancy generation from BEV/noise, then video/LiDAR from that occupancy) is used.
minor comments (4)
  1. [Section V-A, Table VIII] The text states that removing the VAE 3D axial attention reduces mIoU by 33.52%, but the table values (21.42 vs 32.22) correspond to a 50.4% relative improvement. The F3D 'improvement' percentages also appear to describe relative reductions (140.23→48.24 is 65.6%), so the wording should be made consistent with the arithmetic.
  2. [Section III-C, Eq. (7)] Eq. (7) uses BEVLabel(Proj(v)) but the projection and label lookup are not specified in detail. Please define Proj(v) explicitly (e.g., vertical ray casting, which types of BEV map layers are used, and how occlusions are handled) and fix the typo 'correponding.'
  3. [Section IV-A2, Eqs. (10)–(11)] The thresholds θe and θo are introduced but no values or sensitivity analysis are provided. Since the spatio-temporal disentanglement is a central contribution, please report the chosen values and ideally a small sensitivity study.
  4. [Tables VI–VII and Section V-A] Typographical issues: 'MigicDrive' appears in both tables, and 'Vista∗' vs 'Vista' is used inconsistently. In Table I, the name 'Nuscenes-Occupancy' is used alongside Occ3D/SurroundOcc; please standardize.

Circularity Check

2 steps flagged

Video and downstream fidelity evaluations condition on ground-truth occupancy of the same target scenes, so part of the reported FVD/IoU gains is borrowed from the conditioning signal; the core method itself is not otherwise circular.

specific steps
  1. fitted input called prediction [Section IV-B (Eqs. 12-13, Eq. 17) and Section V-A Table IV / Table IX]
    "achieving 8.32 FID and 63.29 FVD with ground truth occupancy, respectively. ... w/o. Sparse Rendered Semantic Map - 12.27 110.79; w/o. Sparse Rendered Depth Map - 12.05 108.21"

    The video DiT is conditioned on depth and semantic sparse maps rendered from the target scene's ground-truth occupancy (Eqs. 12-13, used in Eq. 17). The Table IX ablation shows that removing these maps raises FVD from 63.29 to roughly 108-111, i.e., the reported SOTA video score is largely supplied by the oracle occupancy of the exact scene being evaluated. Baselines such as Vista do not receive this target-derived conditioning, so the FVD/FID comparison partly measures the injected conditioning signal rather than independent generative fidelity.

  2. fitted input called prediction [Section V-A, Table VI (Generation Fidelity Evaluation)]
    "We evaluate our model’s ability to generate realistic driving scenarios using ground truth occupancy conditions."

    The generated camera and LiDAR data in Table VI are conditioned on the same ground-truth occupancy that is later used as the label for MonoScene/LMSCNet occupancy prediction. Since the video conditioning includes a semantic map S = argmax of rendered one-hot occupancy (Eq. 13), the downstream 'prediction' can recover the conditioning labels from the generated data. The reported IoU/mIoU gains over baselines that lack this conditioning therefore partly reflect ground-truth occupancy being copied through the generator into the evaluation, not newly synthesized scene content.

full rationale

The core technical derivation is largely self-contained: the occupancy VAE/DiT, video DiT, and LiDAR UNet are trained with standard losses (Eqs. 8, 9, 17, 27), and ablations isolate component contributions. There is no load-bearing self-citation or imported uniqueness theorem; UniScene [6] is an extension baseline rather than the proof of the method. The Nuplan-Occ label pipeline (FBSA, NKSR, Eq. 6-7) raises data-quality concerns, but it defines the benchmark rather than making the derivation circular. The main circularity is evaluative: video-fidelity and downstream-perception results are obtained under ground-truth occupancy conditioning on the same scenes used as evaluation targets, and the ablations show that this conditioning accounts for most of the gain. This reduces the strength of the headline SOTA comparisons but does not invalidate the internal consistency of the proposed architecture. Accordingly, the paper deserves a moderate circularity score of 4 rather than a higher one.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claims rest on domain assumptions about the correctness of auto-annotated occupancy labels and learned reconstruction, plus several undisclosed free parameters (speed thresholds, loss weights, filter scale, compression ratio). No unexplained physical or architectural entities are introduced: gaussian primitives, occupancy grids, and embeddings are representational constructs with defined roles.

free parameters (7)
  • θe (ego speed threshold, spatial/temporal data split) = not stated
    Eq. (10): scenes with vego(x) > θe form the spatial training set. The value determines the composition of both generators and is never disclosed or ablated.
  • θo (surrounding-vehicle speed threshold) = not stated
    Eq. (11): scenes with vego(x) < θe and vother(x) > θo form the temporal training set. Undisclosed; affects forecasting quality.
  • k (statistical outlier filter threshold) = not stated
    Eq. (1): k·σ threshold for background point filtering during dataset curation; affects geometry density and floaters in occupancy labels.
  • Gaussian primitive scale (default 0.01) = 0.01
    Table IX: 0.01 chosen via ablation on the mini validation set; the headline video numbers use the tuned value.
  • Loss weights λ1, λ2 (Eq. 8) and λ1, λ2, λ3 (Eq. 27) = not stated
    VAE and LiDAR objective balancing coefficients are not reported.
  • UT hyperparameters α=1.0, β=2.0, κ=0.0 = α=1.0, β=2.0, κ=0.0
    Eq. (14): taken from 3DGUT [84]; standard for that method but still a modeling choice affecting projected conic quality.
  • Default compression ratio 512; Tc=1; Tf=6 = 512; Tc=1; Tf=6
    VAE compression ratio and forecasting window; Tf=6 follows OccWorld [52], Tc=1 is a stated efficiency choice.
axioms (5)
  • domain assumption Nuplan object bounding boxes and BEV maps are accurate enough to serve as semantic occupancy labels.
    Section III-C, Eqs. (6)-(7): foreground labels are box-membership and background labels are BEV-map projections; no point-level ground truth exists in Nuplan, and errors in boxes/maps propagate directly into the dataset and every downstream evaluation.
  • domain assumption NKSR mesh reconstruction produces geometrically correct surfaces from aggregated, noisy LiDAR.
    Section III-B: NKSR [39] is a learned reconstruction; on sparse or misaligned point clouds it can hallucinate plausible geometry. The paper provides no check that reconstructed vertices match real surfaces.
  • domain assumption The 10-class Nuplan semantic set and 0.25 m voxel resolution capture driving-scene semantics adequately for generation and evaluation.
    Table I/Fig. 1: the label space and grid resolution define what 'semantic occupancy' means here; finer-grained classes or geometry would change the reported quality.
  • standard math Standard VAE and diffusion objectives (cross-entropy + Lovász-softmax + KL; MSE on predicted noise) are sufficient training targets for these generators.
    Eqs. (8)-(9) follow OccWorld [52] and DiT [77]; standard practice in the literature.
  • standard math The diffusion latent space factorizes spatial and temporal structure as assumed by the axial-attention VAE and spatial/temporal DiT blocks.
    Section IV-A: stacked spatial/temporal transformer blocks and 3D axial attention are the core architectural assumption; the very large ablation drops (Table VIII) suggest the model is highly sensitive to this structure.

pith-pipeline@v1.3.0-alltime-deepseek · 22972 in / 23416 out tokens · 244975 ms · 2026-08-04T08:01:37.588378+00:00 · methodology

0 comments
read the original abstract

Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. Repo: https://github.com/Arlo0o/UniScene-Unified-Occupancy-centric-Driving-Scene-Generation/tree/v2

Figures

Figures reproduced from arXiv: 2510.22973 by Bohan Li, Chao Ma, Hao Zhao, Hongsi Liu, Hu Zhu, Jiazhe Guo, Kaiwen Cai, Ruikai Li, Wenjun Zeng, Xiaokang Yang, Xin Jin, Yueming Jin.

Figure 1
Figure 1. Figure 1: Overview of Nuplan-Occ dataset and the UniScenev2 pipeline. We introduce the largest semantic occupancy dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Nuplan-Occ provides dense semantic occupancy labels for 10HZ all frames in the Nuplan [38] dataset. Compared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Nuplan-Occ dataset curation pipeline with the proposed Foreground-Background Separate Aggregation (FBSA) strategy. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall framework of UniScenev2. The joint generation process facilitates large-scale dynamic generation with an [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) Architecture of the occupancy generation model, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The architecture of the video generation model, which [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Gaussian-based sparse point map rendering with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The architecture of the LiDAR generation model, which [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Sensor-specific Embedding for decoupled LiDAR [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Visualization of the LiDAR ray smoothness regular [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of scene expansion and forecasting results. UniScenev2 enables spatio-temporally disentangled generation, [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative evaluation for occupancy generation. Our [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative evaluation for video generation. Our method [PITH_FULL_IMAGE:figures/full_fig_p013_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative evaluation for LiDAR generation. Our [PITH_FULL_IMAGE:figures/full_fig_p013_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FPSGen: Flexible Point Cloud Scene Generation with BEV-Supported Transport Flows

    cs.CV 2026-07 conditional novelty 6.5

    A two-stage flow-matching model that seeds point-cloud generation from a generated BEV density map, using teacher-estimated point pairings to keep transport paths straight, achieves SOTA JSD/IoU on SemanticKITTI compl...

Reference graph

Works this paper leans on

94 extracted references · 29 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Diffusion probabilistic models for 3d point cloud generation,

    S. Luo and W. Hu, “Diffusion probabilistic models for 3d point cloud generation,” inCVPR, 2021, pp. 2837–2845

    2021

  2. [2]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inCVPR, 2022, pp. 10 684–10 695

    2022

  3. [3]

    Conditional diffusion process for inverse halftoning,

    H. Jiang and Y . Mu, “Conditional diffusion process for inverse halftoning,” NeurIPS, vol. 35, pp. 5498–5509, 2022

    2022

  4. [4]

    One at a time: Progressive multi-step volumetric probability learning for reliable 3d scene perception,

    B. Li, Y . Sun, J. Dong, Z. Zhu, J. Liu, X. Jin, and W. Zeng, “One at a time: Progressive multi-step volumetric probability learning for reliable 3d scene perception,” inAAAI, 2024

    2024

  5. [5]

    Dreamdrive: Generative 4d scene modeling from street view images,

    J. Mao, B. Li, B. Ivanovic, Y . Chen, Y . Wang, Y . You, C. Xiao, D. Xu, M. Pavone, and Y . Wang, “Dreamdrive: Generative 4d scene modeling from street view images,”arXiv preprint arXiv:2501.00601, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Uniscene: Unified occupancy-centric driving scene generation,

    B. Li, J. Guo, H. Liu, Y . Zou, Y . Ding, X. Chen, H. Zhu, F. Tan, C. Zhang, T. Wanget al., “Uniscene: Unified occupancy-centric driving scene generation,”arXiv preprint arXiv:2412.05435, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Stag-1: Towards realistic 4d driving simulation with video generation model,

    L. Wang, W. Zheng, D. Du, Y . Zhang, Y . Ren, H. Jiang, Z. Cui, H. Yu, J. Zhou, J. Luet al., “Stag-1: Towards realistic 4d driving simulation with video generation model,”arXiv preprint arXiv:2412.05280, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    History-guided video diffusion,

    K. Song, B. Chen, M. Simchowitz, Y . Du, R. Tedrake, and V . Sitzmann, “History-guided video diffusion,”arXiv preprint arXiv:2502.06764, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Bevcontrol: Accu- rately controlling street-view elements with multi-perspective consistency via bev sketch layout,

    K. Yang, E. Ma, J. Peng, Q. Guo, D. Lin, and K. Yu, “Bevcontrol: Accu- rately controlling street-view elements with multi-perspective consistency via bev sketch layout,”arXiv preprint arXiv:2308.01661, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Street-view image generation from a bird’s-eye view layout,

    A. Swerdlow, R. Xu, and B. Zhou, “Street-view image generation from a bird’s-eye view layout,”IEEE Robotics and Automation Letters, 2024

    2024

  11. [11]

    Drivedreamer: Towards real-world-driven world models for autonomous driving,

    X. Wang, Z. Zhu, G. Huang, X. Chen, and J. Lu, “Drivedreamer: Towards real-world-driven world models for autonomous driving,”ECCV, 2024

    2024

  12. [12]

    Magicdrive: Street view generation with diverse 3d geometry control,

    R. Gao, K. Chen, E. Xie, L. Hong, Z. Li, D.-Y . Yeung, and Q. Xu, “Magicdrive: Street view generation with diverse 3d geometry control,” inICLR, 2024

    2024

  13. [13]

    Panacea: Panoramic and controllable video generation for autonomous driving,

    Y . Wen, Y . Zhao, Y . Liu, F. Jia, Y . Wang, C. Luo, C. Zhang, T. Wang, X. Sun, and X. Zhang, “Panacea: Panoramic and controllable video generation for autonomous driving,” 2023

    2023

  14. [14]

    Lidardm: Generative lidar simulation in a generated world,

    V . Zyrianov, H. Che, Z. Liu, and S. Wang, “Lidardm: Generative lidar simulation in a generated world,”arXiv preprint arXiv:2404.02903, 2024

    arXiv 2024

  15. [15]

    Open- vocabulary object segmentation with diffusion models,

    Z. Li, Q. Zhou, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Open- vocabulary object segmentation with diffusion models,” inICCV, 2023

    2023

  16. [16]

    Datasetdm: Synthesizing data with perception annotations using diffusion models,

    W. Wu, Y . Zhao, H. Chen, Y . Gu, R. Zhao, Y . He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” inNeurIPS, 2023

    2023

  17. [17]

    Fairdiff: Fair segmentation with point-image diffusion,

    W. Li, H. Xu, G. Zhang, H.-a. Gao, M. Gao, M. Wang, and H. Zhao, “Fairdiff: Fair segmentation with point-image diffusion,” inInternational Conference on Medical Image Computing and Computer-Assisted Inter- vention. Springer

  18. [18]

    Gan augmentation: Augmenting training data using generative adversarial networks,

    C. Bowles, L. Chen, R. Guerrero, P. Bentley, R. Gunn, A. Hammers, D. A. Dickie, M. V . Hernández, J. Wardlaw, and D. Rueckert, “Gan augmentation: Augmenting training data using generative adversarial networks,”arXiv preprint arXiv:1810.10863, 2018

    Pith/arXiv arXiv 2018

  19. [19]

    Integrating geometric control into text-to-image diffusion models for high-quality de- tection data generation via text prompt,

    K. Chen, E. Xie, Z. Chen, L. Hong, Z. Li, and D.-Y . Yeung, “Integrating geometric control into text-to-image diffusion models for high-quality de- tection data generation via text prompt,”arXiv preprint arXiv:2306.04607, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Detdiffusion: Synergizing generative and perceptive models for enhanced data generation and perception,

    Y . Wang, R. Gao, K. Chen, K. Zhou, Y . Cai, L. Hong, Z. Li, L. Jiang, D.-Y . Yeung, Q. Xuet al., “Detdiffusion: Synergizing generative and perceptive models for enhanced data generation and perception,”CVPR, 2024

    2024

  21. [21]

    Is synthetic data from generative models ready for image recognition?

    R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi, “Is synthetic data from generative models ready for image recognition?” arXiv preprint arXiv:2210.07574, 2022

    Pith/arXiv arXiv 2022

  22. [22]

    Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks,

    A. G. Møller, J. A. Dalsgaard, A. Pera, and L. M. Aiello, “Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks,”arXiv preprint arXiv:2304.13861, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Drivedreamer4d: World models are effective data machines for 4d driving scene representation,

    G. Zhao, C. Ni, X. Wang, Z. Zhu, X. Zhang, Y . Wang, G. Huang, X. Chen, B. Wang, Y . Zhanget al., “Drivedreamer4d: World models are effective data machines for 4d driving scene representation,”arXiv preprint arXiv:2410.13571, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Panacea: Panoramic and controllable video generation for autonomous driving,

    Y . Wen, Y . Zhao, Y . Liu, F. Jia, Y . Wang, C. Luo, C. Zhang, T. Wang, X. Sun, and X. Zhang, “Panacea: Panoramic and controllable video generation for autonomous driving,” inCVPR, 2024

    2024

  25. [25]

    Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception,

    X. Wang, Z. Zhu, W. Xu, Y . Zhang, Y . Wei, X. Chi, Y . Ye, D. Du, J. Lu, and X. Wang, “Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception,”ICCV, 2023

    2023

  26. [26]

    Bevfusion: A simple and robust lidar-camera fusion framework,

    T. Liang, H. Xie, K. Yu, Z. Xia, Z. Lin, Y . Wang, T. Tang, B. Wang, and Z. Tang, “Bevfusion: A simple and robust lidar-camera fusion framework,” NeurIPS, 2022

    2022

  27. [27]

    Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection,

    Y . Li, A. W. Yu, T. Meng, B. Caine, J. Ngiam, D. Peng, J. Shen, Y . Lu, D. Zhou, Q. V . Leet al., “Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection,” inCVPR, 2022

    2022

  28. [28]

    Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,

    X. Bai, Z. Hu, X. Zhu, Q. Huang, Y . Chen, H. Fu, and C.-L. Tai, “Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,” inCVPR, 2022

    2022

  29. [29]

    Camera-lidar integration: Probabilistic sensor fusion for semantic mapping,

    J. S. Berrio, M. Shan, S. Worrall, and E. Nebot, “Camera-lidar integration: Probabilistic sensor fusion for semantic mapping,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 7637–7652, 2021. SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 15

    2021

  30. [30]

    Learning to generate realistic lidar point cloud,

    V . Zyrianov, X. Zhu, and S. Wang, “Learning to generate realistic lidar point cloud,” inECCV, 2022

    2022

  31. [31]

    Towards realistic scene generation with lidar diffusion models,

    H. Ran, V . Guizilini, and Y . Wang, “Towards realistic scene generation with lidar diffusion models,” inCVPR, 2024

    2024

  32. [32]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,

    G. Zhao, X. Wang, Z. Zhu, X. Chen, G. Huang, X. Bao, and X. Wang, “Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,”arXiv preprint arXiv:2403.06845, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving,

    X. Tian, T. Jiang, L. Yun, Y . Mao, H. Yang, Y . Wang, Y . Wang, and H. Zhao, “Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving,”NeurIPS, 2024

    2024

  34. [34]

    Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving,

    Y . Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu, “Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving,” inICCV, 2023

    2023

  35. [35]

    Scene as occupancy,

    W. Tong, C. Sima, T. Wang, L. Chen, S. Wu, H. Deng, Y . Gu, L. Lu, P. Luo, D. Linet al., “Scene as occupancy,” inICCV, 2023, pp. 8406– 8415

    2023

  36. [36]

    Planning-oriented autonomous driving,

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wanget al., “Planning-oriented autonomous driving,” inCVPR, 2023

    2023

  37. [37]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” inCVPR, 2020

    2020

  38. [38]

    Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles,

    K. T. e. a. H. Caesar, J. Kabzan, “Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles,” inCVPR ADP3 workshop, 2021

    2021

  39. [39]

    Neural kernel surface reconstruction,

    J. Huang, Z. Gojcic, M. Atzmon, O. Litany, S. Fidler, and F. Williams, “Neural kernel surface reconstruction,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 4369–4379

    2023

  40. [40]

    Indoor segmentation and support inference from rgbd images

    N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images.”ECCV, 2012

    2012

  41. [41]

    Scenenn: A scene meshes dataset with annotations,

    B.-S. Hua, Q.-H. Pham, D. T. Nguyen, M.-K. Tran, L.-F. Yu, and S.-K. Yeung, “Scenenn: A scene meshes dataset with annotations,” in2016 fourth international conference on 3D vision (3DV). Ieee, 2016, pp. 92–101

    2016

  42. [42]

    Synthcity: A large scale synthetic point cloud,

    D. Griffiths and J. Boehm, “Synthcity: A large scale synthetic point cloud,”arXiv preprint arXiv:1907.04758, 2019

    Pith/arXiv arXiv 1907

  43. [43]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” inCVPR, 2017

    2017

  44. [44]

    Semanticposs: A point cloud dataset with large quantity of dynamic instances,

    Y . Pan, B. Gao, J. Mei, S. Geng, C. Li, and H. Zhao, “Semanticposs: A point cloud dataset with large quantity of dynamic instances,” 2020. [Online]. Available: https://arxiv.org/abs/2002.09147

    Pith/arXiv arXiv 2020

  45. [45]

    Semantickitti: A dataset for semantic scene understanding of lidar sequences,

    J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “Semantickitti: A dataset for semantic scene understanding of lidar sequences,” inICCV, 2019

    2019

  46. [46]

    Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,

    Y . Liao, J. Xie, and A. Geiger, “Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” 2022. [Online]. Available: https://arxiv.org/abs/2109.13410

    Pith/arXiv arXiv 2022

  47. [47]

    Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving,

    O. Contributors, “Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving,” https://github.com/ OpenDriveLab/OpenScene, 2023

    2023

  48. [48]

    Monoscene: Monocular 3d semantic scene completion,

    A.-Q. Cao and R. de Charette, “Monoscene: Monocular 3d semantic scene completion,” inCVPR, 2022

    2022

  49. [49]

    Bridging stereo geometry and bev representation with reliable mutual interaction for semantic scene completion,

    B. Li, Y . Sun, Z. Liang, D. Du, Z. Zhang, X. Wang, Y . Wang, X. Jin, and W. Zeng, “Bridging stereo geometry and bev representation with reliable mutual interaction for semantic scene completion,” inIJCAI, 2024

    2024

  50. [50]

    Tri-perspective view for vision-based 3d semantic occupancy prediction,

    Y . Huang, W. Zheng, Y . Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic occupancy prediction,” inCVPR, 2023

    2023

  51. [51]

    Hierarchical temporal context learning for camera-based semantic scene completion,

    B. Li, J. Deng, W. Zhang, Z. Liang, D. Du, X. Jin, and W. Zeng, “Hierarchical temporal context learning for camera-based semantic scene completion,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 131–148

    2024

  52. [52]

    Occworld: Learning a 3d occupancy world model for autonomous driving,

    W. Zheng, W. Chen, Y . Huang, B. Zhang, Y . Duan, and J. Lu, “Occworld: Learning a 3d occupancy world model for autonomous driving,”arXiv preprint arXiv:2311.16038, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Occsora: 4d occupancy generation models as world simulators for autonomous driving,

    L. Wang, W. Zheng, Y . Ren, H. Jiang, Z. Cui, H. Yu, and J. Lu, “Occsora: 4d occupancy generation models as world simulators for autonomous driving,”arXiv preprint arXiv:2405.20337, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Fb-occ: 3d occupancy prediction based on forward-backward view transformation,

    Z. Li, Z. Yu, D. Austin, M. Fang, S. Lan, J. Kautz, and J. M. Alvarez, “Fb-occ: 3d occupancy prediction based on forward-backward view transformation,”arXiv preprint arXiv:2307.01492, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Semcity: Semantic scene generation with triplane diffusion,

    J. Lee, S. Lee, C. Jo, W. Im, J. Seon, and S.-E. Yoon, “Semcity: Semantic scene generation with triplane diffusion,” inCVPR, 2024

    2024

  56. [56]

    Pyramid diffusion for fine 3d large scene generation,

    Y . Liu, X. Li, X. Li, L. Qi, C. Li, and M.-H. Yang, “Pyramid diffusion for fine 3d large scene generation,”ECCV, 2024

    2024

  57. [57]

    Occllama: An occupancy-language-action generative world model for autonomous driving,

    J. Wei, S. Yuan, P. Li, Q. Hu, Z. Gan, and W. Ding, “Occllama: An occupancy-language-action generative world model for autonomous driving,”arXiv preprint arXiv:2409.03272, 2024

    Pith/arXiv arXiv 2024

  58. [58]

    Occupancy anticipation for efficient exploration and navigation,

    S. K. Ramakrishnan, Z. Al-Halah, and K. Grauman, “Occupancy anticipation for efficient exploration and navigation,” inComputer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer, 2020, pp. 400–418

    2020

  59. [59]

    Structured 3d latents for scalable and versatile 3d generation,

    J. Xiang, Z. Lv, S. Xu, Y . Deng, R. Wang, B. Zhang, D. Chen, X. Tong, and J. Yang, “Structured 3d latents for scalable and versatile 3d generation,”arXiv preprint arXiv:2412.01506, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving,

    Y . Yang, J. Mei, Y . Ma, S. Du, W. Chen, Y . Qian, Y . Feng, and Y . Liu, “Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 9, 2025, pp. 9327–9335

    2025

  61. [61]

    Dynamiccity: Large-scale 4d occupancy generation from dynamic scenes,

    H. Bian, L. Kong, H. Xie, L. Pan, Y . Qiao, and Z. Liu, “Dynamiccity: Large-scale 4d occupancy generation from dynamic scenes,” 2025. [Online]. Available: https://arxiv.org/abs/2410.18084

    arXiv 2025

  62. [62]

    Vista: A generalizable driving world model with high fidelity and versatile controllability,

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li, “Vista: A generalizable driving world model with high fidelity and versatile controllability,”Advances in Neural Information Processing Systems, vol. 37, pp. 91 560–91 596, 2025

    2025

  63. [63]

    Bridging stereo geometry and bev representation with reliable mutual interaction for semantic scene completion,

    B. Li, Y . Sun, Z. Liang, D. Du, Z. Zhang, X. Wang, Y . Wang, X. Jin, and W. Zeng, “Bridging stereo geometry and bev representation with reliable mutual interaction for semantic scene completion,”arXiv preprint arXiv:2303.13959, 2023

    Pith/arXiv arXiv 2023

  64. [64]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,

    Y . Wang, J. He, L. Fan, H. Li, Y . Chen, and Z. Zhang, “Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 749–14 759

    2024

  65. [65]

    Vista: A generalizable driving world model with high fidelity and versatile controllability,

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li, “Vista: A generalizable driving world model with high fidelity and versatile controllability,”arXiv preprint arXiv:2405.17398, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Stable video diffusion: Scaling latent video diffusion models to large datasets,

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Lettset al., “Stable video diffusion: Scaling latent video diffusion models to large datasets,”arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  67. [67]

    Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation,

    J. Lu, Z. Huang, J. Zhang, Z. Yang, and L. Zhang, “Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation,”arXiv preprint arXiv:2312.02934, 2023

    Pith/arXiv arXiv 2023

  68. [68]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inICML, 2021

    2021

  69. [69]

    Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control,

    R. Gao, K. Chen, B. Xiao, L. Hong, Z. Li, and Q. Xu, “Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control,”arXiv preprint arXiv:2411.13807, 2024

    Pith/arXiv arXiv 2024

  70. [70]

    Lidar snowfall simulation for robust 3d object detection,

    M. Hahner, C. Sakaridis, M. Bijelic, F. Heide, F. Yu, D. Dai, and L. Van Gool, “Lidar snowfall simulation for robust 3d object detection,” inCVPR, 2022

    2022

  71. [71]

    Learning compact representations for lidar completion and generation,

    Y . Xiong, W.-C. Ma, J. Wang, and R. Urtasun, “Learning compact representations for lidar completion and generation,” inCVPR, 2023

    2023

  72. [72]

    Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields,

    J. Zhang, F. Zhang, S. Kuang, and L. Zhang, “Nerf-lidar: Generating realistic lidar point clouds with neural radiance fields,”Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7178–7186, Mar. 2024

    2024

  73. [73]

    Gs-lidar: Generating realistic lidar point clouds with panoramic gaussian splatting,

    J. Jiang, C. Gu, Y . Chen, and L. Zhang, “Gs-lidar: Generating realistic lidar point clouds with panoramic gaussian splatting,” 2025

    2025

  74. [74]

    KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way,

    I. Vizzo, T. Guadagnino, B. Mersch, L. Wiesmann, J. Behley, and C. Stachniss, “KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way,”IEEE Robotics and Automation Letters (RA-L), vol. 8, no. 2, pp. 1029–1036, 2023

    2023

  75. [75]

    The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,

    M. Berman, A. R. Triki, and M. B. Blaschko, “The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,” inCVPR, 2018

    2018

  76. [76]

    Latte: Latent diffusion transformer for video generation,

    X. Ma, Y . Wang, G. Jia, X. Chen, Z. Liu, Y .-F. Li, C. Chen, and Y . Qiao, “Latte: Latent diffusion transformer for video generation,”arXiv preprint arXiv:2401.03048, 2024

    Pith/arXiv arXiv 2024

  77. [77]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inICCV, 2023, pp. 4195–4205

    2023

  78. [78]

    Cogvideox: Text-to-video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Fenget al., “Cogvideox: Text-to-video diffusion models with an expert transformer,”arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  79. [79]

    Open-sora plan: Open-source large video generation model,

    B. Lin, Y . Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y . Ye, S. Yuan, L. Chenet al., “Open-sora plan: Open-source large video generation model,”arXiv preprint arXiv:2412.00131, 2024. SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 16

    Pith/arXiv arXiv 2024

  80. [80]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,

    J. Z. Wu, Y . Ge, X. Wang, S. W. Lei, Y . Gu, Y . Shi, W. Hsu, Y . Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” inICCV, 2023

    2023

Showing first 80 references.