Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-supervised latent world model can predict driving futures well enough to lift occupancy, flow, and planning performance while its encoder stays frozen.

desk verdict DriveX is a solid, well-ablated world model paper; the frozen-backbone result is the strongest claim, but pseudo-label dependence tempers 'generalizable semantic knowledge'. read the letter →

arxiv 2505.19239 v1 pith:QNGXB7QB submitted 2025-05-25 cs.CV

Chen Shi , Shaoshuai Shi , Kehua Sheng , Bo Zhang , Li Jiang This is my paper
classification cs.CV
keywords worldmodelsautonomousdrivingbird's-eyeviewrepresentationself-supervisedlearningpointcloudforecastingoccupancypredictionend-to-endlatentspace
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a world model for autonomous driving can be trained without human annotations and still transfer to multiple downstream tasks. DriveX encodes multi-view video into a bird's-eye-view latent space, supervises that space with depth, auto-generated semantic labels, and color, then learns to predict future latent states conditioned on ego action. Downstream models draw on those predicted states through a spatial attention mechanism. The reported results include lower future point-cloud Chamfer distance, higher occupancy and flow accuracy, and competitive end-to-end driving scores, with a frozen world encoder performing almost as well as a fine-tuned one.

What carries the argument

Omni Scene Modeling (OSM) is the central module: it casts rays from camera and LiDAR views through a BEV feature volume and uses volume rendering to predict semantic labels, color, and depth, forcing the world encoder to store geometry, class, and texture in one grid. The second piece is decoupled latent world modeling: a flow-based future decoder predicts future grid features directly, conditioned on ego action, and warps them with ego-motion transformations, avoiding autoregressive error accumulation. The third piece is Future Spatial Attention (FSA), which lets task-specific queries sample from these predicted future BEV features by transforming sampling points into future-frame coordinates, so existing perception and planning heads can consume the world model's forecasts without architectural changes.

What would settle it

Retrain the same pipeline with human-annotated semantic labels in place of the foundation-model pseudo-labels and compare the occupancy mIoU gain over the depth-only variant; a substantial drop in that gain would show the reported semantic transfer is an artifact of label noise rather than of the world model.

Watch

Extended reading notes

Core claim

The central claim is that world knowledge for driving can be captured in a spatially structured latent representation by combining three self-supervised signals at once: LiDAR depth through volume rendering, semantic categories from foundation-model auto-labels, and raw color. DriveX then separates learning into two stages: first train a streaming bird's-eye-view encoder with these Omni Scene Modeling losses, then freeze it and train a future decoder that predicts latent BEV states at 0.5-second intervals over a 3-second horizon, aligned to the encoder's own future encodings and regularized by the same scene-modeling losses. Downstream tasks query these future states with Future Spatial Attention. On nuScenes and FlowOcc3D, the paper reports a 6% relative reduction in 3-second Chamfer distance for point-cloud forecasting, +0.93 mIoU and +1.08 IoU_geo over the ViewFormer baseline for occupancy with flow, an average endpoint error of 0.385, and an 84.5 PDMS on NAVSIM; when the world encoder is frozen instead of fine-tuned, the end-to-end score drops only from 83.7 to 83.6, compared with drops of 6.6 and 1.5 for ViDAR and DriveWorld.

Load-bearing premise

The semantic branch of the self-supervision uses auto-generated labels from foundation models that the paper concedes lose information, and if those labels are systematically inconsistent with the human-annotated labels used in downstream benchmarks, the claimed transferable semantic knowledge would be compromised.

Editorial extensions

If this is right

  • A single pretrained, frozen world encoder can serve as shared infrastructure: adding a lightweight adapter and FSA to a task-specific head recovers most of the fine-tuning benefit, so downstream training cost drops.
  • Future states can be predicted directly at arbitrary horizons instead of autoregressively, which removes error accumulation and lets downstream tasks choose how many future frames to attend to.
  • Because all three supervision signals are self-supervised or auto-labeled, the same training recipe scales with unlabeled driving video; the paper reports consistent Chamfer-distance gains from 10% to 100% of the training data.
  • The gains on occupancy, flow, and planning jointly imply that geometric, semantic, and motion knowledge can live in one latent space, rather than being traded off against each other as prior work assumes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the same frozen checkpoint could be shared across perception, prediction, and planning modules in one vehicle stack, since each task only adds a query head; the paper demonstrates this one task at a time.
  • A testable extension is replacing the foundation-model semantic labels with a purely self-supervised clustering or contrastive objective; if the occupancy gains persist, the semantic branch may not actually be carrying class knowledge.
  • Because the world encoder is camera-only, the recipe may transfer to sensor configurations without LiDAR; LiDAR enters only as depth supervision during training, not as a runtime input.
  • The dynamic-aware ray sampling depends on an external tracker to identify moving objects, so the self-supervision claim is strongest for static regions; learning motion saliency from the future decoder's own flow estimates would close that gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DriveX is a self-supervised world model for autonomous driving that learns bird's-eye-view (BEV) latent representations from multi-view video and ego actions. The paper's two principal technical contributions are (i) Omni Scene Modeling (OSM), which supervises the latent representation with three signals—LiDAR depth, 2D semantic labels auto-generated by Grounded SAM/OpenSeeD, and image color reconstruction—and (ii) a decoupled latent world modeling strategy that first trains the world encoder with OSM losses and then trains a future decoder to predict latent BEV states, supported by dynamic-aware ray sampling. Downstream integration is handled by a Future Spatial Attention (FSA) module. Experiments on nuScenes and NAVSIM report improvements in 3D point cloud forecasting over prior work, in 3D occupancy and flow prediction on FlowOcc3D, and in NAVSIM end-to-end driving, together with ablations of the OSM components, prediction horizon, decoupled training, and data scale.

Significance. If the claims hold, this is a useful step toward general-purpose driving world models: the decoupled training strategy is a clean way to separate representation learning from dynamics modeling, the unified FSA adapter has practical value, and the experimental coverage across three task families is a strength. The ablations are systematic, and Table 10 provides a welcome robustness check over weather and lighting conditions. However, the central claim of learning generalizable semantic world knowledge rests on a load-bearing assumption that is not tested: that pseudo-labels from Grounded SAM/OpenSeeD align with the human-annotated or auto-labeled ground truth used in the downstream benchmarks. The paper also overstates its self-supervision claim because dynamic-aware ray sampling relies on an external object detector. These issues, together with the absence of statistical precision, make the current evidence conditional.

major comments (4)
  1. [Sec 3.1 / Table 5] Table 5 shows mIoU jumping from 3.96 with depth-only supervision to 42.53 when semantic pseudo-labels from Grounded SAM/OpenSeeD are added, while color reconstruction contributes only a further 0.94. This makes the pseudo-label branch load-bearing for the occupancy and semantic-transfer claims, yet the paper never quantifies agreement between these auto-generated labels and the FlowOcc3D labels used for evaluation. Because the evaluation benchmarks (OpenScene [7], FlowOcc3D [30], OccNeRF classes [68]) may share a similar auto-labeling pipeline, the large semantic gains could reflect label-distribution alignment rather than generalizable world knowledge. Please report per-class IoU or another agreement metric between the pseudo-labels and the downstream benchmark labels, and either add an experiment with human-annotated semantic labels or evaluate on a benchmark with a different label ontology.
  2. [Sec 3.2 / training details (Sec 4.1)] The abstract and Sec 3.1 describe the approach as fully self-supervised, but dynamic-aware ray sampling uses DetZero [41] to identify moving objects. DetZero is a 3D object detector that must be trained or configured with supervision. The paper does not disclose whether DetZero was used with human-annotated ground-truth boxes or as a pretrained detector, nor does it ablate this component against an unsupervised dynamics proxy. Please clarify the source of DetZero's moving-object annotations and quantify the impact of dynamic-aware ray sampling in an ablation that does not use DetZero (e.g., using point-cloud residuals or optical flow).
  3. [Tables 3 and 4] Several headline improvements are small relative to expected run-to-run variation: +0.77 and +0.93 mIoU in Table 3, -0.027 mA VE, and +0.9 PDMS in Table 4. The paper reports only single runs and uses author re-implementations for ViDAR, DriveWorld, and DiffusionDrive-C. Without multiple seeds, standard deviations, or significance tests, it is unclear whether these differences exceed noise, especially for the NAVSIM comparison against a re-implemented DiffusionDrive-C baseline. Please provide mean and standard deviation over at least three seeds for the main comparisons, and make the re-implementation configurations public or describe them in enough detail to reproduce.
  4. [Eq. (8)] The latent alignment loss in Eq. (8) supervises the future decoder with B_{t+k}, which is produced by the same world encoder that the future decoder is supposed to learn from. This is a self-distillation signal, and the paper does not isolate its contribution from the future OSM loss L_scene in Eq. (9). Without an ablation that removes L_latent, the decoupling claim in Table 7 may reflect optimization benefits rather than genuinely new predictive knowledge. Please report the ablation of Eq. (8) against using only L_scene for future decoding, and show whether the decoupled training gain persists without latent alignment.
minor comments (6)
  1. [Table 4] The 'Improvement' row is misaligned: the first value under the 'NC' column appears as '-' and the subsequent deltas are shifted, making it hard to read which metric each improvement refers to.
  2. [Table 9] The data-scale ablation does not state how the 10%, 50%, and 100% subsets were constructed. Please specify whether these are random, seed-fixed subsets of the training set and report the seed used.
  3. [Sec 3.1] There is a typo in Eq. (4): 'attribution filed' should read 'attribution field'.
  4. [Sec 4.4] The name 'DriffusionDrive-C' appears in the efficiency paragraph; this should be 'DiffusionDrive-C'.
  5. [Sec 3, first paragraph] The references in 'Another line of world models [11,32,55,56] explores to generate future video sequences' are appropriate, but the earlier sentence describing world modeling as an auxiliary task cites video-generation papers [17,24,54]; these citations do not correspond to driving auxiliary-task world models and should be corrected or moved to the video-generation context.
  6. [Sec 3.1] The semantic labels use 15 classes defined in [68] (OccNeRF). Please clarify explicitly whether the same class mapping and label schema are used for the FlowOcc3D and OpenScene evaluation labels, since a mismatch could confound the semantic transfer results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predicted future states are grounded in external sensor supervision and evaluated on independent benchmarks; the latent alignment objective is supplemented by direct future OSM losses.

full rationale

DriveX's central claims are checked against external data and benchmarks: future point cloud forecasting is measured by Chamfer distance against nuScenes LiDAR sweeps, occupancy and flow use FlowOcc3D annotations, and driving performance uses the NAVSIM PDMS. The latent alignment term in Eq. (8) is an internal teacher-student objective, but it is not the only training signal for the future decoder: Eq. (9) adds Lscene at future timestamps, which supervises predicted BEV features through volume rendering against raw future depth, future images, and auto-generated semantic maps, so the predicted future states are anchored to actual future sensor data rather than merely to the world encoder's own outputs. The semantic supervision from Grounded SAM and OpenSeeD is an external input; the authors' caveat that auto-labeled semantics 'inevitably suffer from information loss' identifies a real validity risk for transferable semantic knowledge, but it is not a circularity unless the downstream evaluation labels were produced by the same auto-labeling pipeline, which the paper does not claim and this analysis therefore does not assume. Self-citations appearing in the references are routine related-work citations and are not load-bearing for any derivation. No equation or experimental result reduces to its own inputs by construction, and no fitted parameter is renamed as a prediction. The paper is self-contained against external benchmarks, so the appropriate circularity finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claims rest on standard neural network architecture choices, hand-tuned hyperparameters, and several domain assumptions about the quality of pseudo-labels and the suitability of BEV space for future prediction. No new physical entities are introduced, but the reliance on foundation-model semantic labels and DetZero for motion identification means the system is not fully self-contained.

free parameters (6)
  • Loss weights for latent alignment and scene modeling = ωl=1.0, ωs=0.5
    Chosen in Sec 4.1 without sensitivity analysis; directly weights the future decoder objective in Eq. 9.
  • Ray counts for supervision = 8000 LiDAR rays, 10000 image rays, 2000 dynamic rays
    Hand-picked in Sec 4.1; affects training efficiency and the balance between static and dynamic regions.
  • Future horizon frames = 2 for occupancy/flow, 3 for end-to-end driving
    Selected to balance performance and cost (Table 6); not derived.
  • Number of semantic classes = 15 classes
    Defined in Sec 4.1 following OccNeRF [68]; determines the semantic supervision head.
  • Input image resolutions = 256x704 (DriveX-S) and 900x1600 (DriveX-B)
    Two model variants; the larger variant is used for the point cloud forecasting SOTA claim.
  • Ray waypoint distances = predefined distances (not specified numerically)
    Used in Eq. 2 for volume rendering; the exact lambda schedule is omitted from the paper.
assumptions (6)
  • domain assumption Foundation model pseudo-labels (Grounded SAM, OpenSeeD) provide sufficiently accurate semantic supervision for world representation learning.
    Invoked in Sec 3.1; the authors note these labels have information loss, yet the semantic reasoning of DriveX is built on them.
  • domain assumption The BEV grid discretization and flow-based future propagation (Eq. 7) adequately capture the world dynamics of a driving scene.
    Assumed in Sec 3.2; moving objects and ego motion are handled by predicted flow and ego-motion transforms.
  • standard math The volume rendering equations (Eq. 5) correctly aggregate per-waypoint attributes into scene predictions.
    Standard neural rendering formulation used in Sec 3.1.
  • domain assumption The future action conditioning for perception tasks via AD-MLP is accurate enough to provide useful future context.
    Described in Sec 3.3; no ablation on action prediction error is provided.
  • domain assumption DetZero's offboard detection correctly identifies moving objects for dynamic-aware ray sampling.
    Used in Sec 3.2 and Sec 4.1; if objects are missed, motion supervision is weakened.
  • domain assumption Training the future decoder against the world encoder's own output B_{t+k} (Eq. 8) yields a latent space that is a useful target for downstream tasks.
    Self-supervised target defined by the same encoder; downstream transfer is empirical evidence but not a derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving." pith.science (2026). https://pith.science/paper/QNGXB7QB

@misc{pith2026250519239,
  author       = {Pith},
  title        = {Pith review of: DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNGXB7QB}},
  note         = {Machine review of arXiv:2505.19239}
}
read the original abstract

Data-driven learning has advanced autonomous driving, yet task-specific models struggle with out-of-distribution scenarios due to their narrow optimization objectives and reliance on costly annotated data. We present DriveX, a self-supervised world model that learns generalizable scene dynamics and holistic representations (geometric, semantic, and motion) from large-scale driving videos. DriveX introduces Omni Scene Modeling (OSM), a module that unifies multimodal supervision-3D point cloud forecasting, 2D semantic representation, and image generation-to capture comprehensive scene evolution. To simplify learning complex dynamics, we propose a decoupled latent world modeling strategy that separates world representation learning from future state decoding, augmented by dynamic-aware ray sampling to enhance motion modeling. For downstream adaptation, we design Future Spatial Attention (FSA), a unified paradigm that dynamically aggregates spatiotemporal features from DriveX's predictions to enhance task-specific inference. Extensive experiments demonstrate DriveX's effectiveness: it achieves significant improvements in 3D future point cloud prediction over prior work, while attaining state-of-the-art results on diverse tasks including occupancy prediction, flow estimation, and end-to-end driving. These results validate DriveX's capability as a general-purpose world model, paving the way for robust and unified autonomous driving frameworks.

Figures

Figures reproduced from arXiv: 2505.19239 by the authors.

Figure 1
Figure 1. Comparison of different methods for integrating world [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Top: An overview of our DriveX framework. The learning process consists of two stages: world representation learning, where the model learns temporal and geometric semantics through Omni Scene Modeling, and latent future decoding, where the model predicts future states in the learned latent space. Both stages are trained in a self-supervised manner. Bottom: Illustration of the FSA paradigm. Various driving tasks can… view at source ↗
Figure 3
Figure 3. Illustration of the flow-based future forecasting pipeline. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniNWM: Omniscient Driving Navigation World Models

    cs.CV 2025-10 conditional novelty 6.0 of 10

    OmniNWM jointly generates long panoramic multi-modal driving videos, controls them precisely via normalized Plücker ray-maps, and derives dense driving rewards from generated 3D occupancy.

  2. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

74 extracted references · 57 canonical work pages · cited by 2 Pith papers

  1. [7]

    Openscene: The largest up-to- date 3d occupancy prediction benchmark in autonomous driving.https://github.com/OpenDriveLab/ OpenScene, 2023

    OpenScene Contributors. Openscene: The largest up-to- date 3d occupancy prediction benchmark in autonomous driving.https://github.com/OpenDriveLab/ OpenScene, 2023. 7

    work page 2023

  2. [30]

    Viewformer: Exploring spatiotem- poral modeling for multi-view 3d occupancy perception via view-guided transformers

    Jinke Li, Xiao He, Chonghua Zhou, Xiaoqiang Cheng, Yang Wen, and Dan Zhang. Viewformer: Exploring spatiotem- poral modeling for multi-view 3d occupancy perception via view-guided transformers. InECCV, 2024. 7, 8, 9

    work page 2024

  3. [68]

    Occnerf: Advancing 3d occupancy prediction in lidar-free environments.arXiv preprint arXiv:2312.09243, 2023

    Chubin Zhang, Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yan- song Tang, Yueqi Duan, and Jiwen Lu. Occnerf: Advancing 3d occupancy prediction in lidar-free environments.arXiv preprint arXiv:2312.09243, 2023. 7

    arXiv 2023

  4. [41]

    Detzero: Rethinking offboard 3d object detection with long-term sequential point clouds

    Tao Ma, Xuemeng Yang, Hongbin Zhou, Xin Li, Botian Shi, Junjie Liu, Yuchen Yang, Zhizheng Liu, Liang He, Yu Qiao, et al. Detzero: Rethinking offboard 3d object detection with long-term sequential point clouds. InICCV, 2023. 7

    work page 2023

  5. [1]

    Uno: Unsupervised occupancy fields for perception and forecasting

    Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. InCVPR, 2024. 1

    work page 2024

  6. [2]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos J Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in atari. NeurIPS, 2024. 3

    work page 2024

  7. [3]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InCVPR, 2020. 6, 7, 8

    work page 2020

  8. [4]

    Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024

    Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation.arXiv preprint arXiv:2411.00769, 2024. 3

    arXiv 2024

Show all 74 references
  1. [5]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. InCVPR, 2024. 3

    2024

  2. [6]

    Transfuser: Imitation with transformer-based sensor fusion for autonomous driv- ing.TPAMI, 2022

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driv- ing.TPAMI, 2022. 7

    2022

  3. [8]

    Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking. InNeurIPS, ...

    2024

  4. [9]

    Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023. 3

    2023 arXiv

  5. [10]

    Magicdrive3d: Controllable 3d genera- tion for any-view rendering in street scenes.arXiv preprint arXiv:2405.14475, 2024

    Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive3d: Controllable 3d genera- tion for any-view rendering in street scenes.arXiv preprint arXiv:2405.14475, 2024

    2024 arXiv

  6. [11]

    Vista: A generalizable driving world model with high fidelity and versatile controllability.NeurIPS, 2024

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yi- hang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability.NeurIPS, 2024. 1, 2, 3

    2024

  7. [12]

    End-to-end autonomous driving without costly modularization and 3d manual annotation.arXiv preprint arXiv:2406.17680, 2024

    Mingzhe Guo, Zhipeng Zhang, Yuan He, Ke Wang, and Lip- ing Jing. End-to-end autonomous driving without costly modularization and 3d manual annotation.arXiv preprint arXiv:2406.17680, 2024. 1

    2024 arXiv

  8. [13]

    Recurrent world models facilitate policy evolution.NeurIPS, 2018

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution.NeurIPS, 2018. 3

    2018

  9. [14]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. InICLR, 2020

    2020

  10. [15]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. InICLR, 2021

    2021

  11. [16]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 3

    2023 arXiv

  12. [17]

    Flexible diffusion modeling of long videos.NeurIPS, 2022

    William Harvey, Saeid Naderiparizi, Vaden Masrani, Chris- tian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos.NeurIPS, 2022. 2, 3

    2022

  13. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR,

  14. [19]

    Denoising diffu- sion probabilistic models.NeurIPS, 33, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models.NeurIPS, 33, 2020. 3 9

    2020

  15. [20]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. InICLR, 2023. 3

    2023

  16. [21]

    Gaia-1: A generative world model for au- tonomous driving.arXiv preprint arXiv:2309.17080, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving.arXiv preprint arXiv:2309.17080, 2023. 2, 3

    2023 arXiv

  17. [22]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023. 1, 7

    2023

  18. [23]

    Vad: Vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. InICCV, 2023. 1

    2023

  19. [24]

    Text2video-zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- to-image diffusion models are zero-shot video generators. In ICCV, 2023. 2, 3

    2023

  20. [25]

    Differentiable raycasting for self-supervised occupancy forecasting

    Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable raycasting for self-supervised occupancy forecasting. InECCV, 2022. 3

    2022

  21. [26]

    Point cloud forecasting as a proxy for 4d occupancy forecasting

    Tarasha Khurana, Peiyun Hu, David Held, and Deva Ra- manan. Point cloud forecasting as a proxy for 4d occupancy forecasting. InCVPR, 2023. 2, 3, 6

    2023

  22. [27]

    Pointpillars: Fast encoders for object detection from point clouds

    Alex H Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. InCVPR, 2019. 1

    2019

  23. [28]

    A path towards autonomous machine intelli- gence version 0.9

    Yann LeCun. A path towards autonomous machine intelli- gence version 0.9. 2, 2022-06-27.Open Review, 2022. 3

    2022

  24. [29]

    T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback.NeurIPS, 2024

    Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback.NeurIPS, 2024. 3

    2024

  25. [31]

    Navigation-guided sparse scene representation for end-to-end autonomous driving

    Peidong Li and Dixiao Cui. Navigation-guided sparse scene representation for end-to-end autonomous driving. InICLR,

  26. [32]

    Drivingdiffusion: Layout-guided multi-view driving scenarios video genera- tion with latent diffusion model

    Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scenarios video genera- tion with latent diffusion model. InECCV, 2024. 1, 2, 3

    2024

  27. [33]

    Semi-supervised vision-centric 3d occu- pancy world model for autonomous driving

    Xiang Li, Pengfei Li, Yupeng Zheng, Wei Sun, Yan Wang, and yilun chen. Semi-supervised vision-centric 3d occu- pancy world model for autonomous driving. InICLR, 2025. 3

    2025

  28. [34]

    Enhancing end-to-end autonomous driving with latent world model

    Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. InICLR, 2025. 1, 3

    2025

  29. [35]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chong- hao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. InECCV, 2022. 4, 7

    2022

  30. [36]

    Fb-bev: Bev representa- tion from forward-backward view transformations

    Zhiqi Li, Zhiding Yu, Wenhai Wang, Anima Anandkumar, Tong Lu, and Jose M Alvarez. Fb-bev: Bev representa- tion from forward-backward view transformations. InICCV,

  31. [37]

    Is ego status all you need for open- loop end-to-end autonomous driving? InCVPR, 2024

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving? InCVPR, 2024. 1

    2024

  32. [38]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving

    Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. InCVPR, 2025. 7, 8, 9

    2025

  33. [39]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. InCVPR, 2017. 4, 7

    2017

  34. [40]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 7

    2019

  35. [42]

    Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks

    Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. InCoRL, 2022. 3

    2022

  36. [43]

    Driveworld: 4d pre-trained scene understanding via world models for au- tonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, Liping Jing, Yiming Nie, and Bin Dai. Driveworld: 4d pre-trained scene understanding via world models for au- tonomous driving. InCVPR, 2024. 2, 3

    2024

  37. [44]

    Driveworld: 4d pre-trained scene understanding via world models for autonomous driving

    Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. InCVPR, 2024. 1, 3

    2024

  38. [45]

    Renderocc: Vision-centric 3d occupancy prediction with 2d rendering supervision

    Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Hongwei Xie, Bing Wang, Li Liu, and Shanghang Zhang. Renderocc: Vision-centric 3d occupancy prediction with 2d rendering supervision. InICRA, 2024. 3, 5

    2024

  39. [46]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159,

    arXiv

  40. [47]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. InICML, 2020. 3

    2020

  41. [48]

    Pv- rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection.IJCV, 2023

    Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv- rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection.IJCV, 2023. 1

    2023

  42. [49]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. InICLR, 2021. 3

    2021

  43. [50]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 3

    2024 arXiv

  44. [51]

    Exploring object-centric temporal modeling 10 for efficient multi-view 3d object detection

    Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xi- angyu Zhang. Exploring object-centric temporal modeling 10 for efficient multi-view 3d object detection. InICCV, 2023. 1

    2023

  45. [52]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning.arXiv preprint arXiv:2405.01533, 2024. 1

    2024 arXiv

  46. [53]

    Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions. In CVPR, 2023. 4, 7

    2023

  47. [54]

    Videocomposer: Compositional video synthesis with motion controllability.NeurIPS, 2024

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability.NeurIPS, 2024. 2, 3

    2024

  48. [55]

    Drivedreamer: Towards real-world- driven world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- driven world models for autonomous driving. InECCV,

  49. [56]

    Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving. InCVPR, 2024. 1, 2, 3

    2024

  50. [57]

    Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for se- quential pose forecasting

    Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for se- quential pose forecasting. InCoRL, 2021. 3

    2021

  51. [58]

    S2net: Stochastic sequential pointcloud forecasting

    Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAl- lister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Ki- tani. S2net: Stochastic sequential pointcloud forecasting. In ECCV, 2022. 3

    2022

  52. [59]

    Para-drive: Parallelized architecture for real- time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real- time autonomous driving. InCVPR, 2024. 7

    2024

  53. [60]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. InCoRL, 2023. 3

    2023

  54. [61]

    Pred: pre-training via semantic rendering on lidar point clouds

    Hao Yang, Haiyang Wang, Di Dai, and Liwei Wang. Pred: pre-training via semantic rendering on lidar point clouds. NeurIPS, 2024. 5

    2024

  55. [62]

    Driving in the occupancy world: Vision-centric 4d occupancy forecast- ing and planning via world models for autonomous driving

    Yu Yang, Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric 4d occupancy forecast- ing and planning via world models for autonomous driving. arXiv preprint arXiv:2408.14197, 2024. 1, 2, 3, 6, 7, 8

    2024 arXiv

  56. [63]

    Visual point cloud forecasting enables scalable autonomous driving

    Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. InCVPR, 2024. 1, 2, 3, 6, 7

    2024

  57. [64]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. InICLR, 2025. 3

    2025

  58. [65]

    Cvt-occ: Cost volume temporal fusion for 3d occupancy prediction

    Zhangchen Ye, Tao Jiang, Chenfeng Xu, Yiming Li, and Hang Zhao. Cvt-occ: Cost volume temporal fusion for 3d occupancy prediction. InECCV, 2024. 7

    2024

  59. [66]

    Center- based 3d object detection and tracking

    Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center- based 3d object detection and tracking. InCVPR, 2021. 1

    2021

  60. [67]

    Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023

    Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023. 6

    2023 arXiv

  61. [69]

    A simple framework for open-vocabulary segmentation and detection

    Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. InICCV,

  62. [70]

    Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion

    Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. InICLR, 2024. 2, 3

    2024

  63. [71]

    Occformer: Dual-path transformer for vision-based 3d semantic occu- pancy prediction

    Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision-based 3d semantic occu- pancy prediction. InICCV, 2023. 1

    2023

  64. [72]

    Occworld: Learning a 3d occupancy world model for autonomous driving

    Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. InECCV,

  65. [73]

    Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation.arXiv preprint arXiv:2501.14729, 2025

    Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation.arXiv preprint arXiv:2501.14729, 2025. 6, 7

    2025 arXiv

  66. [74]

    Deformable detr: Deformable transformers for end-to-end object detection.NeurIPS, 2020

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection.NeurIPS, 2020. 4 11

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.