REVIEW 7 cited by
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.
Forward citations
Cited by 7 Pith papers
-
VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models
VISA improves closed-set 3D occupancy mIoU on nuScenes by using VLM instance audits as reliability-weighted semantic supervisors during training of existing world models.
-
S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation
S4-Driver uses a multimodal LLM with a sparse 3D spatio-temporal volume representation to achieve self-supervised motion planning that rivals supervised methods on nuScenes and WOMD.
-
Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving
A driving planner that predicts a future-ego-trajectory latent and retrieves executable trajectories from a fixed memory reaches 91.3 PDMS on NAVSIM v1 without reconstructing the scene.
-
OmniNWM: Omniscient Driving Navigation World Models
OmniNWM jointly generates long panoramic multi-modal driving videos, controls them precisely via normalized Plücker ray-maps, and derives dense driving rewards from generated 3D occupancy.
-
Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models
Occ-LLM tokenizes 4D occupancy with a motion/static separation VAE and uses Llama-2 to forecast occupancy, plan ego motion, and answer scene questions, reporting state-of-the-art results on nuScenes.
-
Quo Vadis, World Modeling?
An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.
-
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
UniScenev2 scales occupancy-centric driving-scene generation to NuPlan scale, releasing a 3.6M-frame semantic-occupancy dataset and jointly generating occupancy, video, and LiDAR that beats published baselines on its ...
Discussion (0). Continue with ORCID to comment.