REVIEW 5 cited by
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model, enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.
Forward citations
Cited by 5 Pith papers
-
STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.
-
CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining
Forecasting future LiDAR from historical camera–radar inputs pretrains a transferable CR BEV backbone that improves long-horizon geometry prediction and many nuScenes driving tasks.
-
OmniNWM: Omniscient Driving Navigation World Models
OmniNWM jointly generates long panoramic multi-modal driving videos, controls them precisely via normalized Plücker ray-maps, and derives dense driving rewards from generated 3D occupancy.
-
Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
A joint video and LiDAR generation framework for driving scenes, conditioned on shared scene layouts, VLM captions, and BEV features, achieves SOTA generation and downstream perception gains on nuScenes.
-
DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving
DriveX predicts future latent BEV features from driving video and shows consistent, modest gains on occupancy, flow, and end-to-end driving, though no code is released.
Discussion (0). Continue with ORCID to comment.