Pith. sign in

REVIEW 6 cited by

Embodied Understanding of Driving Scenarios

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.04593 v1 pith:TXS7T252 submitted 2024-03-07 cs.CV

Yunsong Zhou , Linyan Huang , Qingwen Bu , Jia Zeng , Tianyu Li , Hang Qiu , Hongzi Zhu , Minyi Guo
show 2 more authors
Yu Qiao Hongyang Li
This is my paper · ORCID
classification cs.CV
keywords drivingunderstandingembodiedspatialagentsaspectsautonomousmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 80K-clip dataset of unstructured driving scenarios with Q&A annotations improves VLA performance on NeuroNCAP and nuScenes benchmarks.

  2. TUMTraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A new roadside-traffic video benchmark unifies multiple-choice QA, referred object captioning, and spatio-temporal grounding, with a Qwen baseline showing open models still struggle on spatial reasoning.

  3. Embodied Scene Understanding for Vision Language Models via MetaVQA

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Fine-tuning on the auto-generated MetaVQA VQA corpus improves VLMs' spatial reasoning accuracy and partially improves their closed-loop driving safety in simulation.

  4. WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Joint training on driving-knowledge QA (LingoQA, DRAMA) and CARLA trajectory data yields a VLM (WiseAD) that improves closed-loop driving score by 11.9% over trajectory-only training.

  5. LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking

    cs.AI 2025-01 conditional novelty 5.0 of 10

    A dual-process knowledge-driven driving framework combining VLM perception, contrastive scene tokens, and a memory bank improves closed-loop driving scores in CARLA and DriveArena simulators.

  6. RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving

    cs.CV 2024-12 conditional novelty 5.0 of 10

    One 8B multimodal model trained jointly on six driving datasets outperforms individual specialists on average and transfers zero-shot to three unseen driving benchmarks.

Pith tools