REVIEW 8 cited by
CarLLaVA: Vision language models for camera-only closed-loop driving
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this technical report, we present CarLLaVA, a Vision Language Model (VLM) for autonomous driving, developed for the CARLA Autonomous Driving Challenge 2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA architecture as backbone, achieving state-of-the-art closed-loop driving performance with only camera input and without the need for complex or expensive labels. Additionally, we show preliminary results on predicting language commentary alongside the driving output. CarLLaVA uses a semi-disentangled output representation of both path predictions and waypoints, getting the advantages of the path for better lateral control and the waypoints for better longitudinal control. We propose an efficient training recipe to train on large driving datasets without wasting compute on easy, trivial data. CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving Challenge 2.0 outperforming the previous state of the art by 458% and the best concurrent submission by 32.6%.
Forward citations
Cited by 8 Pith papers
-
TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving
TaCarla releases 2.85M CARLA Leaderboard 2.0 frames with nuScenes-style sensors, multi-task annotations, planning baselines, and a text-based rarity score.
-
AlignDrive: Aligned Lateral-Longitudinal Planning for End-to-End Autonomous Driving
Conditioning speed planning on the predicted path and relabeling synthetic cut-ins yields SOTA Bench2Drive scores (DS 89.07, SR 73.18%).
-
DriveQA: Passing the Driving Knowledge Test
DriveQA is a new multimodal driving-knowledge benchmark showing that LLMs and MLLMs struggle with right-of-way, numerical traffic rules, and sign variations, with modest transfer gains to nuScenes and BDD.
-
ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models
An asynchronous dual-system architecture forecasts large-model features into the current frame and adds a small-model update to drive in near real time, scoring 69.53 on Bench2Drive at 50 ms.
-
Generalized Trajectory Scoring for End-to-end Multimodal Planning
GTRS combines super-dense vocabulary training, dropout, sensor augmentation, and diffusion proposals to reach 49.4 EPDMS on the Navhard benchmark, approaching the privileged PDM-Closed method.
-
LaViPlan : Language-Guided Visual Path Planning with RLVR
LaViPlan uses RLVR with GRPO and ADE/FDE rewards to fine-tune a 2B VLM for trajectory prediction, improving ADE/FDE on ROADWork and a normalized safety score on CODA-LM over supervised fine-tuning.
-
A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
A position/review paper argues data-driven model predictive control is the best route to safe, adaptive, human-like autonomous-driving motion planning, but provides no new derivation or experiment.
-
Generative AI for Autonomous Driving: A Review
A review of generative models (VAEs, GANs, diffusion, transformers, LLMs) applied to map generation, scenario generation, trajectory prediction, and motion planning for autonomous driving.
Discussion (0). Continue with ORCID to comment.