Pith. sign in

REVIEW 5 cited by

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01957 v2 pith:OCZ4BE3N submitted 2023-10-03 cs.RO cs.AIcs.CLcs.CV

Long Chen , Oleg Sinavski , Jan Hünermann , Alice Karnsund , Andrew James Willmott , Danny Birch , Daniel Maund , Jamie Shotton This is my paper · ORCID
classification cs.ROcs.AIcs.CLcs.CV
keywords drivingvectorautonomousintroducelanguagellmsmodalitiesnumeric
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique object-level multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available for further exploration.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DriveQA: Passing the Driving Knowledge Test

    cs.CV 2025-08 conditional novelty 6.0 of 10

    DriveQA is a new multimodal driving-knowledge benchmark showing that LLMs and MLLMs struggle with right-of-way, numerical traffic rules, and sign variations, with modest transfer gains to nuScenes and BDD.

  2. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 80K-clip dataset of unstructured driving scenarios with Q&A annotations improves VLA performance on NeuroNCAP and nuScenes benchmarks.

  3. Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models

    cs.RO 2025-02 conditional novelty 6.0 of 10

    Occ-LLM tokenizes 4D occupancy with a motion/static separation VAE and uses Llama-2 to forecast occupancy, plan ego motion, and answer scene questions, reporting state-of-the-art results on nuScenes.

  4. IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A visual heatmap tool lets people rate vision-language model reliability in video by inspecting patterns of green and red cells, with user ratings tracking objective F1 scores when those exist.

  5. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools