Pith. sign in

REVIEW 9 cited by

Dolphins: Multimodal Language Model for Driving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.00438 v1 pith:PWDXUCZN submitted 2023-12-01 cs.CV

Yingzi Ma , Yulong Cao , Jiachen Sun , Marco Pavone , Chaowei Xiao This is my paper · ORCID
classification cs.CV
keywords dolphinsdrivinghuman-likemodelscenariosunderstandingcapabilitiescomplex
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The quest for fully autonomous vehicles (AVs) capable of navigating complex real-world scenarios with human-like understanding and responsiveness. In this paper, we introduce Dolphins, a novel vision-language model architected to imbibe human-like abilities as a conversational driving assistant. Dolphins is adept at processing multimodal inputs comprising video (or image) data, text instructions, and historical control signals to generate informed outputs corresponding to the provided instructions. Building upon the open-sourced pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's reasoning capabilities through an innovative Grounded Chain of Thought (GCoT) process. Then we tailored Dolphins to the driving domain by constructing driving-specific instruction data and conducting instruction tuning. Through the utilization of the BDD-X dataset, we designed and consolidated four distinct AV tasks into Dolphins to foster a holistic understanding of intricate driving scenarios. As a result, the distinctive features of Dolphins are characterized into two dimensions: (1) the ability to provide a comprehensive understanding of complex and long-tailed open-world driving scenarios and solve a spectrum of AV tasks, and (2) the emergence of human-like capabilities including gradient-free instant adaptation via in-context learning and error recovery via reflection.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    S4-Driver uses a multimodal LLM with a sparse 3D spatio-temporal volume representation to achieve self-supervised motion planning that rivals supervised methods on nuScenes and WOMD.

  2. UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A fine-tuned small multimodal LLM outperforms much larger general models on urban tasks in a new benchmark, with caveats about benchmark overlap with training data.

  3. AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AD^2-Bench is a new adverse-weather driving benchmark with hierarchical chain-of-thought annotations and LLM-based quality metrics; 12 MLLMs all scored below 60%.

  4. Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving

    cs.CV 2025-01 conditional novelty 6.0 of 10

    CAD is a transfer-based black-box attack using CLIP embeddings and ChatGPT-generated deceptive reasoning text to make vision-language autonomous driving models take unsafe actions.

  5. Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A new benchmark with clean, corrupted, and text-only driving inputs shows that vision-language models can answer many driving questions without visual information, so standard accuracy metrics overestimate visual grounding.

  6. LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking

    cs.AI 2025-01 conditional novelty 5.0 of 10

    A dual-process knowledge-driven driving framework combining VLM perception, contrastive scene tokens, and a memory bank improves closed-loop driving scores in CARLA and DriveArena simulators.

  7. CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting

    cs.CV 2026-08 conditional novelty 4.0 of 10

    CosmosAlign adapts Cosmos3-Nano with two-stage LoRA, medoid sample selection, and motion-adaptive blending, achieving first place (76.49) on the AI City Challenge 2026 Track 5 traffic video forecasting benchmark.

  8. Explainability for Vision Foundation Models: A Survey

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A structured review of 122 papers on explainability for vision foundation models, with a taxonomy and the finding that quantitative evaluation is rare (36%).

  9. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

Pith tools