Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that state-of-the-art vision language models cannot yet be trusted in safety-critical driving scenes: across 14 models up to 72 billion parameters, none scored above 40% accuracy on DVBench, a new 10,000-question…

desk verdict A valuable benchmark whose headline numbers are undercut by model-dependent item selection and questions that aren't visually answerable from the clips; the dataset is worth engaging with, the performance claims need major caveats. read the letter →

arxiv 2504.14526 v1 pith:B4NV64EN submitted 2025-04-20 cs.CV cs.CL

Tong Zeng , Longfeng Wu , Liang Shi , Dawei Zhou , Feng Guo This is my paper · ORCID
classification cs.CVcs.CL
keywords DVBenchvisionlanguagemodelsautonomousdrivingsafetysafety-criticaleventsvideounderstandingbenchmarknaturalisticdatamultiple-choicequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer a practical question: can today's vision language models—systems that watch video and answer questions about it—understand the moments that matter most in driving, namely crashes and near-crashes? To find out, it builds DVBench, a 10,000-question multiple-choice benchmark drawn from real naturalistic driving footage, in which every question targets the perceptual and reasoning tasks an automated driving system would face. Testing 14 open models, the paper finds that none scores above 40% accuracy under its stricter consistent-answer evaluation, with reasoning tasks such as fault analysis and event-trigger identification faring worse than perception. The paper concludes that general-purpose vision language models are not road-ready for safety-critical driving, and shows that fine-tuning on domain data lifts accuracy by 5.24 to 10.94 percentage points, indicating targeted adaptation is necessary.

What carries the argument

The load-bearing object is DVBench itself: roughly 10,000 multiple-choice questions, each built from a five-second, 432×324 front-camera clip extracted from naturalistic driving video at the annotated event time, with question-answer pairs generated from 75 expert-annotated variables per event and filtered by manual checking and VLLM majority voting. The questions are organized by a three-level ability taxonomy (2 perception/reasoning roots, 10 sub-abilities such as Event Understanding and Risk & Hazard Assessment, 25 granular abilities) aligned with established scenario-description frameworks for automated driving. The second mechanism is GroupEval: each question is asked N times with the correct option rotated and the others shuffled, and a model passes only if it answers correctly every time, which neutralizes option-position bias and yields the reported sub-40% scores.

What would settle it

Re-run the benchmark feeding each model the full SHRP 2 context the original annotators had—speed, brake and accelerator signals, forward radar, and the additional camera views—while keeping the questions identical; if accuracy jumps well past 40%, the gap is an information gap. Alternatively, have human raters answer the same questions from only the same 5-second front clips; if humans also stall near 40%, the benchmark's ceiling is set by the clips, not the models.

Watch

Extended reading notes

Core claim

The central claim is that current vision language models have a large and systematic performance gap on safety-critical driving video understanding: no model among 14 state-of-the-art systems, ranging from 0.5B to 72B parameters, exceeds 40% accuracy on DVBench under GroupEval, the paper's consistency-based scoring that requires a model to pick the correct answer across several shuffled option orderings. The paper further claims that perception abilities (reading weather, road geometry, traffic control) are substantially stronger than reasoning abilities (event triggers, fault attribution, maneuver evaluation), and that the gap can be partially closed by fine-tuning on domain-specific question-answer pairs, with gains up to 10.94 percentage points. If true, the finding means that the fluent video commentary these models produce does not generalize to the high-stakes judgments an automated driving system must make.

Load-bearing premise

The benchmark assumes each question is answerable from a single 5-second, 432×324 front-camera clip alone; the paper's own quality-control section concedes that certain answers (e.g., lane counts, 'icy' surfaces) cannot be directly inferred from the video, so if the clips routinely omit the information needed, low accuracy measures missing data rather than failed reasoning.

Editorial extensions

If this is right

  • No current open-weight vision language model reliably performs the perception and reasoning an automated driving system needs in safety-critical moments; even the best model fails most questions when option position is randomized.
  • Reasoning about events—what triggered them, who was at fault, what the maneuver was worth—is markedly harder for these models than basic scene description, pointing to where training data is most lacking.
  • Single-trial evaluation overstates model ability by roughly 10 to 20 percentage points; consistent-answer evaluation should be the default for safety claims.
  • Domain-specific fine-tuning produces meaningful but partial recovery (up to +10.94 points, 43.59% relative), so closing the road-readiness gap requires targeted adaptation rather than scale alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sub-40% ceiling likely mixes capability failure with information loss, since the models see one 5-second front view at low resolution while the original human annotators had multiple camera views plus vehicle telemetry (speed, brake, radar); giving models that metadata would separate the two.
  • The same taxonomy used to grade models could seed a driving-specific instruction-tuning curriculum, transforming DVBench from an exam into a training set for safety-critical video reasoning.
  • A human baseline—people answering the same questions with only the same 5-second clips—would calibrate how much of the gap is genuinely about VLLMs, since some questions (lane counts, icy surfaces) are visually ambiguous even for people.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DVBench, a multiple-choice benchmark for safety-critical driving video understanding, built from SHRP2 crash and near-crash videos. It defines a three-level ability taxonomy (2 perception/reasoning categories, 10 L2 abilities, 25 L3 abilities), curates roughly 10,000 questions with expert-annotated ground truth, and pairs each question with a 5-second 432x324 front-camera clip. The authors evaluate 14 open-weights VLLMs using an individual single-trial protocol and a GroupEval repeated-trial protocol, report that no model exceeds 40% under GroupEval, and show accuracy gains after fine-tuning two Qwen2-VL models on DVBench data. The central claim is that current VLLMs cannot reliably perceive and reason about safety-critical driving events and are therefore not road-ready.

Significance. DVBench targets a genuine and under-served evaluation gap: safety-critical events are rare, existing driving VQA benchmarks are dominated by normal conditions or static frames, and the temporal/causal reasoning needed for crash and near-crash understanding is not systematically tested. The use of SHRP2 naturalistic driving data with expert annotations is a strong empirical basis, and the PEGASUS/NHTSA-inspired taxonomy gives the benchmark a defensible structure. The open-sourced toolbox and fine-tuned models are useful community assets, and the GroupEval repeated-trial protocol is a sensible idea for exposing option-position bias. If the construction issues discussed below are resolved, DVBench could become a standard resource for driving VLLM evaluation. As it stands, however, the headline accuracy claims are not yet supported because question selection is partly based on the evaluated models, the reported GroupEval numbers are pass rates rather than top-1 accuracy, and many retained items are not visually answerable from the supplied clip.

major comments (4)
  1. [Section 3.2 and Section 3.3] The benchmark construction uses the evaluated models to select its own items. Section 3.2 states that questions are filtered when 'all LLMs consistently answered either correctly or incorrectly,' and Section 3.3 describes a 'VLLM Majority Voting' step that removes unanimously correct or incorrect questions. Because the same model families are then used to report DVBench accuracy, the difficulty distribution is not independent of the measured systems. This makes the headline result a property of the filtering procedure as much as of the models. Please either remove this filter, apply it with a held-out model set, or report results with and without the filtered items.
  2. [Section 3.2, Section 3.3, Appendix A.2, Table 8] The central inference that low accuracy reflects failed perception and reasoning requires that the retained questions are answerable from the 5-second, 432x324 front-camera clip. Section 3.2 says only a 'critical 5-second segment' is extracted from each SHRP2 video, but the SHRP2 ground truth described in Section 3.1 uses four video views, vehicle network information, and forward radar. Retained question types such as Fault Analysis (401 questions), Event Severity (401), Maneuver Evaluation (200), and Vehicle Maneuvers (602) in Table 8 are not in general determinable from a single front-facing clip, and Figure 3 even shows a 'Who was at fault' example. Section 3.3 itself admits 'Certain answers cannot be directly inferred from the video.' The authors should remove or re-annotate such items so that the ground truth is visually verifiable from the model input, or demonstrate otherwise; otherwise the 'no model over 40%' result conflates missing information with failed reasoning.
  3. [Section 4.2, Table 2, Table 3, Abstract] GroupEval is defined in Section 4.2 as a pass rate: a question is passed only if the model answers correctly in all N trials with shuffled option positions. This is not top-1 accuracy, yet Table 3 labels such numbers as 'Top-1 accuracy,' Table 2 reports them in the same accuracy units as IndividualEval, and the abstract's 'no model achieving over 40% accuracy' refers to this pass rate. The metrics are not comparable: a per-trial accuracy of p corresponds to a pass probability of roughly p^N under independence, and pure position bias is penalized differently. Report IndividualEval and GroupEval separately, describe GroupEval as a consistency or pass rate, and avoid stating the headline result as an accuracy claim without qualification.
  4. [Section 4.6 and Table 5] The fine-tuning experiment does not state whether the 2,880 question-answer pairs used for training are disjoint from the evaluation set. If the models are fine-tuned and evaluated on the same DVBench items, the reported +5.24 and +10.94 percentage-point gains reflect memorization of the benchmark rather than adaptability to safety-critical driving understanding. Specify the train/test split and report fine-tuned performance on a held-out DVBench subset or on an external driving benchmark.
minor comments (5)
  1. [Table 8] The sum of the Level 1 counts is 10,263 (3,925 + 6,338), while the abstract, Section 3.2, and Table 1 state about 10,000; the discrepancy should be reconciled or the totals rounded consistently.
  2. [Table 6 vs. Tables 2-4] Table 6 lists 'LLaVA-One-Vision-5B,' but Tables 2-4 and Figure 1 refer to 'LLaVA-One-Vision-0.5B'; the parameter size should be consistent.
  3. [Figure 3] Figure 3 contains typos in the option text, including 'Sta b le flow,' 'Da y light,' and 'U nsta b le flow,' and its 'Who was at fault' example illustrates the visual-answerability concern raised in the major comments.
  4. [Section 4.5] The sentence 'no model surpassed 40% overall accuracy' should specify that this refers to the GroupEval pass rate, especially since the same section reports IndividualEval values above 40% for some models.
  5. [Appendix A.2] The definition of Event Severity says to 'infer from the video what is the severity of the crash,' but Section 3.3 lists lack of visual clarity as a known issue; please clarify which abilities were retained only when the ground truth is visually verifiable.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark items are filtered by the very VLLMs being evaluated, so the 'no model over 40%' result is partly an artifact of benchmark construction; the human-annotated ground truth keeps the benchmark from being fully self-definitional.

  1. fitted input called prediction [Section 3.3 (VLLM Majority Voting) and Section 3.2 (Ground Truth and Choice Selection)]
    "Specifically, several models generate responses for each sampled question, and those consistently yielding unanimous incorrect (or correct) answers are flagged as potentially misleading or overly simplistic. This consensus-based quality check filters out low-quality or ambiguous questions... Questions that were consistently answered incorrectly or correctly by all VLLMs were excluded, as they lack the variability required for effective benchmarking."

    The retained DVBench items are selected using the responses of the same class of models that the paper then evaluates. Unanimously-correct questions are removed, which lowers achievable scores, and unanimously-incorrect questions are also removed, which discards failures; the benchmark therefore preferentially retains questions where VLLMs give split votes. The headline claim 'no model achieving over 40% accuracy' is thus partly a property of this majority-vote filtering rule rather than an independent measure of safety-critical driving understanding. Because the ground-truth labels come from independent SHRP2 expert annotations, this is item-selection circularity rather than full self-derivation, but the central performance comparison is calibrated by the subjects being measured.

full rationale

DVBench's ground truth is independently sourced from SHRP2 expert annotation, so the benchmark is not self-definitional in its answers. The main circularity is in question selection: the paper explicitly uses VLLM majority voting to filter out questions that all evaluated models answer unanimously, and then reports those same models' accuracy on the remaining set as evidence of their limitations. This makes the reported performance gap partly an artifact of the filtering rule. The 'Lack of Visual Clarity' admission in Section 3.3 is a validity threat to the inference from low accuracy to poor reasoning, but it is not itself circularity, so it is not counted as a circular step. No load-bearing self-citation chain or imported uniqueness theorem was found; the only self-citation (ScVLM) is descriptive in Related Work. Overall, the central benchmark construction is partially circular through model-based item filtering, but the human-annotated ground truth and the diversity of remaining questions give the result independent content, warranting a moderate score of 4.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in the central benchmark claim. The fine-tuning step trains model weights, but those are outputs, not inputs, and the headline accuracy numbers do not depend on any fitted constant. The benchmark's difficulty is shaped instead by the post-hoc item filtering documented in the red flags. No new physical or theoretical entities are introduced.

assumptions (4)
  • domain assumption SHRP2 expert annotations are accurate and reliable ground truth for the multiple-choice answers.
    All answers are drawn from the 75-variable SHRP2 reduction protocol (Section 3.2); if those annotations are wrong, the benchmark's labels are wrong.
  • domain assumption The retained questions are visually answerable from a 5-second, 432x324 front-camera clip.
    The evaluation presents only front-view frames (Appendix prompt), yet some questions concern fault, severity, and maneuvers that may require metadata or multiple views; Section 3.3 itself notes that some answers cannot be directly inferred from the video.
  • ad hoc to paper Removing questions that all models answer correctly or incorrectly does not bias the measured capability gap.
    Section 3.3 excludes 'questions that were consistently answered incorrectly or correctly by all VLLMs' to keep discriminative power; this choice uses model outputs to shape the test and presupposes the resulting difficulty is still representative of real safety-critical understanding.
  • ad hoc to paper GroupEval pass rates can be reported as 'accuracy' comparable to standard single-trial accuracy.
    GroupEval requires all N shuffled trials correct, a joint probability, not top-1 accuracy; the paper's Table 2 labels it as such, which changes the chance baseline and the interpretation of 'no model over 40%.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding." pith.science (2026). https://pith.science/paper/B4NV64EN

@misc{pith2026250414526,
  author       = {Pith},
  title        = {Pith review of: Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4NV64EN}},
  note         = {Machine review of arXiv:2504.14526}
}
read the original abstract

Vision Large Language Models (VLLMs) have demonstrated impressive capabilities in general visual tasks such as image captioning and visual question answering. However, their effectiveness in specialized, safety-critical domains like autonomous driving remains largely unexplored. Autonomous driving systems require sophisticated scene understanding in complex environments, yet existing multimodal benchmarks primarily focus on normal driving conditions, failing to adequately assess VLLMs' performance in safety-critical scenarios. To address this, we introduce DVBench, a pioneering benchmark designed to evaluate the performance of VLLMs in understanding safety-critical driving videos. Built around a hierarchical ability taxonomy that aligns with widely adopted frameworks for describing driving scenarios used in assessing highly automated driving systems, DVBench features 10,000 multiple-choice questions with human-annotated ground-truth answers, enabling a comprehensive evaluation of VLLMs' capabilities in perception and reasoning. Experiments on 14 SOTA VLLMs, ranging from 0.5B to 72B parameters, reveal significant performance gaps, with no model achieving over 40% accuracy, highlighting critical limitations in understanding complex driving scenarios. To probe adaptability, we fine-tuned selected models using domain-specific data from DVBench, achieving accuracy gains ranging from 5.24 to 10.94 percentage points, with relative improvements of up to 43.59%. This improvement underscores the necessity of targeted adaptation to bridge the gap between general-purpose VLLMs and mission-critical driving applications. DVBench establishes an essential evaluation framework and research roadmap for developing VLLMs that meet the safety and robustness requirements for real-world autonomous systems. We released the benchmark toolbox and the fine-tuned model at: https://github.com/tong-zeng/DVBench.git.

Figures

Figures reproduced from arXiv: 2504.14526 by the authors.

Figure 1
Figure 1. Performance of nine representative Vision Large [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The DVBench Ability Hierarchy is structured across [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of L2 Ability Taxonomy questions: perception and reasoning. Perception abilities focus on low-level skills [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: GroupEval Strategy. In GroupEval, each question is tested multiple times, with the correct answer’s position changing [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Distribution of Ground-Truth Answers and Sample VLLM Predictions: In DVBench, there exist some questions that [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The system prompt for VLLM answer generation in DVBench. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The prompt template for zero-shot VLLM inference. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.

Reference graph

Works this paper leans on

55 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    G Audi, AG Volkswagen, et al. [n. d.]. The PEGASUS Method

    work page

  3. [3]

    Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939 (2024)

    arXiv 2024

  4. [4]

    Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie-Francine Moens. 2019. Talk2car: Taking control of your self-driving car. arXiv preprint arXiv:1909.10838 (2019)

    arXiv 2019

  5. [5]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv preprint arXiv:2405.21075 (2024)

    arXiv 2024

  6. [6]

    Xiaowei Gao, Pengxiang Li, xinke Jiang, James Haworth, Jonathan Cardoso-Silva, and Ming Li. 2023. DriveScenify: Boosting Driving Scene Understanding with Advanced Vision-Language Models. https://github.com/pixeli99/DSify

    work page 2023

  7. [7]

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chenming Zhang, Shuai Liu, and Long Chen. 2024. Drivemllm: A benchmark for spatial understanding with multimodal large language models in autonomous driving. arXiv preprint arXiv:2411.13112 (2024)

    arXiv 2024

  8. [8]

    Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jianbing Shen. 2024. Dme- driver: Integrating human decision logic and 3d scene perception in autonomous driving. arXiv preprint arXiv:2401.03641 (2024)

    arXiv 2024

Show all 55 references
  1. [9]

    Jonathan M Hankey, Miguel A Perez, and Julie A McClafferty. 2016. Description of the SHRP 2 naturalistic database and the crash, near-crash, and baseline data sets. Technical Report. Virginia Tech Transportation Institute

    2016

  2. [10]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  3. [11]

    Sandesh Jain, Surendrabikram Thapa, Kuan-Ting Chen, A Lynn Abbott, and Abhijit Sarkar. 2024. Semantic Understanding of Traffic Scenes with Large Vision Language Models. In 2024 IEEE Intelligent Vehicles Symposium (IV) . IEEE, 1580– 1587

    2024

  4. [12]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169 (2023)

    2023 arXiv

  5. [13]

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata

  6. [14]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    2024 arXiv

  7. [15]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 22195–22206

    2024

  8. [16]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2025. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision . Springer, 323–340

    2025

  9. [17]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    2023 arXiv

  10. [18]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    2024

  11. [19]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  12. [20]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2025. Mmbench: Is your multi-modal model an all-around player?. In European Conference on Computer Vision. Springer, 216–233

    2025

  13. [21]

    Yuhang Lu, Yichen Yao, Jiadong Tu, Jiangnan Shao, Yuexin Ma, and Xinge Zhu

  14. [22]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  15. [23]

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. 2023. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision . 1043–1052

    2023

  16. [24]

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. 2023. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415 (2023)

    2023 arXiv

  17. [25]

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. 2024. LingoQA: Visual question answering for autonomous driving. In European Conference on Computer Vision . S...

    2024

  18. [26]

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. 2025. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision . Springer, 292–308

    2025

  19. [27]

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang

  20. [28]

    Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, and Behzad Dariush. 2024. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. In Proceedings of the IEEE/CVF winter conference on applications ...

    2024

  21. [29]

    Liang Shi, Boyu Jiang, and Feng Guo. 2024. ScVLM: a Vision-Language Model for Driving Safety Critical Event Understanding. arXiv:2410.00982 (2024)

    2024 arXiv

  22. [30]

    Maged Shoman, Dongdong Wang, Armstrong Aboah, and Mohamed Abdel-Aty

  23. [31]

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. 2024. Driv- elm: Driving with graph visual question answering. In European Conference on Computer Vision. Springer, 256–274

    2024

  24. [32]

    In Proceedings of the AAAI Conference on Artificial Intelligence, Vol

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 4542–4550

  25. [33]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  26. [34]

    Eric Thorn, Shawn C Kimmel, Michelle Chaka, Booz Allen Hamilton, et al. 2018. A framework for automated driving system testable cases and scenarios . Technical Report. United States. Department of Transportation. National Highway Traffic Safety

    2018

  27. [35]

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. 2024. Drivevlm: The conver- gence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289 (2024)

    2024 arXiv

  28. [36]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

    Enhancing Traffic Safety with Parallel Dense Video Captioning for End- to-End Event Analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops . 7125–7133

  29. [37]

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. 2025. Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives. arXiv preprint arXiv:2501.04003 (2025)

    2025 arXiv

  30. [38]

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al

  31. [39]

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. 2020. Explainable object-induced action decision for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9523–9532

    2020

  32. [40]

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. 2024. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters (2024)

    2024

  33. [41]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024)

    2024 arXiv

  34. [42]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sig- moid loss for language image pre-training. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision . 11975–11986

    2023

  35. [43]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  36. [44]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  37. [45]

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994 (2024)

    2024 arXiv

  38. [50]

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-NeXT: A Strong Zero-shot Video Understanding Model. https://llava-vl.github.io/blog/2024-04-30-llava-next- video/

    2024

  39. [52]

    Snowy",

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36 (2023), 50117–50143. 9 A Details of Hierarchical Ability Taxonomy The DVBench Ability Tax...

    2023

  40. [53]

    Carefully analyze the objects, environment, and events within the video to understand its context and details

  41. [54]

    Select the correct answer from the provided options by identifying only the option letter (e.g., A, B, C, or D)

  42. [55]

    Based on your observations, select the best option that accurately addresses the question

    Respond solely with the correct option letter, ensuring precision in your choice based on the observed content. Based on your observations, select the best option that accurately addresses the question. System Prompt Figure 6: The system prompt for VLLM answer generation in DV...

    1924

  43. [2018]

    InProceedings of the European conference on computer vision (ECCV)

    Textual explanations for self-driving vehicles. InProceedings of the European conference on computer vision (ECCV) . 563–578

  44. [2020]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2446–2454

  45. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  46. [2023]

    arXiv preprint arXiv:2306.05424 (2023)

    Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    2023 arXiv

  47. [2024]

    arXiv preprint arXiv:2409.02914 (2024)

    Can LVLMs Obtain a Driver’s License? A Benchmark Towards Reliable AGI for Autonomous Driving. arXiv preprint arXiv:2409.02914 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.