{"id":"ab5a5807-5fcc-45b8-a24d-a1e16c8cc1b9","arxiv_id":"2504.14526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DVBench introduces 10,000 expert-annotated questions on crash and near-crash driving videos and reports that no tested vision LLM exceeds 40 percent accuracy under its strict GroupEval scoring.","lead":"A new benchmark, DVBench, tests vision language models on 10,000 multiple-choice questions about safety-critical driving videos from real crash and near-crash events. It finds all 14 tested models score below 40 percent, but the way difficulty and accuracy are measured makes that headline hard to trust.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim presumes retained questions are answerable from the 5-second front-camera clip; the paper's own Section 3.3 admits they are not, so low scores may reflect missing information rather than failed reasoning.","rationale":"The reader's weakest assumption identifies exactly the concern I consider most load-bearing: the benchmark's validity depends on retained questions being visually answerable from the single front-camera clip. The paper itself, in Section 3.3, documents that this condition fails for at least some questions, and the filtering procedure described there does not remove or label them. This is more central than the GroupEval scoring issue or the post-hoc consensus filtering, because if a meaningful fraction of retained items are unanswerable from the provided video, then low VLLM accuracy is not evidence about video understanding at all. The GroupEval issue is about how scores are labeled and interpreted; the consensus-filtering issue is about benchmark difficulty being partly constructed by the evaluated models. Both are real, but the visual-answerability issue threatens construct validity of the central measure. My proposed test is a direct check: if video-only human experts cannot agree with the SHRP2 ground truth on the retained items, then those items should be separated into 'visually grounded' and 'annotation-dependent' subsets, and the road-readiness claim should be restricted to the former. The dataset remains valuable either way, so the reader's CONDITIONAL verdict is appropriate; I would not change it. The paper's other contributions, including the taxonomy, the 10,000-item MCQ bank, and the reproducible evaluation toolbox, are independent merits that do not depend on the contested headline interpretation.","tokens_in":19171,"tokens_out":3949,"duration_ms":39812,"concrete_test":"Take a stratified random sample of 200 retained DVBench questions, covering all 25 L3 abilities, and have three trained SHRP2-style annotators answer each question using only the provided 5-second 432x324 front-camera clip, with no SHRP2 metadata or annotation context. Compute per-question and per-ability agreement between video-only human answers and the published ground truth. If video-only human accuracy on reasoning abilities such as Causal & Responsibility, Event Understanding, or Behavior & Maneuver Analysis is substantially below the same annotators' accuracy when given the full SHRP2 annotation context, those questions are not visually grounded, and the headline performance gap should be recomputed on the visually grounded subset only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference is that low VLLM accuracy on DVBench demonstrates that current VLLMs cannot reliably perceive and reason about safety-critical driving video. For that inference to hold, every retained question must be answerable, at least in principle, from the 5-second 432x324 front-camera clip shown to the model. Section 3.2 says only 'a critical 5-second segment' from each SHRP2 video was extracted, and Section 3.3 explicitly lists 'Lack of Visual Clarity. Certain answers cannot be directly inferred from the video.' The retained set nevertheless includes items such as 'Who was at fault' (Figure 3) and, per Appendix A.2, Event Severity, Vehicle Maneuvers, Maneuver Evaluation, and Fault Analysis. Those ground-truth labels come from SHRP2's 75-variable expert annotation, which used four camera views, vehicle network data, and forward radar; the front-only clip may not contain the evidence needed to determine them. The VLLM majority-voting filter removes questions that all evaluated models answer unanimously, but it does not establish visual answerability. Thus a model can fail these items because the information is absent from the provided input, not because it cannot reason about visible driving scenes. This directly undercuts the headline 'no model over 40%, therefore not road-ready.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DVBench, a multiple-choice benchmark for safety-critical driving video understanding, built from SHRP2 crash and near-crash videos. It defines a three-level ability taxonomy (2 perception/reasoning categories, 10 L2 abilities, 25 L3 abilities), curates roughly 10,000 questions with expert-annotated ground truth, and pairs each question with a 5-second 432x324 front-camera clip. The authors evaluate 14 open-weights VLLMs using an individual single-trial protocol and a GroupEval repeated-trial protocol, report that no model exceeds 40% under GroupEval, and show accuracy gains after fine-tuning two Qwen2-VL models on DVBench data. The central claim is that current VLLMs cannot reliably perceive and reason about safety-critical driving events and are therefore not road-ready.","tokens_in":19347,"tokens_out":10544,"duration_ms":93385,"significance":"DVBench targets a genuine and under-served evaluation gap: safety-critical events are rare, existing driving VQA benchmarks are dominated by normal conditions or static frames, and the temporal/causal reasoning needed for crash and near-crash understanding is not systematically tested. The use of SHRP2 naturalistic driving data with expert annotations is a strong empirical basis, and the PEGASUS/NHTSA-inspired taxonomy gives the benchmark a defensible structure. The open-sourced toolbox and fine-tuned models are useful community assets, and the GroupEval repeated-trial protocol is a sensible idea for exposing option-position bias. If the construction issues discussed below are resolved, DVBench could become a standard resource for driving VLLM evaluation. As it stands, however, the headline accuracy claims are not yet supported because question selection is partly based on the evaluated models, the reported GroupEval numbers are pass rates rather than top-1 accuracy, and many retained items are not visually answerable from the supplied clip.","major_comments":[{"comment":"The benchmark construction uses the evaluated models to select its own items. Section 3.2 states that questions are filtered when 'all LLMs consistently answered either correctly or incorrectly,' and Section 3.3 describes a 'VLLM Majority Voting' step that removes unanimously correct or incorrect questions. Because the same model families are then used to report DVBench accuracy, the difficulty distribution is not independent of the measured systems. This makes the headline result a property of the filtering procedure as much as of the models. Please either remove this filter, apply it with a held-out model set, or report results with and without the filtered items.","section":"Section 3.2 and Section 3.3"},{"comment":"The central inference that low accuracy reflects failed perception and reasoning requires that the retained questions are answerable from the 5-second, 432x324 front-camera clip. Section 3.2 says only a 'critical 5-second segment' is extracted from each SHRP2 video, but the SHRP2 ground truth described in Section 3.1 uses four video views, vehicle network information, and forward radar. Retained question types such as Fault Analysis (401 questions), Event Severity (401), Maneuver Evaluation (200), and Vehicle Maneuvers (602) in Table 8 are not in general determinable from a single front-facing clip, and Figure 3 even shows a 'Who was at fault' example. Section 3.3 itself admits 'Certain answers cannot be directly inferred from the video.' The authors should remove or re-annotate such items so that the ground truth is visually verifiable from the model input, or demonstrate otherwise; otherwise the 'no model over 40%' result conflates missing information with failed reasoning.","section":"Section 3.2, Section 3.3, Appendix A.2, Table 8"},{"comment":"GroupEval is defined in Section 4.2 as a pass rate: a question is passed only if the model answers correctly in all N trials with shuffled option positions. This is not top-1 accuracy, yet Table 3 labels such numbers as 'Top-1 accuracy,' Table 2 reports them in the same accuracy units as IndividualEval, and the abstract's 'no model achieving over 40% accuracy' refers to this pass rate. The metrics are not comparable: a per-trial accuracy of p corresponds to a pass probability of roughly p^N under independence, and pure position bias is penalized differently. Report IndividualEval and GroupEval separately, describe GroupEval as a consistency or pass rate, and avoid stating the headline result as an accuracy claim without qualification.","section":"Section 4.2, Table 2, Table 3, Abstract"},{"comment":"The fine-tuning experiment does not state whether the 2,880 question-answer pairs used for training are disjoint from the evaluation set. If the models are fine-tuned and evaluated on the same DVBench items, the reported +5.24 and +10.94 percentage-point gains reflect memorization of the benchmark rather than adaptability to safety-critical driving understanding. Specify the train/test split and report fine-tuned performance on a held-out DVBench subset or on an external driving benchmark.","section":"Section 4.6 and Table 5"}],"minor_comments":[{"comment":"The sum of the Level 1 counts is 10,263 (3,925 + 6,338), while the abstract, Section 3.2, and Table 1 state about 10,000; the discrepancy should be reconciled or the totals rounded consistently.","section":"Table 8"},{"comment":"Table 6 lists 'LLaVA-One-Vision-5B,' but Tables 2-4 and Figure 1 refer to 'LLaVA-One-Vision-0.5B'; the parameter size should be consistent.","section":"Table 6 vs. Tables 2-4"},{"comment":"Figure 3 contains typos in the option text, including 'Sta b le flow,' 'Da y light,' and 'U nsta b le flow,' and its 'Who was at fault' example illustrates the visual-answerability concern raised in the major comments.","section":"Figure 3"},{"comment":"The sentence 'no model surpassed 40% overall accuracy' should specify that this refers to the GroupEval pass rate, especially since the same section reports IndividualEval values above 40% for some models.","section":"Section 4.5"},{"comment":"The definition of Event Severity says to 'infer from the video what is the severity of the crash,' but Section 3.3 lists lack of visual clarity as a known issue; please clarify which abilities were retained only when the ground truth is visually verifiable.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The underlying data and taxonomy are valuable, and the paper addresses an important gap in driving VLLM evaluation. The most serious issue is that the benchmark's item filtering uses the evaluated models and that many retained questions may not be answerable from the provided clip; both are fixable but require rerunning or re-annotating parts of the benchmark. I would encourage the editor to request a revision rather than reject, but the current headline result should not be cited as evidence of VLLM limitations until these points are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on DVBench. The dataset is a genuine contribution: the first MCQ benchmark drawn from SHRP2 crash and near-crash videos, with 10k human-annotated questions mapped to a PEGASUS/NHTSA-aligned taxonomy. The evaluation of 14 VLLMs and the fine-tuning experiments are clearly described. If the benchmark is properly cleaned and documented, it could become a standard test for safety-critical driving VQA.\n\nThe soft spots are real and they hit the central claim. First, the item-selection process uses the models under test. Section 3.2 removes questions that all models answer correctly or incorrectly, and Section 3.3's VLLM majority voting filters out items where models are unanimous. That means the difficulty distribution is selected using the models being measured. This is fixable by releasing filtering logs and reporting results on the full item pool, but as it stands it's a form of circularity.\n\nSecond, Table 2 labels GroupEval as 'Top-1 accuracy,' but GroupEval requires all-N correct across shuffled trials. The chance baseline for that is far below standard top-1, so the absolute numbers are not comparable to other benchmarks. The drop from IndividualEval to GroupEval is mostly a stricter scoring rule, not evidence of a different cognitive failure.\n\nThird—and this is the one that matters most—the paper's own Section 3.3 admits that 'certain answers cannot be directly inferred from the video.' Yet the retained set includes Fault Analysis, Event Severity, Vehicle Maneuvers, and similar items whose ground truth in SHRP2 came from four camera views, vehicle network data, and forward radar. The models only see a 5-second, 432x324 front-camera clip. So low accuracy on those items may reflect missing information, not poor reasoning. The stress-test note is right: this undermines the headline 'no model over 40%, therefore not road-ready.'\n\nNone of this kills the resource. The authors should separate visually-grounded questions from annotation-only questions, report standard accuracy with error bars, and document what information each question actually requires. They also should stop calling GroupEval scores 'top-1 accuracy.'\n\nWho should read this? Anyone building driving VQA benchmarks or using VLLMs for safety-critical perception. It deserves a serious referee—the idea is good and the execution is fixable—but the current version's headline numbers should not be cited without caveats.","headline":"A valuable benchmark whose headline numbers are undercut by model-dependent item selection and questions that aren't visually answerable from the clips; the dataset is worth engaging with, the performance claims need major caveats.","tokens_in":19944,"tokens_out":2282,"would_cite":true,"duration_ms":18729,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that state-of-the-art vision language models cannot yet be trusted in safety-critical driving scenes: across 14 models up to 72 billion parameters, none scored above 40% accuracy on DVBench, a new 10,000-question…","keywords":["DVBench","vision language models","autonomous driving safety","safety-critical events","driving video understanding","benchmark","naturalistic driving data","multiple-choice question answering"],"falsifier":"Re-run the benchmark feeding each model the full SHRP 2 context the original annotators had—speed, brake and accelerator signals, forward radar, and the additional camera views—while keeping the questions identical; if accuracy jumps well past 40%, the gap is an information gap. Alternatively, have human raters answer the same questions from only the same 5-second front clips; if humans also stall near 40%, the benchmark's ceiling is set by the clips, not the models.","tokens_in":18857,"feed_emoji":"🚗","tokens_out":5034,"duration_ms":40242,"temperature":0.7,"pith_summary":"The paper sets out to answer a practical question: can today's vision language models—systems that watch video and answer questions about it—understand the moments that matter most in driving, namely crashes and near-crashes? To find out, it builds DVBench, a 10,000-question multiple-choice benchmark drawn from real naturalistic driving footage, in which every question targets the perceptual and reasoning tasks an automated driving system would face. Testing 14 open models, the paper finds that none scores above 40% accuracy under its stricter consistent-answer evaluation, with reasoning tasks such as fault analysis and event-trigger identification faring worse than perception. The paper concludes that general-purpose vision language models are not road-ready for safety-critical driving, and shows that fine-tuning on domain data lifts accuracy by 5.24 to 10.94 percentage points, indicating targeted adaptation is necessary.","feed_headline":"No vision LLM tops 40% on crash-scene video questions","feed_subtitle":"Built from real crash and near-crash dashcam footage, DVBench shows even 72B-parameter models fall short of driving-grade reliability.","key_machinery":"The load-bearing object is DVBench itself: roughly 10,000 multiple-choice questions, each built from a five-second, 432×324 front-camera clip extracted from naturalistic driving video at the annotated event time, with question-answer pairs generated from 75 expert-annotated variables per event and filtered by manual checking and VLLM majority voting. The questions are organized by a three-level ability taxonomy (2 perception/reasoning roots, 10 sub-abilities such as Event Understanding and Risk & Hazard Assessment, 25 granular abilities) aligned with established scenario-description frameworks for automated driving. The second mechanism is GroupEval: each question is asked N times with the correct option rotated and the others shuffled, and a model passes only if it answers correctly every time, which neutralizes option-position bias and yields the reported sub-40% scores.","core_discovery":"The central claim is that current vision language models have a large and systematic performance gap on safety-critical driving video understanding: no model among 14 state-of-the-art systems, ranging from 0.5B to 72B parameters, exceeds 40% accuracy on DVBench under GroupEval, the paper's consistency-based scoring that requires a model to pick the correct answer across several shuffled option orderings. The paper further claims that perception abilities (reading weather, road geometry, traffic control) are substantially stronger than reasoning abilities (event triggers, fault attribution, maneuver evaluation), and that the gap can be partially closed by fine-tuning on domain-specific question-answer pairs, with gains up to 10.94 percentage points. If true, the finding means that the fluent video commentary these models produce does not generalize to the high-stakes judgments an automated driving system must make.","pith_inferences":["The sub-40% ceiling likely mixes capability failure with information loss, since the models see one 5-second front view at low resolution while the original human annotators had multiple camera views plus vehicle telemetry (speed, brake, radar); giving models that metadata would separate the two.","The same taxonomy used to grade models could seed a driving-specific instruction-tuning curriculum, transforming DVBench from an exam into a training set for safety-critical video reasoning.","A human baseline—people answering the same questions with only the same 5-second clips—would calibrate how much of the gap is genuinely about VLLMs, since some questions (lane counts, icy surfaces) are visually ambiguous even for people."],"forward_implications":["No current open-weight vision language model reliably performs the perception and reasoning an automated driving system needs in safety-critical moments; even the best model fails most questions when option position is randomized.","Reasoning about events—what triggered them, who was at fault, what the maneuver was worth—is markedly harder for these models than basic scene description, pointing to where training data is most lacking.","Single-trial evaluation overstates model ability by roughly 10 to 20 percentage points; consistent-answer evaluation should be the default for safety claims.","Domain-specific fine-tuning produces meaningful but partial recovery (up to +10.94 points, 43.59% relative), so closing the road-readiness gap requires targeted adaptation rather than scale alone."],"supporting_citations":[{"why":"Supplies the entire video corpus: 4,300 years of naturalistic driving data with crash, near-crash, and baseline events and 75 expert-annotated variables per event, from which DVBench's clips and ground-truth answers are built.","marker":"[9]"},{"why":"Grounds the ability taxonomy in the PEGASUS scenario-based testing method for automated driving systems.","marker":"[2]"},{"why":"Grounds the ability taxonomy in the NHTSA framework for automated driving system testable cases and scenarios.","marker":"[34]"},{"why":"Provides the prior reliability-, data-, and metric-focused study of VLMs in driving that DVBench extends by shifting to safety-critical crash and near-crash events.","marker":"[37]"},{"why":"Exemplifies existing driving QA benchmarks that DVBench contrasts against, lacking temporal-spatial understanding of dynamic crash scenarios.","marker":"[27]"},{"why":"Supplies five of the evaluated models (LLaVA-One-Vision 0.5B/7B/72B), the family that attains the top overall GroupEval accuracy.","marker":"[14]"},{"why":"Supplies the Qwen2-VL models whose 2B and 7B variants are fine-tuned on DVBench data, producing the largest reported gains.","marker":"[36]"},{"why":"Documents premise-order sensitivity in LLMs, motivating GroupEval's rotated-option design to counter position bias.","marker":"[3]"}],"fun_headline_variants":["Best vision LLM fails driving safety test under 40%","All 14 vision LLMs score below 40% on driving videos","Driving benchmark exposes critical reasoning gap in vision LLMs","Fine-tuning lifts vision LLM driving accuracy by up to 43.59%","Even 72B vision LLMs fail crash-scene video reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes each question is answerable from a single 5-second, 432×324 front-camera clip alone; the paper's own quality-control section concedes that certain answers (e.g., lane counts, 'icy' surfaces) cannot be directly inferred from the video, so if the clips routinely omit the information needed, low accuracy measures missing data rather than failed reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Best vision LLM fails driving safety test under 40%","All 14 vision LLMs score below 40% on driving videos","Driving benchmark exposes critical reasoning gap in vision LLMs","Fine-tuning lifts vision LLM driving accuracy by up to 43.59%","Even 72B vision LLMs fail crash-scene video reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":3016,"prompt_tokens":1039,"completion_tokens":1977,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1885}},"tokens_in":655,"tokens_out":1977,"duration_ms":14071,"temperature":1.0,"reasoning_tokens":1885,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:46:23.487028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark feeding each model the full SHRP 2 context the original annotators had—speed, brake and accelerator signals, forward radar, and the additional camera views—while keeping the questions identical; if accuracy jumps well past 40%, the gap is an information gap. Alternatively, have human raters answer the same questions from only the same 5-second front clips; if humans also stall near 40%, the benchmark's ceiling is set by the clips, not the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the entire video corpus: 4,300 years of naturalistic driving data with crash, near-crash, and baseline events and 75 expert-annotated variables per event, from which DVBench's clips and ground-truth answers are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the ability taxonomy in the PEGASUS scenario-based testing method for automated driving systems."},{"cited_title":"In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops","cited_arxiv_id":null,"evidence_quote":"Supplies the Qwen2-VL models whose 2B and 7B variants are fine-tuned on DVBench data, producing the largest reported gains."}],"review_version":1}