{"id":"ab61b846-364b-46fa-a97d-fdcebd45a771","arxiv_id":"2601.14702","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new three-level driving benchmark shows that for current vision-language models, correct perception does not reliably translate into correct driving decisions.","lead":"This paper builds a 6,650-question benchmark that tests vision-language models on object recognition, scene understanding, and driving decisions, and finds that even models that perceive well often make poor decisions. The benchmark adds risk-level splits and an automated error-labeling tool to help researchers see where driving AI fails.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Correlation evidence for the central claim is only reported for Qwen 7B/72B, yet the paper concludes about 'current VLMs' broadly.","rationale":"The reader's weakest assumption was annotation ambiguity and lack of inter-annotator agreement. That is a legitimate foundational concern, but the correlation analysis being reported for only two models is a more immediate and concrete threat to the asserted scope of the central claim. Even granting perfect ground truth, the paper's empirical evidence for 'current VLMs struggle to translate perception into decisions' is drawn from a single model family. The similar-scene results provide some breadth, but they do not directly measure perception-to-decision translation. My proposed test is straightforward: the data already exist (the authors computed correlations for Qwen), so reporting them for all models would either confirm the pattern or reveal a generalization gap. This does not overturn the paper's value; it changes the strength and scope of the headline conclusion. I therefore recommend keeping the CONDITIONAL verdict, with the additional condition that per-model correlation evidence be made available or the claim be narrowed.","tokens_in":32813,"tokens_out":7318,"duration_ms":82737,"concrete_test":"Re-run the same per-image phi correlation (Pearson on binary correctness, over the common-image sets) for all evaluated models, especially GPT-4.1, Gemini-2.5-Pro, Llama-90B, Llava-72B, and Phi-6B, and report the full correlation matrices in an appendix. Then check whether the median Object–Decision correlation across models remains weak (|r| < 0.2). If a frontier model exhibits r > 0.3 (e.g., GPT-4.1 Object-1 vs Decision-1), the paper's broad 'current VLMs' claim is not supported and should be narrowed to the Qwen family or explicitly qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'current VLMs struggle to translate perceptual information into decisions' (§4.4) rests primarily on weak Pearson correlations between Object/Scene and Decision accuracies. But the correlation analysis presented in Figure 4 and Appendix D.3 (Figure 15) covers only Qwen2.5-VL-7B and Qwen2.5-VL-72B. Section 3.2 promises correlations 'for each model,' but no correlation matrices or summary statistics are shown for GPT-4.1, Gemini-2.5-Pro, Llama-90B, Llava-72B, Phi-6B, or the other evaluated models. The similar-scene robustness test (§4.5, Table 7) does cover nine models and shows joint-accuracy drops, but that test measures sensitivity to visually similar image pairs, not whether correct perception is being translated into correct decisions. Thus, the empirical support for the headline generalization is a two-model observation. If, for example, GPT-4.1 or Llama-90B show moderate Object–Decision correlations, the conclusion would need to be qualified to small/medium open-weight models or revised. The paper currently provides no way to evaluate this because the per-model correlation data are not reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Drive-P2D (referred to as AutoDriDM in the body), a benchmark for vision-language models in autonomous driving comprising 6,650 QA items over Object, Scene, and Decision levels built from nuScenes, KITTI, and BDD100K. All items are multiple-choice with objectively scored answers, and models are prompted to produce structured chain-of-thought reasoning. The authors evaluate a range of open- and closed-source VLMs under zero- and few-shot settings, report risk-stratified results, analyze Pearson correlations between task accuracies, test robustness on 60 visually similar scene pairs, study InternVL scaling, and train a 7B analyzer to tag reasoning error modes. The central claim is that current VLMs perform acceptably on object perception but fail to translate this into reliable decision-making, as evidenced by weak perception-decision correlations and drops in joint accuracy on similar-scene pairs, and that reasoning traces reveal systematic logical errors.","tokens_in":33127,"tokens_out":5475,"duration_ms":55724,"significance":"If the evidence were complete, the benchmark would fill a real gap: it couples perception and decision in a single progressive protocol, uses objective scoring instead of LLM-judged free-form outputs, and adds risk-stratified and similar-scene analyses. The objective answer format and the attempt to use reasoning traces for failure-mode analysis are useful methodological contributions for the driving-VLM community. However, the headline generalization about 'current VLMs' is currently supported by only a subset of the evaluated models, and the benchmark's validity depends on annotation quality that is not yet reported. The paper's contributions are promising and within scope for a computer-vision/robotics venue, but the empirical backbone needs substantial strengthening.","major_comments":[{"comment":"The central claim that 'current VLMs struggle to translate perceptual information into decisions' rests on the correlation analysis, but this analysis is shown only for Qwen2.5-VL-7B and 72B. Section 3.2 says correlations are computed 'for each model,' yet no matrices or summary statistics are reported for GPT-4.1, Gemini-2.5-Pro, Llama-90B, Llava-72B, Phi-6B, or the InternVL family. Please report per-model correlations (or at least the Object–Decision and Scene–Decision coefficients for all models) and qualify the conclusion accordingly, or the headline statement is a two-model observation. Also, Figure 15 states each cell is computed over the images common to the two tasks, but the number of common images is not reported; since Scene tasks exist only in BDD100K while Object/Decision tasks exist in all three datasets, different cells are based on different subsets, which complicates com","section":"§4.4, Fig. 4, Fig. 15, §3.2"},{"comment":"No inter-annotator agreement is reported despite the protocol in §3.2 of two independent annotators plus a third arbitrator. Because the benchmark uses only single front-facing images and the Limitations acknowledge the absence of temporal/multi-sensor context, some Scene-2 and Decision items may have no well-defined answer. Low model accuracies on Scene-2 (e.g., 0.89–46.44% in Table 1) could reflect question ambiguity rather than model deficiency. Report per-task agreement (Cohen's kappa or Fleiss' kappa), and preferably analyze robustness of the main results when contested items are excluded.","section":"§3.2, Limitations, Tables 1 and 6"},{"comment":"The explainability finding that logical reasoning errors, semantic feature omissions, and hallucinations dominate is based on Qwen 7B/72B only, and it is unclear whether Figure 6 was produced by the fine-tuned analyzer of §4.8 or by manual annotation. If the analyzer was used, the analysis is partly circular: the 7B analyzer was trained on 1,500 traces labeled by the authors with the same nine-category taxonomy, and its Exact Match is 65.36% with 23.13% Mismatch on only 180 evaluation traces. State explicitly how Figure 6 labels were obtained, provide a manual validation sample for a subset of traces, and restrict the conclusion to the models actually analyzed.","section":"§4.7–§4.8, Fig. 6, Table 2"},{"comment":"The similar-scene robustness test is the only multi-model evidence for the perception-to-decision link, but it has 60 pairs, uses a squared single-image-accuracy baseline that assumes equal per-image difficulty, and reports only star thresholds without confidence intervals. It should be clarified that the Decision-1 values in Table 7 are measured on the 60 similar pairs, not on the full benchmark (they are 10–15 points lower than the full-benchmark Decision-1). Please provide bootstrap confidence intervals or per-pair results, and state how the 60 pairs were selected from the similarity filtering.","section":"§4.5, Table 7, Appendix D.5"}],"minor_comments":[{"comment":"The abstract and title use 'Drive-P2D' while the body uses 'AutoDriDM'. This inconsistency should be resolved throughout the manuscript.","section":"Title and Abstract"},{"comment":"The figure contains a stray caption line 'Figure 1: Correlation matrix for Qwen2.5-VL-Instruct' inside the main Figure 4. Remove this artifact.","section":"Fig. 4"},{"comment":"The repetition protocol averages two runs, but no variance or confidence intervals are reported. For binary accuracies, especially with the small 60-pair test, reporting standard deviations or CIs would support comparisons.","section":"§C.3, Tables 1, 6, 7"},{"comment":"The caption calls the values 'Pearson ϕ correlation'; for binary indicators, the phi coefficient and Pearson correlation coincide, but the wording is confusing. Use one term consistently and define it.","section":"Fig. 15 caption"},{"comment":"Specify the exact base model and fine-tuning setup for 'Ours 7B' (e.g., Qwen2.5-VL-7B-Instruct), and state the training/evaluation split for the 1,500/180 traces.","section":"Table 2"},{"comment":"The text mentions 'Blue superscript stars' but the table does not show colored stars; align the table formatting with the description.","section":"Appendix D.5"}],"recommendation":"major_revision","confidential_remarks":"The title mismatch between the abstract (Drive-P2D) and the body (AutoDriDM) suggests a version-control or assembly issue that the editor may want to check. The benchmark has potential value, but the central empirical claim currently rests on two Qwen models and a small similar-scene test; the requested additions of per-model correlations, annotator agreement, and clarification of the explainability pipeline are necessary before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things first. The work behind the title is real: a progressive Object-Scene-Decision benchmark on nuScenes, KITTI, and BDD100K, with risk splits, 60 similar-scene pairs, and a fine-tuned 7B analyzer for reasoning-error tagging. But the arXiv title says Drive-P2D and the manuscript body calls it AutoDriDM from the abstract onward. That identity mismatch has to be resolved before anyone can cite it. Second, the central claim — current VLMs don't translate perception into decisions — is supported mainly by correlations for Qwen-7B and Qwen-72B, not for the other nine models. The similar-scene joint-accuracy test covers more models, but it tests consistency under visual similarity, not the perception-decision link. So the headline generalization is really a two-model observation. The authors should either report per-model correlation matrices or soften the conclusion.\n\nWhat is actually new: the combination of progressive task levels, risk-aware splits, and the similar-scene joint-accuracy test. The 60-pair test is small but a good idea; the low Scene-2 scores and weak correlations are consistent with prior work like Xie et al. 2025, but the benchmark gives the community a concrete instrument. The automated error tagging is useful, and the analyzer beats GPT-4.1 on their metric. Just recognize it is semi-circular: they define the taxonomy, label 1,500 traces themselves, train the analyzer, then use its outputs as evidence about error modes. That is fine for an exploratory analysis, but it should not be read as independent confirmation.\n\nThe soft spots are in proportion. No inter-annotator agreement is reported for the 6,650 ground-truth labels, and the benchmark uses only single front-facing images; some Decision questions may not have a well-defined correct answer without temporal context. That is a real fairness concern, not a fatal one. Ask for agreement statistics and perhaps a low-agreement flag. Only two runs are averaged, so model comparisons lack error bars. The scaling-behavior claim for InternVL-38B is presented as a figure without a numeric table; that is easy to fix. Data and code are not released despite the ethics statement saying they will be; fine for a preprint, but reviewers should ask for them.\n\nVerdict: don't desk reject. The benchmark is plausible, internally consistent, and the main empirical pattern — low Scene-2, weak correlations, joint accuracy below the squared baseline — is credible. A serious referee should engage; expect major revision on presentation and reproducibility.","headline":"A useful progressive benchmark for VLM driving evaluation, but the paper ships under the wrong title, over-generalizes a two-model correlation result, and needs reproducibility fixes before it can be cited.","tokens_in":33618,"tokens_out":2242,"would_cite":false,"duration_ms":27155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current vision-language models can describe road scenes but largely fail to turn that perception into safe driving decisions — a 6,650-question benchmark exposes the gap.","keywords":["vision-language models","autonomous driving","benchmark","perception-to-decision gap","driving decision-making","chain-of-thought reasoning","similar-scene robustness","error taxonomy"],"falsifier":"Re-annotate a random subsample of the 6,650 questions — especially the Decision and Scene-2 items and the 60 similar-scene pairs — using new drivers, and measure inter-annotator agreement. If agreement is low, or if the newly agreed labels reproduce the models' 'errors' as plausible alternatives, the weak perception–decision correlations would reflect question ambiguity rather than a capability gap. A second check: for each similar-scene pair, ask human drivers to state which visual difference justifies different answers; pairs where drivers cannot articulate such a difference are weak evidenc","tokens_in":32749,"feed_emoji":"🚗","tokens_out":7891,"duration_ms":83160,"temperature":0.7,"pith_summary":"Drive-P2D is a benchmark for testing whether vision-language models (VLMs) can do what an autonomous driver must: move from seeing an object to choosing a safe action. Built on 1,295 front-view road images and 6,650 multiple-choice questions, it chains three levels — object identification, scene understanding, and decision-making — so a model's answer at each stage can be checked against the others. The paper's central finding is that current VLMs, open-weight and commercial alike, perform better at object and scene perception than at decisions, and that the two barely correlate: knowing what is in the image does not predict whether the model picks the right action. On pairs of near-identical scenes, models also fail both images more often than chance would predict, which the authors read as evidence that decisions ride on superficial cues rather than causal reasoning. The paper argues that combining objective scoring with chain-of-thought reasoning analysis is necessary to expose this gap, and that ordinary scale alone will not close it.","feed_headline":"6,650-question benchmark shows driving AI perceives but can't decide","feed_subtitle":"Perception and decision scores barely correlate on real road scenes, so today's models are not ready to drive.","key_machinery":"The load-bearing tool is the progressive three-level protocol itself: six tasks arranged in a chain — Object-1 (key object), Object-2 (object state), Scene-1 (weather/illumination), Scene-2 (special scene factors), Decision-1 (optimal action), Decision-2 (risk of a suboptimal action) — with two metrics per level and the same image reused across tasks, so a model's failure can be localized to a stage. Around this core sit three instruments. A separated reasoning-and-answer protocol asks each model to emit a chain-of-thought rationale in <think> tags and a final answer in <answer> tags, so answers are scored objectively while reasoning is analyzed separately. A correlation analysis computes Pe","core_discovery":"The paper's claim, stated on its own terms, is that \"current VLMs struggle to translate perceptual information into decisions.\" The evidence comes in three forms. First, pairwise Pearson correlations among the six task scores (object, scene, decision) run mostly between −0.2 and 0.2; decision tasks correlate only mildly with object recognition and not with scene understanding, so a model's perception scores say little about its decision scores. Second, on 60 pairs of visually similar scenes, the joint accuracy — the chance a model answers both images in a pair correctly — falls below the squared baseline of the single-image accuracies, significantly so for several small models; the authors r","pith_inferences":["My inference: the similar-scene protocol could be turned into a direct causal test — alter one decision-relevant cue (brake light on or off, pedestrian present or not) between paired images and measure whether the decision flips; the paper does not run this intervention, but its design invites it.","My inference: the weak-coupling result, if it holds for single images, may not survive the introduction of temporal information; a fourth video-based level would test whether motion cues — the missing modality in this benchmark — are what connect perception to decision.","My inference: the paper never reports inter-annotator agreement for its two-expert-plus-arbitrator labeling, so a re-annotation study reporting agreement rates would establish how much of the measured perception–decision gap is model failure versus answer ambiguity.","My note: the manuscript's title, abstract, and running text use different names for the benchmark (Drive-P2D versus AutoDriDM); readers should confirm which name the released artifacts use."],"forward_implications":["If the weak coupling is real, a VLM's object and scene perception scores cannot be used as evidence of driving readiness; decision accuracy must be evaluated directly.","Scaling alone will not close the gap: larger models get more stable and less negatively coupled, but not more coupled, so integrating perception and decision requires architectural or training changes beyond model size.","Choice-only benchmarks overstate capability: correct final answers frequently come with flawed reasoning, so reasoning-trace analysis is needed to expose actual failure modes.","Few-shot prompting is an unreliable lever for decision tasks — it can bias the answer distribution or degrade performance — so example selection needs its own care.","Performance does not scale monotonically: one intermediate 38-billion-parameter model in a scaled family shows a pronounced drop caused by overly constrained reasoning, so evaluations should sample multiple sizes."],"fun_headline_variants":["Driving AI: perception scores don't predict decisions","Benchmark: VLMs see but can't act in driving scenarios","Perception and decision barely correlate in driving VLMs","New benchmark shows VLMs fail to link seeing and deciding","6,650 questions: driving VLMs struggle to decide from sight"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that every question has one correct, unambiguous answer, even though each item is judged from a single front-view image by two annotators (with a third arbitrator breaking ties) and the paper does not report how often annotators disagreed.","fun_headline_variants_meta":{"raw":{"variants":["Driving AI: perception scores don't predict decisions","Benchmark: VLMs see but can't act in driving scenarios","Perception and decision barely correlate in driving VLMs","New benchmark shows VLMs fail to link seeing and deciding","6,650 questions: driving VLMs struggle to decide from sight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2723,"prompt_tokens":727,"completion_tokens":1996,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1913}},"tokens_in":471,"tokens_out":1996,"duration_ms":14741,"temperature":1.0,"reasoning_tokens":1913,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:05:12.865447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subsample of the 6,650 questions — especially the Decision and Scene-2 items and the 60 similar-scene pairs — using new drivers, and measure inter-annotator agreement. If agreement is low, or if the newly agreed labels reproduce the models' 'errors' as plausible alternatives, the weak perception–decision correlations would reflect question ambiguity rather than a capability gap. A second check: for each similar-scene pair, ask human drivers to state which visual difference justifies different answers; pairs where drivers cannot articulate such a difference are weak evidenc","supporting_citations":[],"review_version":1}