{"id":"d43e48f9-689f-422f-be32-8f1e24e926a9","arxiv_id":"2505.05098","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"X-Driver combines LLaVA with chain-of-thought prompts to produce explainable driving decisions, reporting modest gains over UniAD on the Bench2Drive closed-loop benchmark.","lead":"The paper introduces X-Driver, an autonomous driving model that uses a vision-language model (LLaVA) with chain-of-thought reasoning to make step-by-step driving decisions in the CARLA simulator. It reports higher driving scores than a 2023 baseline, UniAD, but compares no other recent methods and omits error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The closed-loop 'SOTA' claim is supported only by a comparison to UniAD, a single 2023 baseline trained on fewer frames than X-Driver; no comparison to current MLLM or CoT baselines, so the central superiority claim is not yet established.","rationale":"The reader's identified weakest assumption (ViT vs VQ-VAE preserving distant traffic lights, Section 3.2) is a real design justification but it is not the most load-bearing point for the paper's central claim. Even if that encoding choice is exactly as described, the claim that X-Driver 'surpasses the current SOTA' could still fail because the only closed-loop evidence is confounded. Conversely, if the encoding assumption were wrong, the authors could swap encoders and still have a plausible pipeline; but if the evaluation is not fair, no conclusion about superiority follows. The paper presents no leaderboard position, no comparison with the MLLM baselines it cites, and no statistical reliability. In good faith, the architecture is plausible and the qualitative CoT example (Figure 4) supports interpretability to some degree, but quantitative evidence for the headline claim is thin. Therefore the appropriate outcome remains conditional: the central claim should not be accepted until a fair, multi-baseline, repeated-seed comparison is supplied. This does not change the reader's verdict, so 'UNCHANGED' is the verdict recommendation.","tokens_in":5774,"tokens_out":3772,"duration_ms":36973,"concrete_test":"Reproduce Table 4 under controlled conditions: train or fine-tune UniAD on the same 3M-frame corpus used for X-Driver, and additionally evaluate at least one current closed-loop MLLM/CoT baseline (e.g., DriveCoT, LMDrive, or SCE2DriveX) on the same 50-route and 220-route Bench2Drive splits with the official evaluator. Run each configuration with at least 3 seeds and report mean and standard deviation of Driving Score and Success Rate. If X-Driver does not exceed all baselines by more than the seed variance, the 'surpassing current SOTA' claim in the abstract is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's 'surpassing the current state-of-the-art (SOTA).' Table 4 (Section 4.3) is the only quantitative closed-loop evidence, and it compares X-Driver only with UniAD [4], a non-MLLM 2023 planner. Two confounds make the comparison unable to support the claim. First, UniAD is trained on 2M Bench2Drive frames while X-Driver (both with and without CoT) uses 3M frames; the driving score and success-rate gaps (e.g., 51.7 vs 45.9 and 18.1% vs 17.5% on bench2drive220) may reflect data quantity rather than the CoT architecture. Second, the paper's own related work cites more recent closed-loop MLLM/CoT methods (DriveCoT [15], LMDrive [11], EMMA [5], CoT-Drive [7]) that are not included in Table 4. Since the abstract makes an absolute 'current SOTA' claim, omitting all close baselines leaves the main scientific assertion unsupported. Additionally, no error bars, route-level breakdown, or statistical significance test is reported, so the observed ~2.6-point driving score and ~0.6-point success-rate differences are within plausible run-to-run variability. If a fair comparison on equal training data and against current MLLM baselines reverses or shrinks the gap, the central contribution collapses to an engineering recipe.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents X-Driver, a LLaVA-based vision-language model for closed-loop autonomous driving that uses chain-of-thought (CoT) reasoning to decompose driving into perception, prediction, and decision subtasks. The method employs a continuous ViT image encoder, autoregressive multimodal fusion, and supervised fine-tuning with CoT prompts. The authors report open-loop object detection and waypoint prediction results on Bench2Drive, and closed-loop CARLA results in which they claim to surpass the current state of the art (UniAD). The paper argues that CoT provides interpretability and improves closed-loop performance.","tokens_in":6192,"tokens_out":5603,"duration_ms":49097,"significance":"If the closed-loop results were properly supported, X-Driver would be a useful step toward interpretable, MLLM-based closed-loop driving, and the qualitative CoT-vs-auxiliary comparison in Fig. 4 is a suggestive demonstration of the behavioral effect of reasoning. However, the current evidence is insufficient: the SOTA claim rests on a single mismatched baseline, the open-loop waypoint statement is internally inconsistent, and a key encoding assumption is unverified. The paper does not provide code, model weights, or machine-checked proofs; its contribution is primarily an empirical architecture proposal, so the empirical support is the crux.","major_comments":[{"comment":"The central claim of surpassing current SOTA is supported only by a comparison against UniAD [4], a 2023 non-MLLM planner. Table 4 omits the recent closed-loop MLLM/CoT baselines cited in the related work (LMDrive [11], DriveCoT [15], EMMA [5], CoT-Drive [7]), and it compares with UniAD trained on 2M Bench2Drive frames while X-Driver uses 3M frames. The observed gaps (e.g., driving score 51.7 vs 45.9 on bench2drive220) could therefore be due to data quantity rather than the proposed architecture; no error bars or statistical significance tests are provided. To support the abstract's absolute SOTA claim, the authors should compare against current MLLM/CoT baselines under matched training data and report run-to-run variance.","section":"§4.3, Table 4"},{"comment":"The claim that the model 'achieves prediction accuracy of less than 1% for future trajectories' is incorrect: the reported ADE values are 0.679 m (0.5s), 0.837 m (1s), 1.128 m (2s), and 1.488 m (3s), and FDE is 2.472 m, none of which are percentages. This statement should be corrected with explicit units and a proper interpretation, preferably with a comparison to existing waypoint predictors, so that the open-loop evaluation supports rather than undermines the paper's claims.","section":"§4.2, Table 3 and text"},{"comment":"The paper asserts that continuous ViT encoding preserves distant traffic-light information that VQ-VAE encoding would lose, and that this preservation is important for CoT reasoning ('when a traffic light appears in the distant scene, the use of VQ-VAE encoding can result in the loss of crucial information'). No experiment, ablation, or quantitative evidence supports this assertion. Since the closed-loop performance and the claimed advantage of the architecture depend on this property, please provide a perception-level experiment or an ablation across encoders.","section":"§3.2"},{"comment":"The CoT row in Table 4 is labeled 'Ours(CoT: 3 million frames, auxiliary task version, split into 4 subtasks)', which conflates CoT with an auxiliary-task decomposition. The w/o CoT row is not described as having those auxiliary subtasks, so the comparison may not isolate CoT. To claim that CoT reasoning is the cause of improvement, the authors should clarify the exact training differences between the two variants and ensure that CoT is the only manipulated variable.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The abstract and introduction misuse the citation 'multi-modal large language models[2]' (bracket notation) and contain informal phrases such as 'And we already finished closed-loop experiments' in the contributions list; these should be cleaned up.","section":"Abstract and Section 1"},{"comment":"The sentence 'From this table, we can see that my object recognition method achieved a remarkably high accuracy' uses the first person 'my' and is unmeasured; please rephrase objectively.","section":"Section 4.2"},{"comment":"Figure cross-references are incomplete (e.g., 'As illustrated in 1', 'As illustrated in 3'), and some figure captions like 'Fig. 2. examples of Training Process' are truncated or lack periods.","section":"General presentation"},{"comment":"The column headers of Table 3 are ambiguous: it is unclear which numeric values are ADE and which are FDE under each time horizon; please restructure the table so that each metric has a clear column.","section":"Table 3"},{"comment":"The closed-loop evaluation reports only aggregate Driving Score and Success Rate; a per-route breakdown would help identify where the CoT version gains or loses, especially given the small aggregate differences.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a preliminary technical report rather than a polished journal submission. The core idea (CoT for closed-loop MLLM driving) is reasonable and the qualitative ablation is suggestive, but the quantitative evidence base is too narrow and contains internal inconsistencies. I recommend major revision with a clear requirement to expand the comparison set, correct the waypoint metrics, and verify the encoding claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward integration of LLaVA with DriveCoT-style chain-of-thought prompting, evaluated closed-loop on Bench2Drive. The core idea is sensible, but the headline claim of surpassing SOTA is not supported by the experiments: Table 4 compares only against UniAD, a 2023 non-MLLM planner trained on 2M frames, while the X-Driver runs use 3M frames. There are no error bars, no significance tests, and none of the MLLM/CoT baselines cited in the related work (DriveCoT, LMDrive, EMMA, CoT-Drive) appear in the comparison. The raw numbers may still indicate a real improvement, but the evidence as presented cannot bear the weight of the abstract's claim.\n\nWhat is new and worth credit: applying CoT prompting with a continuous ViT encoder, rather than VQ-VAE, inside a LLaVA-based closed-loop driver on Bench2Drive, and reporting open-loop 3D detection plus waypoint metrics alongside closed-loop driving scores. The four-subtask decomposition (objects, traffic lights, signs, lanes) is a reasonable instantiation of CoT. The qualitative example in Figure 4, where CoT avoids a pedestrian collision while the non-CoT version fails, is illustrative even if not statistically strong.\n\nSoft spots, in order of severity. The SOTA claim is the biggest, for the reasons above. Second, the waypoint results in Table 3 are described as \"less than 1%\" accuracy, but ADE/FDE values like 0.679 and 2.472 are not percentages; that sentence is wrong. Third, the assertion in Section 3.2 that continuous ViT encoding preserves distant traffic lights better than VQ-VAE is plausible but unverified, and no ablation shows it matters for closed-loop performance. Fourth, Table 4 labels the CoT version as \"auxiliary task version,\" which is confusing given the paper describes CoT as the main mechanism. Finally, no code or data release is mentioned, which limits reproducibility.\n\nThis is not a groundbreaking paper, and the citation pattern is fine — they cite the relevant prior work, including the CoT driving papers they build on. But the evaluation is not yet at the level where I would trust the central claim. Who is this for? Researchers working on MLLM-based driving who want a data point that CoT can help closed-loop performance; they should read it cautiously. I would send it to peer review because the question is meaningful and the evidence can be improved with proper baselines, error bars, and equal training data. If the authors do that, the result might still hold, but as written the contribution is an engineering recipe, not a demonstrated SOTA.\n\nRecommendation: major revision, with a request to rerun against current MLLM baselines under matched training conditions and to fix the waypoint percentage error.","headline":"A plausible but under-evidenced LLaVA+CoT driving pipeline claims SOTA on Bench2Drive, yet the table only compares against UniAD with mismatched training frames and no error bars.","tokens_in":6567,"tokens_out":2293,"would_cite":false,"duration_ms":24185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model that reasons step by step outperforms prior closed-loop driving systems while explaining its decisions.","keywords":["autonomous driving","vision-language models","chain-of-thought reasoning","closed-loop evaluation","CARLA simulation","interpretability","3D object detection","waypoint prediction"],"falsifier":"Run X-Driver on the same Bench2Drive routes with a VQ-VAE-discrete encoding of the image stream while keeping all chain-of-thought prompts and training data identical; if success on routes that require stopping at distant red lights does not drop, the paper's claimed information-preservation mechanism is not load-bearing.","tokens_in":5602,"feed_emoji":"🚗","tokens_out":8658,"duration_ms":74445,"temperature":0.7,"pith_summary":"In closed-loop driving, the car's own actions produce the next observation, so a policy must both perceive and act robustly. X-Driver claims that a vision-language model fine-tuned with chain-of-thought (step-by-step) reasoning can do this better than the current state of the art: on the Bench2Drive benchmark, the reasoning version scores higher on driving score and success rate than UniAD, and it can state why it brakes, turns, or accelerates. The paper argues that splitting the task into explicit perception subtasks—objects, traffic lights, traffic signs, lanes—before predicting waypoints reduces errors and makes decisions auditable. If the claim is right, end-to-end autonomous driving does not have to choose between explainability and closed-loop performance.","feed_headline":"Step-by-step driving beats prior closed-loop state of the art","feed_subtitle":"X-Driver's step-by-step reasoning reaches 24.0% success on Bench2Drive, up from 22.0% for UniAD","key_machinery":"The core mechanism is a LLaVA-style multimodal language model—a ViT image encoder aligned with a large language model—fine-tuned so that perception, prediction, and control are generated as one autoregressive sequence. Rather than tokenizing images with VQ-VAE, the paper feeds the model a continuous low-dimensional ViT feature map, arguing that this preserves small distant objects such as traffic lights. Chain-of-thought prompting decomposes the driving task into four subtasks (object detection, traffic light status, traffic sign recognition, lane detection), and the model outputs intermediate reasoning tokens before emitting waypoints and a driving command. The autoregressive factorization, $P(Y \\mid (T, I)) = \\prod_{t} P(y_t \\mid y_{<t}, (T, I))$, lets all of these outputs share one probability model, which is what makes the reasoning and the decision jointly trainable and jointly interpretable.","core_discovery":"The paper's central claim is that adding chain-of-thought supervision to a vision-language model yields a closed-loop driving policy that outperforms the current SOTA, UniAD, on the Bench2Drive benchmark while remaining explainable. In the closed-loop experiments, the CoT version achieves a driving score of 51.7 and a success rate of 18.1% on the 220-scenario split, and 57.8 and 24.0% on the 50-scenario split, compared with UniAD's 45.9/17.5% and 49.1/22.0%. In open-loop tests, it reports a 3D object-recognition IoU of 0.724 at sample level and 0.806 at box level, and waypoint errors of ADE 0.679 m at 0.5 s rising to 2.472 m at 3 s. The interpretation is that explicit step-by-step reasoning mitigates the hallucinations that otherwise cause MLLM drivers to misread scenes or miss hazards.","pith_inferences":["The headline comparison trains X-Driver on 3 million frames and UniAD on 2 million, so part of the margin may be attributable to data scale rather than the CoT mechanism; a same-data ablation would isolate the reasoning contribution.","If the continuous-encoding claim survives controlled tests, it suggests a design rule for MLLM driving systems: avoid discrete visual tokens whenever distant small objects such as traffic lights or pedestrians are safety-critical.","The four-subtask decomposition creates a natural error audit: log each reasoning stage and see whether collisions trace to perception failures, rule-misreading, or trajectory errors, which could target future data collection."],"forward_implications":["On the 220-route Bench2Drive split, the CoT system scores 51.7 on the driving score and 18.1% success, ahead of UniAD's 45.9 and 17.5%.","On the 50-route split, it reaches 57.8 and 24.0%, against UniAD's 49.1 and 22.0%, so the improvement persists when the evaluation is shorter.","Because the model emits intermediate reasoning tokens, every braking or turning decision can be inspected as a chain of perceived objects, traffic-light states, signs, and lanes before the final waypoint.","The autoregressive formulation means perception, prediction, and control share a single sequence, so the same framework can be extended to other instruction-following tasks without changing the model architecture."],"supporting_citations":[{"why":"Supplies the Bench2Drive benchmark and its closed-loop evaluation protocol, the source of all reported driving scores and success rates.","marker":"[6]"},{"why":"Introduces chain-of-thought reasoning for end-to-end driving, the prompting idea that X-Driver adapts.","marker":"[15]"},{"why":"Provides the LLaVA base model that X-Driver supervises and fine-tunes.","marker":"[8]"},{"why":"Defines UniAD, the planning-oriented SOTA baseline that X-Driver compares against.","marker":"[4]"},{"why":"Supplies the ViT/CLIP image encoder whose continuous features are used instead of VQ-VAE tokens.","marker":"[10]"},{"why":"Provides the VQ-VAE discrete representation that the paper argues loses distant traffic-light information.","marker":"[12]"}],"fun_headline_variants":["Chain-of-thought reasoning lifts driving success past SOTA","Explainable X-Driver beats UniAD in closed-loop tests","CoT boosts closed-loop driving: 24% success on Bench2Drive","Step-by-step vision-language driving outperforms prior methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed closed-loop advantage rests on the untested assumption that a continuous ViT image encoder preserves distant traffic-light information that a VQ-VAE encoder would lose, so the chain-of-thought reasoning can still detect red lights far ahead.","fun_headline_variants_meta":{"raw":{"variants":["Chain-of-thought reasoning lifts driving success past SOTA","Explainable X-Driver beats UniAD in closed-loop tests","CoT boosts closed-loop driving: 24% success on Bench2Drive","Step-by-step vision-language driving outperforms prior methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1538,"prompt_tokens":925,"completion_tokens":613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":542}},"tokens_in":541,"tokens_out":613,"duration_ms":6133,"temperature":1.0,"reasoning_tokens":542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:12:25.283502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run X-Driver on the same Bench2Drive routes with a VQ-VAE-discrete encoding of the image stream while keeping all chain-of-thought prompts and training data identical; if success on routes that require stopping at distant red lights does not drop, the paper's claimed information-preservation mechanism is not load-bearing.","supporting_citations":[],"review_version":1}