{"id":"768c405a-5928-4d50-927a-a063a6843af1","arxiv_id":"2507.20342","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid planner that adds a vision-language model reading raw multi-view images to a real-time trajectory planner achieves modest but consistent gains on nuPlan, with an adaptive gate that cuts VLM inference frequency.","lead":"This paper describes VLMPlanner, a self-driving motion planning system that lets a vision-language model look at live camera images and give advice to a fast learning-based trajectory planner. The system also uses a context-adaptive gate to decide when the language model should run, saving computation in simple scenes and reasoning more in complex ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed-loop SOTA claim rests on a non-standard 8-second, first-frame-only protocol, and the paper does not show that the reported baseline scores were obtained under the same protocol or scenario subset.","rationale":"The reader's weakest assumption correctly identifies the closed-loop evaluation protocol as the load-bearing point. My stress-test agrees but sharpens it: the issue is not only whether the compressed protocol tracks the standard protocol, but whether the published baseline scores in Tables 2-3 were obtained under the same protocol and scenario subset at all. The manuscript's wording describes the compressed protocol for the proposed method and justifies it by NavSim, but it never explicitly states that every baseline in the comparison tables was rerun under these conditions. If the baselines were taken from prior papers' standard-protocol results, the comparison is apples-to-oranges in two ways: the scenario subset (Hard20 selection) and the first-frame-only execution. The claimed margins are small, so this concern is material. I do not think it changes the verdict to REJECT: the open-loop gains, though small, are in the right direction, the architecture is coherent, and the missing evidence is exactly the kind of thing a conditional acceptance should require. The verdict should remain CONDITIONAL, with the condition being a matched, standard-or-clearly-labeled evaluation of all baselines under the identical protocol.","tokens_in":16260,"tokens_out":8276,"duration_ms":94354,"concrete_test":"Re-run the closed-loop evaluation of the top baseline planners (at least GameFormer, PlanTF, and DTPP) under the exact 8-second, first-frame-only protocol described in Sec. 4.1, on the same Close-Hard20 scenarios, and compare those scores with the values reported in Tables 2-3. If any baseline score shifts by more than roughly 1.3 points (VLMPlanner's reported margin over DTPP in the non-reactive setting), the comparison is protocol-confounded and the closed-loop SOTA claim should be withdrawn or re-scoped. As a secondary check, compute the ranking of the same baselines under the official nuPlan 0.1s-replanning protocol; if the top-3 ordering differs from the compressed-protocol ordering, the compressed protocol is not a valid proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VLMPlanner 'outperforms state-of-the-art methods in both open-loop and closed-loop settings' depends on the closed-loop evaluation being a valid and fair comparison. In Sec. 4.1 the authors state that, because nuPlan closed-loop testing provides images only at the initial frame, they 'compress the testing duration to 8 seconds and perform trajectory prediction solely at the first frame, persistently executing this initial prediction throughout the subsequent period,' citing NavSim for consistency with standard closed-loop testing. This is not the standard nuPlan closed-loop protocol, which normally replans every 0.1 s from fresh observations. The paper does not report any direct comparison between this compressed protocol and the official protocol, and it does not state whether the baseline numbers in Tables 2-3 (e.g., GameFormer 73.99, PlanTF 74.89, DTPP 75.09) were recomputed under the identical 8-second, first-frame-only protocol or were taken from published standard-protocol evaluations. Without a matched evaluation, the claimed closed-loop margins of 1.26-2.15 points could be an artifact of protocol mismatch. The same ambiguity applies to the Open-Hard20 subset: Table 1 reports all methods 'on Open-Hard20,' but the selection procedure is only described for constructing the split, not for rerunning every baseline on that split. The CAI-Gate ablation in Table 4 also measures inference intervals only on Open-Hard20, so the closed-loop results never exercise the adaptive-gate mechanism. Because the reported margins are small and no code, error bars, or appendix details are provided, the closed-loop SOTA claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"VLMPlanner is a hybrid autonomous-driving motion planning framework that combines a conventional learning-based real-time planner (GameFormer or PlanTF) with a vision-language model (LLaVA-v1.5-7b). The VLM processes multi-view RGB images together with map features, a system prompt, and a navigation instruction; its last-token hidden state is injected into the planner's decoder through an adaptive gating block borrowed from AsyncDriver. A Context-Adaptive Inference Gate (CAI-Gate) is introduced to decide when the VLM should be invoked, based on a five-level scene-complexity estimate produced either by a learned EfficientNet-B0 ordinal regressor or by a rule-based heuristic. The authors construct two image-text datasets derived from nuPlan (DriveVQA and ReasoningVQA) to fine-tune the VLM, and evaluate the full system on Open-Hard20 and Close-Hard20 scenario subsets of nuPlan, reporting improved open-loop scores (76.54 vs. 76.22 for GameFormer; 79.12 vs. 78.86 for PlanTF) and closed-loop scores (up to 76.35 non-reactive and 66.66 reactive). The paper claims state-of-the-art performance in both open-loop and closed-loop settings and shows that CAI-Gate can reduce VLM inference frequency while approximately maintaining the full-inference score.","tokens_in":16601,"tokens_out":5091,"duration_ms":50938,"significance":"If the empirical claims hold, the paper makes a useful contribution: it demonstrates a concrete way to inject raw-image VLM reasoning into a real-time planner without letting the VLM generate trajectories directly, and it provides an adaptive gating mechanism that addresses the latency problem of LLM/VLM-based planners. The construction of DriveVQA and ReasoningVQA is also a potentially reusable asset for the community. The open-loop results are consistent across two different underlying planners, which strengthens the plausibility of the architectural mechanism. However, the closed-loop state-of-the-art claim rests on a substantially modified evaluation protocol, and the absolute margins over baselines are small; without matched baseline evaluation and variance reporting, the significance of the improvement is not yet established.","major_comments":[{"comment":"The closed-loop experiments do not follow the standard nuPlan protocol, and the paper does not establish that the reported baselines were evaluated under the same modified protocol. Section 4.1 states that testing is compressed to 8 seconds, trajectory prediction is performed only at the first frame, and that initial prediction is persistently executed, citing NavSim for consistency. This is not the nuPlan closed-loop setting used by GameFormer, PlanTF, DTPP, or AsyncDriver, which replan from fresh observations at the simulation frequency. The claim of \"state-of-the-art\" closed-loop performance in Tables 2 and 3 is therefore not supported unless the baseline scores are recomputed under the identical 8-second, first-frame-only protocol and on the same Close-Hard20 scenarios. The paper should report this matched comparison, or explicitly restrict the claim to the modified protocol.","section":"§4.1, Tables 2–3"},{"comment":"The Open-Hard20 and Close-Hard20 subsets are constructed by selecting the 20 worst-performing scenarios per category according to PDM, but the manuscript does not state whether every baseline in Tables 1–3 was rerun on these exact subsets. If the baseline numbers were taken from published results on the original nuPlan test set, the comparison is not matched and the small margins (e.g., 76.54 vs. 76.22 for GameFormer and 79.12 vs. 78.86 for PlanTF in Table 1) are uninterpretable. Please clarify how each baseline score was obtained and, if any score was not rerun, supply the matched evaluation.","section":"§4.1, Table 1"},{"comment":"The open-loop improvements are within what one might expect from run-to-run variability: +0.32 for GameFormer and +0.26 for PlanTF. The paper reports a single run with no error bars, no multiple seeds, and no statistical test. Given the claim that the method \"outperforms state-of-the-art methods in both open-loop and closed-loop settings,\" the authors should report means and standard deviations over multiple seeds, or a paired analysis over the scenarios, before concluding that the VLM guidance improves the planner.","section":"§4.2, Table 1"},{"comment":"The learning-based CAI-Gate is trained on 7,503 image-complexity pairs generated by Gemini, with 70% for training and 30% for testing, but the manuscript does not describe the prompt, the label distribution, or the relationship between the training split and the Open-Hard20 evaluation scenarios on which the \"Average Interval\" in Table 4 is measured. If the gate was trained on images from the same scenarios used for evaluation, the adaptive-interval results could be optimistic. Also, Table 4 reports point scores only (76.50 vs. 76.54 for setting 1), so the claimed efficiency-performance trade-off lacks a variance estimate.","section":"§3.3, Table 4"}],"minor_comments":[{"comment":"Typo: \"blance\" should be \"balance\".","section":"§3.3"},{"comment":"\"Chine\" appears twice in the author affiliations; it should be \"China\".","section":"Author affiliations"},{"comment":"References [2] and [3] are the same GPT-4 technical report, and [1] contains the typo \"nuplan challange\"; the bibliography should be deduplicated and cleaned.","section":"References"},{"comment":"Equation (7) uses K̃h and Ṽh without defining the tilde notation, and the subscript/superscript on s_l is inconsistent; please define all quantities.","section":"§3.2, Eq. (7)"},{"comment":"The paper repeatedly refers to an Appendix for details (system prompt, rule-based CAI-Gate specifications, fine-tuning data construction), but no appendix is present in the submitted version; without these details the reproducibility of the data generation and the rule-based gate cannot be assessed.","section":"§3.4 and §4.1"},{"comment":"Table 5 is formatted ambiguously: the first data row has no markers in the DriveVQA/ReasoningVQA columns, making it unclear which configuration corresponds to \"no pretraining\".","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The core architectural idea is sound and the open-loop results are at least directionally consistent, but the closed-loop evaluation protocol is the main risk. The reported closed-loop margins of 1.26–2.15 points are obtained under a non-standard 8-second, first-frame-only protocol, and the paper does not show that the baselines were run under the same protocol or scenario subset. I would not reject the paper, because the issue is fixable in principle by supplying matched baselines or by clearly restricting the claims. I also suggest the authors report multiple seeds for the main tables, as the open-loop gains are small enough that a single run may be misleading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: VLMPlanner is a competent, interesting engineering paper with a real new combination — raw multi-view images fed through a 3D-aware Q-Former into a VLM whose hidden states guide a real-time planner, plus a learned/rule-based gate that lowers VLM inference frequency. The open-loop numbers are consistent but small: +0.32 over GameFormer and +0.26 over PlanTF on the Open-Hard20 split. The CAI-Gate ablation is the most interesting part: it keeps most of the gain at an average interval of 91 steps, which is a genuinely useful result if it holds.\n\nThe paper does a few things well. The architecture is clearly described, the two fine-tuning datasets (DriveVQA and ReasoningVQA) are a concrete contribution, and the qualitative examples in Fig. 3 illustrate the intended mechanism. There is no circularity: no fitted constants and no prediction that reduces to an input by construction.\n\nThe soft spots are exactly where the reader and stress-test point. The closed-loop SOTA claim rests on a non-standard 8-second, first-frame-only protocol. The paper says this follows NavSim, but it does not show that the baseline numbers in Tables 2 and 3 were recomputed under that same protocol. If they were taken from standard-protocol papers, the margins of 1.26–2.15 points are not interpretable. The paper also reports no error bars or variance, which matters because the gains are small. And the CAI-Gate ablation is only run in open-loop; the closed-loop experiments use full VLM inference at every step, so the paper never demonstrates that the adaptive gate works in closed-loop settings. The appendix is referenced for details that do not exist in the arXiv version, which makes verification harder.\n\nI would not call any of this fatal. The open-loop result and the CAI-Gate ablation are enough to justify a serious look. But the claim of outperforming state-of-the-art in closed-loop is not established as written. It needs either matched baselines under the same protocol or a repositioning to \"we improve over our base planners under a compressed protocol.\"\n\nWho is this for: people doing LLM/VLM-informed planning on nuPlan or similar benchmarks. It deserves a serious referee and a revision request, not a desk reject. I'd bring it to reading group, and I'd cite the architecture if I were working on this line.","headline":"A solid engineering contribution with a genuinely new VLM-plus-planner combination, but the closed-loop SOTA claim outruns the evidence because the evaluation protocol is non-standard and baselines are not shown to be matched.","tokens_in":17153,"tokens_out":2148,"would_cite":true,"duration_ms":21791,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VLMPlanner shows that a vision-language model reading multi-view images can improve a real-time driving planner on hard nuPlan scenarios by injecting hidden-state guidance, while the CAI-Gate cuts how often the VLM must run.","keywords":["vision-language model","motion planning","autonomous driving","nuPlan","multi-view images","context-adaptive inference gate","long-tail scenarios","closed-loop planning"],"falsifier":"Run the standard full-duration nuPlan closed-loop evaluation with multi-view images available at every timestep and compare against the same baselines; if VLMPlanner's score no longer exceeds GameFormer and PlanTF, the central claim is confined to the compressed protocol. A second check: ablate the VLM by replacing image tokens with zeros and see whether trajectories change; if they barely change, the visual guidance is not what is driving the gain.","tokens_in":1848,"feed_emoji":"🚗","tokens_out":3751,"duration_ms":100021,"temperature":0.7,"pith_summary":"This paper introduces VLMPlanner, a hybrid driving-planning framework in which a vision-language model reads multi-view camera images together with map features and a navigation instruction, and injects its final hidden state into a conventional real-time trajectory planner. The authors try to establish that this visual-linguistic guidance improves planning on hard nuPlan scenarios in both open-loop and closed-loop settings, and that a Context-Adaptive Inference Gate can decide when the VLM is needed, preserving most of the benefit while cutting how often the slow model runs. If true, the practical payoff is a division of labor: the fast learned planner handles routine driving, while a VLM acts as an occasional common-sense advisor that can catch visual hazards such as pedestrians, red lights, and accident debris that map and perception inputs miss. The paper also argues that the gate can reduce VLM calls to about one per 91 planner steps on the hardest open-loop split with only a small score drop, which matters for real-time deployment.","feed_headline":"Raw image reasoning improves nuPlan driving scores","feed_subtitle":"A VLM's hidden states steer a fast planner through hard scenarios, cutting collisions while running less often.","key_machinery":"The load-bearing mechanism is hidden-state guidance: after the VLM reads the system prompt, map tokens, compressed multi-view image tokens, and navigation instruction, the feature of its final token is projected and injected through an Adaptive Injection Block into every decoder layer of the real-time planner, with a learned gate controlling how much the VLM's scene understanding alters the planner's attention. The second mechanism is the Context-Adaptive Inference Gate (CAI-Gate), which labels scene complexity in five grades and maps each grade to a VLM inference interval, so the slow VLM runs only when the scene demands it. Supporting machinery includes a 3D-aware Q-Former-style module that turns CLIP image features into fewer 3D queries, and two LoRA fine-tuning datasets, DriveVQA and ReasoningVQA, that adapt the VLM to driving instructions and reasoning.","core_discovery":"The paper's central claim is that a vision-language model can improve a learned planner not by writing trajectories, but by supplying compressed hidden-state features distilled from raw multi-view imagery, capturing visual cues such as a pedestrian entering a crosswalk or a red light that map-level perception compresses away. On the curated nuPlan hard splits, both implementations beat their base planners: GameFormer rises from 76.22 to 76.54 open-loop and from 73.99 to 76.35 in non-reactive closed-loop, while PlanTF rises from 78.86 to 79.12 open-loop; the method also reports the best closed-loop scores among the compared methods. The authors additionally claim that the CAI-Gate can reduce VLM inference to an average interval of roughly 91 planner steps while holding the open-loop score at 76.16, close to the 76.54 of every-step inference.","pith_inferences":["A testable implication is that the hidden-state injection may be doing most of the work rather than the language reasoning: if image tokens alone, without navigation text or system prompt, produce the same planner improvements, the method becomes a general visual-feature guidance mechanism and the VLM's verbal reasoning is optional.","The CAI-Gate depends on the quality of its complexity labels, which were generated by a large multimodal model rather than by human annotators; a human-labeled complexity test set would show whether the gate's five-level schedule is near-optimal or merely fitted to those labels.","The compressed one-shot closed-loop protocol makes the closed-loop numbers hard to compare with full-duration nuPlan results; if the community adopts this protocol, leaderboard comparisons will need to state the compression explicitly."],"forward_implications":["On the curated hard splits, adding VLM hidden-state guidance raises GameFormer from 76.22 to 76.54 open-loop and from 73.99 to 76.35 non-reactive closed-loop.","With PlanTF as the real-time planner, the framework reaches 79.12 open-loop, above PlanTF's 78.86, and 75.82 non-reactive closed-loop.","The CAI-Gate can run the VLM once every 91 planner steps on average and still score 76.16 open-loop, compared with 76.54 when the VLM runs every step.","Qualitative results show the VLM preventing a pedestrian-collision and a red-light-violation that a compared asynchronous planner commits, indicating reduced at-fault collisions from visual reasoning.","Pretraining on the two generated VQA datasets is credited with most of the closed-loop improvement, raising GameFormer non-reactive score from 74.42 to 76.35."],"supporting_citations":[{"why":"Supplies the Adaptive Injection Block that carries VLM hidden states into the planner decoder, and the async decoupling design this paper extends with image input and a complexity gate.","marker":"[7]"},{"why":"GameFormer is the real-time planner backbone of the primary implementation; its map encoder, decoder layers, and training paradigm are reused.","marker":"[25]"},{"why":"PlanTF is the second real-time planner backbone and the strongest open-loop baseline the method must beat.","marker":"[9]"},{"why":"nuPlan provides the dataset, scenario types, hard-split construction, and official open and closed-loop metrics.","marker":"[4]"},{"why":"Supplies the compressed 8-second closed-loop evaluation protocol that lets the authors test with images.","marker":"[28]"},{"why":"CLIP encodes the multi-view images into visual features before the 3D-aware token reduction.","marker":"[46]"},{"why":"The BLIP/Q-Former-style 3D-aware module reduces image tokens and lifts visual features into 3D space.","marker":"[35]"},{"why":"Generates the ReasoningVQA training data used to give the VLM driving-scene reasoning ability.","marker":"[2]"},{"why":"PDM scores all sampled test scenarios so the authors can select the worst 20 per type as the hard splits.","marker":"[14]"}],"fun_headline_variants":["VLM hidden states boost nuPlan planner scores","Raw images, not maps, improve nuPlan driving","Gated VLM saves compute, keeps nuPlan performance","Vision-language model reads roads for better planning","VLMPlanner: hidden features guide planner in nuPlan"],"cache_read_input_tokens":19200,"weakest_assumption_plain":"The closed-loop comparisons assume that an 8-second test in which the trajectory is predicted once at the first frame and then executed without updates remains a fair proxy for standard nuPlan closed-loop evaluation; if that proxy is not faithful, the closed-loop state-of-the-art claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["VLM hidden states boost nuPlan planner scores","Raw images, not maps, improve nuPlan driving","Gated VLM saves compute, keeps nuPlan performance","Vision-language model reads roads for better planning","VLMPlanner: hidden features guide planner in nuPlan"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2775,"prompt_tokens":939,"completion_tokens":1836,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1760}},"tokens_in":555,"tokens_out":1836,"duration_ms":17100,"temperature":1.0,"reasoning_tokens":1760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:44:58.255636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the standard full-duration nuPlan closed-loop evaluation with multi-view images available at every timestep and compare against the same baselines; if VLMPlanner's score no longer exceeds GameFormer and PlanTF, the central claim is confined to the compressed protocol. A second check: ablate the VLM by replacing image tokens with zeros and see whether trajectories change; if they barely change, the visual guidance is not what is driving the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Adaptive Injection Block that carries VLM hidden states into the planner decoder, and the async decoupling design this paper extends with image input and a complexity gate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PlanTF is the second real-time planner backbone and the strongest open-loop baseline the method must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the compressed 8-second closed-loop evaluation protocol that lets the authors test with images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PDM scores all sampled test scenarios so the authors can select the worst 20 per type as the hard splits."}],"review_version":1}