Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

VLMPlanner: Integrating Visual Language Models with Motion Planning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VLMPlanner shows that a vision-language model reading multi-view images can improve a real-time driving planner on hard nuPlan scenarios by injecting hidden-state guidance, while the CAI-Gate cuts how often the VLM must run.

desk verdict A solid engineering contribution with a genuinely new VLM-plus-planner combination, but the closed-loop SOTA claim outruns the evidence because the evaluation protocol is non-standard and baselines are not shown to be matched. read the letter →

arxiv 2507.20342 v1 pith:JXYGJ6FD submitted 2025-07-27 cs.AI cs.RO

Zhipeng Tang , Sha Zhang , Jiajun Deng , Chenjie Wang , Guoliang You , Yuting Huang , Xinrui Lin , Yanyong Zhang This is my paper · ORCID
classification cs.AIcs.RO
keywords vision-languagemodelmotionplanningautonomousdrivingnuPlanmulti-viewimagescontext-adaptiveinferencegatelong-tailscenariosclosed-loop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VLMPlanner, a hybrid driving-planning framework in which a vision-language model reads multi-view camera images together with map features and a navigation instruction, and injects its final hidden state into a conventional real-time trajectory planner. The authors try to establish that this visual-linguistic guidance improves planning on hard nuPlan scenarios in both open-loop and closed-loop settings, and that a Context-Adaptive Inference Gate can decide when the VLM is needed, preserving most of the benefit while cutting how often the slow model runs. If true, the practical payoff is a division of labor: the fast learned planner handles routine driving, while a VLM acts as an occasional common-sense advisor that can catch visual hazards such as pedestrians, red lights, and accident debris that map and perception inputs miss. The paper also argues that the gate can reduce VLM calls to about one per 91 planner steps on the hardest open-loop split with only a small score drop, which matters for real-time deployment.

What carries the argument

The load-bearing mechanism is hidden-state guidance: after the VLM reads the system prompt, map tokens, compressed multi-view image tokens, and navigation instruction, the feature of its final token is projected and injected through an Adaptive Injection Block into every decoder layer of the real-time planner, with a learned gate controlling how much the VLM's scene understanding alters the planner's attention. The second mechanism is the Context-Adaptive Inference Gate (CAI-Gate), which labels scene complexity in five grades and maps each grade to a VLM inference interval, so the slow VLM runs only when the scene demands it. Supporting machinery includes a 3D-aware Q-Former-style module that turns CLIP image features into fewer 3D queries, and two LoRA fine-tuning datasets, DriveVQA and ReasoningVQA, that adapt the VLM to driving instructions and reasoning.

What would settle it

Run the standard full-duration nuPlan closed-loop evaluation with multi-view images available at every timestep and compare against the same baselines; if VLMPlanner's score no longer exceeds GameFormer and PlanTF, the central claim is confined to the compressed protocol. A second check: ablate the VLM by replacing image tokens with zeros and see whether trajectories change; if they barely change, the visual guidance is not what is driving the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that a vision-language model can improve a learned planner not by writing trajectories, but by supplying compressed hidden-state features distilled from raw multi-view imagery, capturing visual cues such as a pedestrian entering a crosswalk or a red light that map-level perception compresses away. On the curated nuPlan hard splits, both implementations beat their base planners: GameFormer rises from 76.22 to 76.54 open-loop and from 73.99 to 76.35 in non-reactive closed-loop, while PlanTF rises from 78.86 to 79.12 open-loop; the method also reports the best closed-loop scores among the compared methods. The authors additionally claim that the CAI-Gate can reduce VLM inference to an average interval of roughly 91 planner steps while holding the open-loop score at 76.16, close to the 76.54 of every-step inference.

Load-bearing premise

The closed-loop comparisons assume that an 8-second test in which the trajectory is predicted once at the first frame and then executed without updates remains a fair proxy for standard nuPlan closed-loop evaluation; if that proxy is not faithful, the closed-loop state-of-the-art claim is not established.

Editorial extensions

If this is right

  • On the curated hard splits, adding VLM hidden-state guidance raises GameFormer from 76.22 to 76.54 open-loop and from 73.99 to 76.35 non-reactive closed-loop.
  • With PlanTF as the real-time planner, the framework reaches 79.12 open-loop, above PlanTF's 78.86, and 75.82 non-reactive closed-loop.
  • The CAI-Gate can run the VLM once every 91 planner steps on average and still score 76.16 open-loop, compared with 76.54 when the VLM runs every step.
  • Qualitative results show the VLM preventing a pedestrian-collision and a red-light-violation that a compared asynchronous planner commits, indicating reduced at-fault collisions from visual reasoning.
  • Pretraining on the two generated VQA datasets is credited with most of the closed-loop improvement, raising GameFormer non-reactive score from 74.42 to 76.35.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication is that the hidden-state injection may be doing most of the work rather than the language reasoning: if image tokens alone, without navigation text or system prompt, produce the same planner improvements, the method becomes a general visual-feature guidance mechanism and the VLM's verbal reasoning is optional.
  • The CAI-Gate depends on the quality of its complexity labels, which were generated by a large multimodal model rather than by human annotators; a human-labeled complexity test set would show whether the gate's five-level schedule is near-optimal or merely fitted to those labels.
  • The compressed one-shot closed-loop protocol makes the closed-loop numbers hard to compare with full-duration nuPlan results; if the community adopts this protocol, leaderboard comparisons will need to state the compression explicitly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. VLMPlanner is a hybrid autonomous-driving motion planning framework that combines a conventional learning-based real-time planner (GameFormer or PlanTF) with a vision-language model (LLaVA-v1.5-7b). The VLM processes multi-view RGB images together with map features, a system prompt, and a navigation instruction; its last-token hidden state is injected into the planner's decoder through an adaptive gating block borrowed from AsyncDriver. A Context-Adaptive Inference Gate (CAI-Gate) is introduced to decide when the VLM should be invoked, based on a five-level scene-complexity estimate produced either by a learned EfficientNet-B0 ordinal regressor or by a rule-based heuristic. The authors construct two image-text datasets derived from nuPlan (DriveVQA and ReasoningVQA) to fine-tune the VLM, and evaluate the full system on Open-Hard20 and Close-Hard20 scenario subsets of nuPlan, reporting improved open-loop scores (76.54 vs. 76.22 for GameFormer; 79.12 vs. 78.86 for PlanTF) and closed-loop scores (up to 76.35 non-reactive and 66.66 reactive). The paper claims state-of-the-art performance in both open-loop and closed-loop settings and shows that CAI-Gate can reduce VLM inference frequency while approximately maintaining the full-inference score.

Significance. If the empirical claims hold, the paper makes a useful contribution: it demonstrates a concrete way to inject raw-image VLM reasoning into a real-time planner without letting the VLM generate trajectories directly, and it provides an adaptive gating mechanism that addresses the latency problem of LLM/VLM-based planners. The construction of DriveVQA and ReasoningVQA is also a potentially reusable asset for the community. The open-loop results are consistent across two different underlying planners, which strengthens the plausibility of the architectural mechanism. However, the closed-loop state-of-the-art claim rests on a substantially modified evaluation protocol, and the absolute margins over baselines are small; without matched baseline evaluation and variance reporting, the significance of the improvement is not yet established.

major comments (4)
  1. [§4.1, Tables 2–3] The closed-loop experiments do not follow the standard nuPlan protocol, and the paper does not establish that the reported baselines were evaluated under the same modified protocol. Section 4.1 states that testing is compressed to 8 seconds, trajectory prediction is performed only at the first frame, and that initial prediction is persistently executed, citing NavSim for consistency. This is not the nuPlan closed-loop setting used by GameFormer, PlanTF, DTPP, or AsyncDriver, which replan from fresh observations at the simulation frequency. The claim of "state-of-the-art" closed-loop performance in Tables 2 and 3 is therefore not supported unless the baseline scores are recomputed under the identical 8-second, first-frame-only protocol and on the same Close-Hard20 scenarios. The paper should report this matched comparison, or explicitly restrict the claim to the modified protocol.
  2. [§4.1, Table 1] The Open-Hard20 and Close-Hard20 subsets are constructed by selecting the 20 worst-performing scenarios per category according to PDM, but the manuscript does not state whether every baseline in Tables 1–3 was rerun on these exact subsets. If the baseline numbers were taken from published results on the original nuPlan test set, the comparison is not matched and the small margins (e.g., 76.54 vs. 76.22 for GameFormer and 79.12 vs. 78.86 for PlanTF in Table 1) are uninterpretable. Please clarify how each baseline score was obtained and, if any score was not rerun, supply the matched evaluation.
  3. [§4.2, Table 1] The open-loop improvements are within what one might expect from run-to-run variability: +0.32 for GameFormer and +0.26 for PlanTF. The paper reports a single run with no error bars, no multiple seeds, and no statistical test. Given the claim that the method "outperforms state-of-the-art methods in both open-loop and closed-loop settings," the authors should report means and standard deviations over multiple seeds, or a paired analysis over the scenarios, before concluding that the VLM guidance improves the planner.
  4. [§3.3, Table 4] The learning-based CAI-Gate is trained on 7,503 image-complexity pairs generated by Gemini, with 70% for training and 30% for testing, but the manuscript does not describe the prompt, the label distribution, or the relationship between the training split and the Open-Hard20 evaluation scenarios on which the "Average Interval" in Table 4 is measured. If the gate was trained on images from the same scenarios used for evaluation, the adaptive-interval results could be optimistic. Also, Table 4 reports point scores only (76.50 vs. 76.54 for setting 1), so the claimed efficiency-performance trade-off lacks a variance estimate.
minor comments (6)
  1. [§3.3] Typo: "blance" should be "balance".
  2. [Author affiliations] "Chine" appears twice in the author affiliations; it should be "China".
  3. [References] References [2] and [3] are the same GPT-4 technical report, and [1] contains the typo "nuplan challange"; the bibliography should be deduplicated and cleaned.
  4. [§3.2, Eq. (7)] Equation (7) uses K̃h and Ṽh without defining the tilde notation, and the subscript/superscript on s_l is inconsistent; please define all quantities.
  5. [§3.4 and §4.1] The paper repeatedly refers to an Appendix for details (system prompt, rule-based CAI-Gate specifications, fine-tuning data construction), but no appendix is present in the submitted version; without these details the reproducibility of the data generation and the rule-based gate cannot be assessed.
  6. [Table 5] Table 5 is formatted ambiguously: the first data row has no markers in the DriveVQA/ReasoningVQA columns, making it unclear which configuration corresponds to "no pretraining".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VLMPlanner's claims are empirical evaluations on held-out nuPlan scenarios using external prior components, with no equation-level reduction or self-citation chain.

full rationale

The paper's central claim is empirical: VLMPlanner injects VLM hidden-state features into a learned nuPlan planner and reports higher official scores on held-out Open-Hard20/Close-Hard20 scenarios. None of the reported quantities are defined in terms of each other: the nuPlan score is computed with the official metric suite from predicted trajectories, the VLM guidance features are outputs of an externally pre-trained CLIP/LLaVA backbone plus trained adapters, and the CAI-Gate ablation compares learned/rule-based inference intervals against fixed intervals on the same evaluation set. The only self-citation ([67], a general pointer to multi-modality fusion for the claim that motion planning is a core component) appears in the introduction and carries no load; all load-bearing prior components (AsyncDriver's Adaptive Injection Block, GameFormer/PlanTF planners, NavSim's protocol argument) are external works with independent support. The 8-second first-frame-only closed-loop protocol and the fact that the hard split was selected by PDM performance are threats to external validity or baseline comparability, not circular reasoning; they do not make any predicted score equal to a fitted input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. DriveVQA and ReasoningVQA are datasets, and the CAI-Gate is a software component rather than a postulated entity. The load-bearing assumptions are the evaluation protocol and the accuracy of LLM-generated training labels.

free parameters (3)
  • CAI-Gate inference interval settings = [10,20,40,60,90]; [20,40,60,90,120]; [40,60,90,120,160]
    These arrays map each of five complexity levels to the number of planner steps between VLM inferences. They are chosen by hand, and the paper uses them to claim an optimal performance and compute balance without explaining how the values were selected.
  • CAI-Gate training data split = 70% train / 30% test of 7,503 Gemini-labeled images
    The split is a hand-chosen design decision for training the EfficientNet-B0 gate; no sensitivity analysis is given.
  • Fine-tuning sample size = 10,000 nuPlan instances
    The joint planner and VLM fine-tuning uses 10,000 curated samples with a category distribution matching nuPlan; this size is arbitrary and not ablated.
assumptions (4)
  • domain assumption nuPlan scores and scenario types are a valid proxy for real-world autonomous driving quality.
    All evaluation is on nuPlan; the paper's generalization claims to real roads are not supported by the benchmark alone.
  • ad hoc to paper The closed-loop protocol that executes a single first-frame prediction for 8 seconds is equivalent to standard closed-loop testing.
    Adopted in Sec 4.1 to make image-based evaluation possible; the paper cites NavSim for consistency but provides no validation for this specific setup.
  • domain assumption Text and complexity labels generated by GPT-4 and Gemini are accurate training signals.
    DriveVQA, ReasoningVQA, and the CAI-Gate labels come from LLM generation; no human verification rate or filtering procedure is reported.
  • domain assumption The CLIP encoder and 3D-aware Q-Former-style module preserve the fine-grained visual cues needed for planning.
    The method depends on compressed image tokens carrying enough detail for VLM reasoning; no quantitative check of information retention is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLMPlanner: Integrating Visual Language Models with Motion Planning." pith.science (2026). https://pith.science/paper/JXYGJ6FD

@misc{pith2026250720342,
  author       = {Pith},
  title        = {Pith review of: VLMPlanner: Integrating Visual Language Models with Motion Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXYGJ6FD}},
  note         = {Machine review of arXiv:2507.20342}
}
read the original abstract

Integrating large language models (LLMs) into autonomous driving motion planning has recently emerged as a promising direction, offering enhanced interpretability, better controllability, and improved generalization in rare and long-tail scenarios. However, existing methods often rely on abstracted perception or map-based inputs, missing crucial visual context, such as fine-grained road cues, accident aftermath, or unexpected obstacles, which are essential for robust decision-making in complex driving environments. To bridge this gap, we propose VLMPlanner, a hybrid framework that combines a learning-based real-time planner with a vision-language model (VLM) capable of reasoning over raw images. The VLM processes multi-view images to capture rich, detailed visual information and leverages its common-sense reasoning capabilities to guide the real-time planner in generating robust and safe trajectories. Furthermore, we develop the Context-Adaptive Inference Gate (CAI-Gate) mechanism that enables the VLM to mimic human driving behavior by dynamically adjusting its inference frequency based on scene complexity, thereby achieving an optimal balance between planning performance and computational efficiency. We evaluate our approach on the large-scale, challenging nuPlan benchmark, with comprehensive experimental results demonstrating superior planning performance in scenarios with intricate road conditions and dynamic elements. Code will be available.

Figures

Figures reproduced from arXiv: 2507.20342 by the authors.

Figure 1
Figure 1. Comparison of LLM-related planner methods. (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of VLMPlanner. Our method comprises two key components: (1) the real-time planner, which [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparative visualization analysis between our VLMPlanner and AsyncDriver in various driving scenarios. In the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Fine-tuning multimodal language models on a new SOTIF-focused driving dataset improves question answering and captioning, but the open-ended gains are measured by an LLM judge with no independent human scoring.

Reference graph

Works this paper leans on

73 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Motional: nuplan challange

    2019. Motional: nuplan challange. https://github.com/motional/nuplan-devkit

    work page 2019

  2. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [4]

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. 2021. nuplan: A closed- loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810 (2021)

    arXiv 2021

  4. [5]

    Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. 2024. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 14093–14100

    work page 2024

  5. [6]

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. 2023. Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4- vision and beyond. arXiv preprint arXiv:2310.02071 (2023)

    arXiv 2023

  6. [7]

    Yuan Chen, Zi-han Ding, Ziqin Wang, Yan Wang, Lijun Zhang, and Si Liu. 2024. Asynchronous large language model enhanced planner for autonomous driving. In European Conference on Computer Vision . Springer, 22–38

    work page 2024

  7. [8]

    Jie Cheng, Yingbing Chen, and Qifeng Chen. 2024. Pluto: Pushing the limit of imitation learning-based planning for autonomous driving. arXiv preprint arXiv:2404.14327 (2024)

    arXiv 2024

  8. [9]

    Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, Bo Li, and Ming Liu

    work page

Show all 73 references
  1. [10]

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. 2022. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence 45, 11 (2022), 12878–12895

    2022

  2. [11]

    Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. 2019. Ex- ploring the limitations of behavior cloning for autonomous driving. InProceedings of the IEEE/CVF international conference on computer vision . 9329–9338

    2019

  3. [12]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2024. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine (2024)

    2024

  4. [13]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al . 2024. A survey on mul- timodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Comp...

    2024

  5. [14]

    Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. 2023. Parting with misconceptions about learning-based vehicle motion planning. In Conference on Robot Learning . PMLR, 1268–1281

    2023

  6. [15]

    Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. 2024. Holistic autonomous driving understanding by bird’s-eye-view injected multi-modal large models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13668–13677

    2024

  7. [16]

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao

  8. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  9. [18]

    In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (W ACVW)

    Drive like a human: Rethinking autonomous driving with large language models. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (W ACVW). IEEE, 910–919

    2024

  10. [19]

    Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jianbing Shen. 2024. Dme- driver: Integrating human decision logic and 3d scene perception in autonomous driving. arXiv preprint arXiv:2401.03641 (2024)

    2024 arXiv

  11. [20]

    Marcel Hallgarten, Martin Stoll, and Andreas Zell. 2023. From prediction to plan- ning with goal conditioned lane graph traversals. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC) . IEEE, 951–958

    2023

  12. [21]

    Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. 2024. Solving motion planning tasks with a scalable generative model. In European Conference on Computer Vision . Springer, 386–404

    2024

  13. [22]

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao

  14. [23]

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al . 2023. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 17853–17862

    2023

  15. [24]

    Zhiyu Huang, Peter Karkus, Boris Ivanovic, Yuxiao Chen, Marco Pavone, and Chen Lv. 2024. Dtpp: Differentiable joint conditional prediction and cost evalua- tion for tree policy planning in autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (IC...

    2024

  16. [25]

    Yihan Hu, Kun Li, Pingyuan Liang, Jingyu Qian, Zhening Yang, Haichao Zhang, Wenxin Shao, Zhuangzhuang Ding, Wei Xu, and Qiang Liu. 2023. Imitation with spatial-temporal heatmap: 2nd place solution for nuplan challenge.arXiv preprint arXiv:2306.15700 (2023)

    2023 arXiv

  17. [26]

    Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. 2023. Differentiable in- tegrated motion prediction and planning with learnable cost function for au- tonomous driving. IEEE transactions on neural networks and learning systems (2023)

    2023

  18. [27]

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. 2024. Emma: End- to-end multimodal model for autonomous driving.arXiv preprint arXiv:2410.23262 (2024)

    2024 arXiv

  19. [28]

    Zhiyu Huang, Haochen Liu, and Chen Lv. 2023. Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3903–3913

    2023

  20. [29]

    Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. 2023. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 21983–21994

    2023

  21. [30]

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. 2024. Senna: Bridging large vision-language models and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313 (2024)

    2024 arXiv

  22. [31]

    NAV IM. 2024. NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simu- lation and Benchmarking. arXiv preprint arXiv:2406.15349 (2024)

    2024 arXiv

  23. [32]

    Ye Jin, Xiaoxi Shen, Huiling Peng, Xiaoan Liu, Jingli Qin, Jiayang Li, Jintao Xie, Peizhong Gao, Guyue Zhou, and Jiangtao Gong. 2023. Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model. arXiv preprint arXiv:230...

    2023 arXiv

  24. [33]

    Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John- Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. 2019. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA) . IEEE, 8248–8254

    2019

  25. [34]

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. 2023. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Visi...

    2023

  26. [35]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

    2022

  27. [36]

    Tengpeng Li, Hanli Wang, Xianfei Li, Wenlong Liao, Tao He, and Pai Peng

  28. [37]

    Katja Kircher and Christer Ahlstrom. 2017. Minimum required attention: A human-centered approach to driver inattention. Human factors 59, 3 (2017), 471–484

    2017

  29. [38]

    Jiaqi Liu, Peng Hang, Xiao Qi, Jianqiang Wang, and Jian Sun. 2023. Mtd-gpt: A multi-task decision-making gpt model for autonomous driving at unsignalized in- tersections. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 5154–5161

    2023

  30. [39]

    Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. 2022. Petr: Position em- bedding transformation for multi-view 3d object detection. InEuropean conference on computer vision. Springer, 531–548

    2022

  31. [40]

    Zhuofan Liu, Wei Yuan, and Yong Ma. 2021. Drivers’ attention strategies before eyes-off-road in different traffic scenarios: adaptation and anticipation. Interna- tional journal of environmental research and public health 18, 7 (2021), 3716

    2021

  32. [41]

    Zenan Li, Fan Nie, Qiao Sun, Fang Da, and Hang Zhao. 2024. Boosting offline reinforcement learning for autonomous driving with hierarchical latent skills. In 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 18362–18369

    2024

  33. [42]

    Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdel- raouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al. 2024. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference on Compute...

    2024

  34. [43]

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. 2023. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415 (2023)

    2023 arXiv

  35. [44]

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. 2023. A language agent for autonomous driving. arXiv preprint arXiv:2311.10813 (2023). VLMPlanner: Integrating Visual Language Models with Motion Planning Conference acronym ’XX, June 03–05, 2025, Dublin, Ireland

    2023 arXiv

  36. [45]

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. 2024. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision. Springer, 403–420

    2024

  37. [46]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  38. [47]

    Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A Koepke, Zeynep Akata, and Andreas Geiger. 2022. Plant: Explainable planning transformers via object- level representations. arXiv preprint arXiv:2210.14222 (2022)

    2022 arXiv

  39. [48]

    Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Błażej Osiński, and Peter On- druska. 2022. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Conference on Robot Learning . PMLR, 718–728

    2022

  40. [49]

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. 2024. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision . Springer, 292–308

    2024

  41. [50]

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. 2024. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15120–15130

    2024

  42. [51]

    SP Sharan, Francesco Pittaluga, Manmohan Chandraker, et al. 2023. Llm-assist: Enhancing closed-loop planning with language-based reasoning. arXiv preprint arXiv:2401.00125 (2023)

    2023 arXiv

  43. [52]

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. 2024. Driv- elm: Driving with graph visual question answering. In European Conference on Computer Vision. Springer, 256–274

    2024

  44. [53]

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. 2023. Languagempc: Large language models as decision makers for autonomous driving. arXiv preprint arXiv:2310.03026 (2023)

    2023 arXiv

  45. [54]

    Sheng Wang, Yao Tian, Xiaodong Mei, Ge Sun, Jie Cheng, Fulong Ma, Pedro V Sander, and Junwei Liang. 2024. LHPF: Look back the History and Plan for the Future in Autonomous Driving. arXiv preprint arXiv:2411.17253 (2024)

    2024 arXiv

  46. [55]

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. 2024. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533 (2024)

    2024 arXiv

  47. [56]

    Shiyi Wang, Yuxuan Zhu, Zhiheng Li, Yutong Wang, Li Li, and Zhengbing He

  48. [57]

    Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning . PMLR, 6105–6114

    2019

  49. [58]

    Yixuan Wang, Ruochen Jiao, Sinong Simon Zhan, Chengtian Lang, Chao Huang, Zhaoran Wang, Zhuoran Yang, and Qi Zhu. 2023. Empowering autonomous driving with large language models: A safety perspective. arXiv preprint arXiv:2312.00812 (2023)

    2023 arXiv

  50. [59]

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. 2023. Dilu: A knowledge-driven approach to autonomous driving with large language models. arXiv preprint arXiv:2309.16292 (2023)

    2023 arXiv

  51. [60]

    Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. 2024. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446 (2024)

    2024 arXiv

  52. [61]

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. 2024. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters (2024)

    2024

  53. [62]

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al . 2023. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245 (2023)

    2023

  54. [63]

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. 2023. Llm4drive: A survey of large language models for autonomous driving. arXiv preprint arXiv:2311.01043 (2023)

    2023 arXiv

  55. [64]

    Ruoyu Yao, Yubin Wang, Haichao Liu, Rui Yang, Zengqi Peng, Lei Zhu, and Jun Ma. 2024. CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model. arXiv preprint arXiv:2412.04209 (2024)

    2024 arXiv

  56. [65]

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. 2024. Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model. arXiv preprint arXiv:2402.10828 (2024)

    2024

  57. [66]

    Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. 2019. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 8660–8669

    2019

  58. [67]

    Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki. 2024. Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following.arXiv preprint arXiv:2402.06559 (2024)

    2024 arXiv

  59. [68]

    Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, et al . 2024. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning. arXiv preprint arXiv:2406.01587 (2024)

    2024 arXiv

  60. [69]

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: Deformable transformers for end-to-end object detection.arXiv preprint arXiv:2010.04159 (2020)

    2020 arXiv

  61. [72]

    Y Zhang, S Zhang, Y Zhang, J Ji, Y Duan, Y Huang, J Peng, and Y Zahng. 2020. Multi-modality fusion perception and computing in autonomous driving. J. Comput. Res. Dev 57 (2020), 1781–1799

    2020

  62. [2022]

    In European Conference on Computer Vision

    St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision . Springer, 533–549

  63. [2023]

    IEEE Transactions on Intelligent Vehicles 8, 12 (2023), 4706–4721

    Chatgpt as your vehicle co-pilot: An initial attempt. IEEE Transactions on Intelligent Vehicles 8, 12 (2023), 4706–4721

    2023

  64. [2024]

    In 2024 IEEE International Conference on Robotics and Automation (ICRA)

    Rethinking imitation-based planners for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 14123–14130

    2024

  65. [2025]

    arXiv preprint arXiv:2501.08861 (2025)

    Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving. arXiv preprint arXiv:2501.08861 (2025)

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.