Pith. sign in

REVIEW 4 major objections 5 minor 61 references

ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Injecting VLM reasoning at strategy, decision, and operation levels cuts planning error by 33% and collisions by 32% over the VAD baseline on nuScenes.

desk verdict The 30% safety claim rests on open-loop metrics; the closed-loop evidence is undermined by a memory-buffer retrieval that skips online VLM reasoning. read the letter →

arxiv 2507.12499 v1 pith:P6BSV6WG submitted 2025-07-16 cs.RO cs.LG

Yuhang Lu , Jiadong Tu , Yuexin Ma , Xinge Zhu This is my paper
classification cs.ROcs.LG
keywords end-to-endautonomousdrivingvision-languagemodelshierarchicalreasoningtrajectoryplanningopen-loopevaluationclosed-loopnuScenesBench2Drive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that end-to-end autonomous driving planners become substantially more accurate and safer when the network is made to reason in the same hierarchy human drivers use: first a high-level driving strategy, then tactical choices such as turns, lane changes, and speed adjustments, and finally a coarse-to-fine trajectory. The proposed framework, ReAL-AD, wraps an existing planner (VAD or UniAD) with a vision-language model that writes the strategy and tactical commands in text, plus a hierarchical variational decoder that converts those commands into trajectories. On nuScenes the authors report a 33% lower average L2 error and a 32% lower collision rate than the VAD baseline, and they report smaller closed-loop gains on Bench2Drive. If the claim holds, language-model reasoning can serve as a training signal for precise low-level control, and end-to-end driving systems gain an inspectable reasoning layer without abandoning end-to-end training.

What carries the argument

The load-bearing object is the three-tier reasoning scaffold. The Strategic Reasoning Injector turns VLM-generated free-text strategy into planning features through a text encoder and adapter, aligned to ground-truth trajectory features by a cosine similarity loss and added residually to the ego-query. The Tactical Reasoning Integrator constrains the VLM to output categorical commands, parses them by regular expressions, and embeds them into reactive ($g_{\text{reactive}} = e_{\text{dir}} \oplus e_{\text{emerg}}$) and regulatory ($g_{\text{regulatory}} = e_{\text{lane}} \oplus e_{\text{speed}}$) features. The Hierarchical Trajectory Decoder is a two-level conditional variational decoder: a coarse latent $z_c \sim \mathcal{N}(\mu_c,\sigma_c)$ is conditioned on ego features and reactive commands, and a fine latent $z_f$ is conditioned on ego features, $z_c$, and regulatory commands; two GRU streams decode coarse and fine trajectories, supervised by a B\'ezier-fitted coarse ground truth and a hierarchical KL divergence. This machinery carries the argument because it converts language-level reasoning into differentiable conditioning at the two points where trajectory uncertainty is resolved.

What would settle it

Run the final model closed-loop on the 220 Bench2Drive routes and compare collision rate and driving score against VAD; if the gap stays near the reported 1.75-point driving-score and 1.36-point success-rate gains while open-loop metrics move by roughly 30%, the headline safety claim is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the gap between end-to-end planners and human driving is not just missing semantic cues but a missing decision hierarchy. ReAL-AD structures planning as Driving Strategy, Driving Decision, and Driving Operation, and instantiates each level with a dedicated module: the Strategic Reasoning Injector aligns VLM-generated strategy text with encoded ground-truth trajectory features and adds it to the ego-query; the Tactical Reasoning Integrator parses VLM output into four command categories (direction, lane, speed, emergency), one-hot encodes them, and splits them into reactive and regulatory features; the Hierarchical Trajectory Decoder samples a coarse latent trajectory conditioned on reactive commands and a fine latent trajectory conditioned on the coarse sample plus regulatory commands, decoding through two GRU streams. The authors report that the complete framework improves on both baselines and on prior VLM-assisted planners, reaching the best nuScenes average L2 error (0.48 m) and collision rate (0.15%) among compared methods, with closed-loop Bench2Drive driving score improving from 39.42 to 41.17 over VAD.

Load-bearing premise

The 30% figure comes from scoring predicted trajectories against recorded human driving on nuScenes, so the claim stands or falls on whether that open-loop score reflects how the vehicle would actually drive and avoid collisions on the road.

Editorial extensions

If this is right

  • Open-loop planning accuracy on nuScenes improves by roughly a third over VAD, and collision rate falls by a similar fraction under the standard evaluation protocol.
  • Closed-loop route completion on Bench2Drive improves: driving score rises from 39.42 to 41.17 and success rate from 10.00% to 11.36% over VAD.
  • VLM-generated strategy text and tactical commands act as an interpretable intermediate layer, so the final trajectory can be checked against the stated strategy and commands.
  • The same three modules can be added to different end-to-end planners (VAD and UniAD) and different VLMs (MiniCPM-Llama3-2.5V and Qwen-VL), indicating the benefit is not tied to one network.
  • Ablations attribute most of the collision-rate drop to the hierarchical decoder, while removing the strategic injector raises average L2 error by about 12% and collision rate by about 19% on the Bench2Drive validation set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If open-loop L2 error is weakly predictive of closed-loop safety, the defensible headline is a modest closed-loop gain alongside a large open-loop gain; the framework's practical value may lie in interpretability and human oversight rather than a literal 30% safety improvement.
  • Because tactical commands are categorical and one-hot, they form a natural hand-over interface: a downstream safety layer could veto a command, such as refusing a lane change, without modifying the network, an option the paper does not propose.
  • The closed-loop speed trick of reusing stored VLM text for similar scenes suggests that VLM throughput, not reasoning quality, is the current bottleneck; a distilled or event-triggered VLM would test how much of the gain depends on per-frame reasoning.
  • Feeding the VLM richer inputs than the front camera, such as BEV maps or object lists, might overcome the limited multi-view gains reported in the ablation, which could be a prompt or fusion artifact rather than a ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReAL-AD, a VLM-augmented end-to-end autonomous driving framework that organizes planning into three hierarchical levels (Driving Strategy, Driving Decision, Driving Operation). The Strategic Reasoning Injector encodes VLM-generated strategy text and aligns it with ground-truth trajectory features; the Tactical Reasoning Integrator converts VLM outputs into categorical driving commands; the Hierarchical Trajectory Decoder uses a two-stage variational latent space to refine coarse trajectories into fine ones. The authors report open-loop L2 and collision-rate improvements on nuScenes and Bench2Drive, closed-loop Bench2Drive Driving Score and Success Rate gains, and an ablation study. The central claim is that integrating ReAL-AD improves planning accuracy and safety by over 30%.

Significance. The idea of explicitly structuring VLM guidance into strategy, tactic, and operation levels is timely and connects with a growing line of work on reasoning-augmented end-to-end driving. The paper's modular design is clear, and the open-loop experiments cover two datasets with consistent baselines. The authors also provide a relatively detailed appendix with ablation studies on design choices. However, the significance of the work depends on whether the reported safety improvements are credible. The closed-loop evaluation is compromised by the memory-buffer mechanism in Appendix A.4, and the headline >30% claim rests almost entirely on open-loop metrics that are weak proxies for real driving performance. The closed-loop gains themselves are small. Thus the current evidence does not yet support the strong safety and real-world-applicability claims.

major comments (4)
  1. [Appendix A.4, Table 2] The closed-loop evaluation is invalidated as evidence for online VLM reasoning. Appendix A.4 states that a memory buffer stores environmental features and corresponding VLM policy text and driving instruction features from the open-loop evaluation dataset, and that during closed-loop evaluation the system retrieves the most similar stored frame and reuses its VLM outputs. This is a non-causal retrieval from the evaluation data, not a closed-loop test of the reasoning modules. Consequently, the closed-loop columns of Table 2 (Driving Score 39.42 to 41.17; Success Rate 10.00 to 11.36) cannot support the abstract's claim that the framework improves driving safety or real-world applicability.
  2. [Eq. (18)-(19), Sec. 3.6] The hierarchical KL loss is not well-defined. Eq. (18) writes DKL(qϕ(zc_curr∥zc_fut)) using a single bar, and Eq. (19) gives the formula for a Gaussian KL divergence, but the paper never defines the distributions qϕ(zc_curr) and qϕ(zc_fut), their parameters, or their relationship to the sampling distributions in Eqs. (10)-(13). The notation 'curr' and 'fut' is unexplained. As written, this loss cannot be implemented or reproduced from the text.
  3. [Abstract, Sec. 1, Tables 1-2] The headline claim of 'over 30% improvement in planning accuracy and safety' is not supported by the closed-loop results. On nuScenes, the reported average L2 reduction is 33% (0.72 to 0.48 m) and collision-rate reduction is 32% (0.22% to 0.15%) relative to VAD, but these are open-loop metrics. On Bench2Drive open-loop, the L2 reduction is 11% (0.94 to 0.84 m), while the closed-loop Driving Score improves only 4.4% (39.42 to 41.17) and Success Rate improves 1.36 points (10.00% to 11.36%). The paper should either substantiate the 'over 30% safety improvement' claim with closed-loop evidence or substantially rephrase the claim.
  4. [Table 3, Sec. 4.3] The ablation study does not consistently support the claim that each module contributes. In Table 3, setting 1 (SRI alone) achieves an average L2 of 0.89 m, the same as the full model in setting 7 (0.89 m), and setting 4 (SRI+TRI) has a higher average L2 (0.95 m) than SRI alone (0.89 m). The discussion of the Tactical Reasoning Integrator only compares settings 0 and 2, ignoring these interactions. This weakens the contribution claims made in the introduction and conclusions.
minor comments (5)
  1. [Sec. 4.2] The paragraph beginning 'To better assess the effectiveness of our framework' is repeated verbatim; please remove the duplicate.
  2. [Table 2] There is a typo in the VAD row: '1,47' should be '1.47'. Also, ensure consistent decimal separators throughout the table.
  3. [Eq. (22), Appendix A.3] The Bezier interpolation formula is garbled: 'Ck+1 (3 k)' does not display as a binomial coefficient. Please rewrite the equation with standard notation.
  4. [Sec. 4.4] The heading 'Dissusion' should read 'Discussion'.
  5. [Table 1 footnote] The footnote says methods using ego-vehicle status features are not considered in the ranking, but the table also uses '†' for UniAD metrics; please clarify the exact inclusion and ranking criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ReAL-AD's reasoning modules are trained by auxiliary distillation against the ground-truth trajectory and evaluated with external baselines, so the reported gains are empirical rather than self-referential.

full rationale

The paper's claimed improvements do not reduce to their inputs by construction. The Strategic Reasoning Injector computes F_strategy from frozen-VLM text via an MLP adapter and aligns it to an encoding of the ground-truth trajectory through a cosine similarity loss (Eq. 3); this is auxiliary supervision, not a prediction derived from the VLM text. The coarse trajectory target in Eq. 17 is explicitly generated from the ground-truth trajectory by Bezier curve fitting (Appendix A.3), and it is used only as an internal intermediate supervisory signal, not as the final claimed result. The final trajectory is produced by the hierarchical decoder and evaluated against external baselines and held-out benchmarks (VAD, UniAD, nuScenes, Bench2Drive), so the reported 33% L2 reduction and 32% collision-rate reduction are empirical comparisons, not fitted parameters renamed as predictions. The paper's only self-citation ([30]) appears in a general review of VLM driving agents and is not load-bearing for the method. Appendix A.4's memory buffer, which reuses cached VLM features during closed-loop evaluation, is a legitimate external-validity concern for the closed-loop Driving Score comparison, but it is an evaluation-protocol issue rather than a circular derivation: the open-loop L2 and collision-rate numbers that anchor the 30% claim are not generated through that buffer. No self-citation chain, imported uniqueness theorem, or ansatz-by-citation is present, so the central claims retain independent empirical content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the VLM's output quality and on the validity of open-loop metrics; no new physical entities are introduced.

free parameters (3)
  • loss weights λ0, λ1, λ2, λ3 = 1.0, 0.5, 0.5, 1.0
    Hand-set hyperparameters in Eq. 15; no sensitivity analysis is provided.
  • KL weights βc, βf = 0.5, 1.0
    Hand-set in Eq. 18; no sensitivity analysis is provided.
  • number of decoder layers = 2
    Chosen based on the ablation in Table 5; 3 layers hurt performance.
assumptions (3)
  • domain assumption VLM (MiniCPM-Llama3-2.5V or Qwen-VL) generates reliable driving strategies and tactical commands from a single front-view image.
    The entire SRI and TRI modules depend on this; no failure-rate statistics are reported, only regeneration on format mismatch (Appendix A.1, A.2).
  • domain assumption Open-loop L2 error and collision rate are meaningful proxies for driving safety.
    Used to support the 30% improvement claim; the paper's closed-loop results show much smaller gains (Table 2).
  • ad hoc to paper The three-tier cognitive model (strategy, decision, operation) is an appropriate decomposition for driving decision-making.
    Motivates the architecture; no independent evidence that these three levels match human cognition or are the right inductive bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving." pith.science (2026). https://pith.science/paper/P6BSV6WG

@misc{pith2026250712499,
  author       = {Pith},
  title        = {Pith review of: ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6BSV6WG}},
  note         = {Machine review of arXiv:2507.12499}
}
read the original abstract

End-to-end autonomous driving has emerged as a promising approach to unify perception, prediction, and planning within a single framework, reducing information loss and improving adaptability. However, existing methods often rely on fixed and sparse trajectory supervision, limiting their ability to capture the hierarchical reasoning process that human drivers naturally employ. To bridge this gap, we propose ReAL-AD, a Reasoning-Augmented Learning framework that structures decision-making in autonomous driving based on the three-tier human cognitive model: Driving Strategy, Driving Decision, and Driving Operation, where Vision-Language Models (VLMs) are incorporated to enhance situational awareness and structured reasoning across these levels. Specifically, we introduce: (1) the Strategic Reasoning Injector, which formulates high-level driving strategies by interpreting complex traffic contexts from VLM-generated insights; (2) the Tactical Reasoning Integrator, which refines strategic intent into interpretable tactical choices such as lane changes, overtaking, and speed adjustments; and (3) the Hierarchical Trajectory Decoder, which progressively translates tactical decisions into precise control actions for smooth and human-like trajectory execution. Extensive evaluations show that integrating our framework improves planning accuracy and safety by over 30%, making end-to-end autonomous driving more interpretable and aligned with human-like hierarchical reasoning. The project page can be found at: \href{https://4dvlab.github.io/project_page/realad}{\texttt{4dvlab.github.io/project\_page/realad}}

Figures

Figures reproduced from arXiv: 2507.12499 by the authors.

Figure 1
Figure 1. Comparison between end-to-end networks and hu￾man driving logic. (a) illustrates the workflow and limitations of the end-to-end autonomous driving system, while (b) depicts the structured decision-making process of a human driver. jectory supervision, which fails to replicate the structured cognitive reasoning inherent to human drivers – a process involving hierarchical understanding of contextual relation￾ships (e.… view at source ↗
Figure 2
Figure 2. Overall pipeline of ReAL-AD. Multi-view images are processed by the Scene Encoder to extract environmental features. The Strategic Reasoning Injector generates driving decisions using structured prompts and utilizes them to enhance ego-query. The Tactical Reasoning Integrator outputs reactive- and regulatory-level command features, which are then fed into the Hierarchical Trajectory Decoder. This decoder progressive… view at source ↗
Figure 3
Figure 3. Example VLM output. Given a front-view image and prompt, VLM generates corresponding driving strategy, reactive￾level and regulatory-level commands. greactive = edir ⊕ eemerg gregulatory = elane ⊕ espeed (9) greactive encodes the driving decision at the instanta￾neous reaction level, while gregulatory encodes the driv￾ing operation command after careful consideration. These two commands function at different layers … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Command prompt structure for tactical decision-making. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Command prompt structure for tactical decision-making with probability-based outputs. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Visualization of our method, depicting a scenario where an error is detected in the current lane while driving, requiring a lane [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Visualization of our methods, depicting a scenario where a pedestrian suddenly appears in front of you and you need to make [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Visualization of our methods, depicting a turn without a signal. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Visualization of our methods, depicting a turn with a signal. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Visualization of VLM-generated driving strategies and tacical commands, demonstrating their alignment with final planning. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 31 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 3, 6

    arXiv 2023

  3. [3]

    nuscenes: A mul- timodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. In CVPR, pages 11621–11631, 2020. 2, 6

    work page 2020

  4. [4]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 3

    arXiv 2023

  5. [5]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093–14100. IEEE, 2024. 3

    work page 2024

  6. [6]

    Vadv2: End-to-end vectorized autonomous driving via probabilistic planning

    Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243,

    arXiv

  7. [7]

    Ppad: Iterative interactions of prediction and planning for end-to-end autonomous driving

    Zhili Chen, Maosheng Ye, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Ppad: Iterative interactions of prediction and planning for end-to-end autonomous driving. 2

    work page

  8. [8]

    Deepseek llm: Scaling open-source language models with longtermism

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954,

    arXiv

Show all 61 references
  1. [9]

    Dualad: Disentangling the dynamic and static world for end-to-end driving

    Simon Doll, Niklas Hanselmann, Lukas Schneider, Richard Schulz, Marius Cordts, Markus Enzweiler, and Hendrik Lensch. Dualad: Disentangling the dynamic and static world for end-to-end driving. In CVPR, pages 14728–14737, 2024. 2

    2024

  2. [10]

    Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools, 2024

    Team GLM et al. Chatglm: A family of large language mod- els from glm-130b to glm-4 all tools, 2024. 3

    2024

  3. [11]

    End-to-end autonomous driving without costly modularization and 3d manual annotation

    Mingzhe Guo, Zhipeng Zhang, Yuan He, Ke Wang, and Lip- ing Jing. End-to-end autonomous driving without costly modularization and 3d manual annotation. arXiv preprint arXiv:2406.17680, 2024. 2, 7

    2024 arXiv

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 3

    2016

  5. [13]

    Safe local motion planning with self- supervised freespace forecasting

    Peiyun Hu, Aaron Huang, John Dolan, David Held, and Deva Ramanan. Safe local motion planning with self- supervised freespace forecasting. In CVPR, pages 12732– 12741, 2021. 7

    2021

  6. [14]

    St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based au- tonomous driving via spatial-temporal feature learning. In ECCV, pages 533–549. Springer, 2022. 1, 2, 7

    2022

  7. [15]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, pages 17853–17862, 2023. 1, 2, 6, 7

    2023

  8. [16]

    Drivegpt: Scaling autoregressive behavior models for driving

    Xin Huang, Eric M Wolff, Paul Vernaza, Tung Phan-Minh, Hongge Chen, David S Hayden, Mark Edmonds, Brian Pierce, Xinxin Chen, Pratik Elias Jacob, et al. Drivegpt: Scaling autoregressive behavior models for driving. arXiv preprint arXiv:2412.14415, 2024. 3

    2024 arXiv

  9. [17]

    Mak- ing large language models better planners with reasoning- decision alignment

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Ze- qun Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Mak- ing large language models better planners with reasoning- decision alignment. In European Conference on Computer Vision, pages 73–90. Springer, 2024

    2024

  10. [18]

    Emma: End-to-end multimodal model for autonomous driving

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262,

    arXiv

  11. [19]

    Bench2drive: Towards multi-ability benchmark- ing of closed-loop end-to-end autonomous driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmark- ing of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877, 2024. 2, 6

    2024 arXiv

  12. [20]

    Vad: Vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, pages 8340– 8350, 2023. 1, 2, 6, 7, 15

    2023

  13. [21]

    Senna: Bridging large vision-language mod- els and end-to-end autonomous driving

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xing- gang Wang. Senna: Bridging large vision-language mod- els and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313, 2024. 3, 7

    arXiv

  14. [22]

    Differentiable raycasting for self-supervised occupancy forecasting

    Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Differentiable raycasting for self-supervised occupancy forecasting. In ECCV, pages 353–369. Springer, 2022. 7

    2022

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 3

    2023

  16. [24]

    Does end-to-end autonomous driving really need perception tasks? arXiv preprint arXiv:2409.18341, 2024

    Peidong Li and Dixiao Cui. Does end-to-end autonomous driving really need perception tasks? arXiv preprint arXiv:2409.18341, 2024. 2

    2024 arXiv

  17. [25]

    Enhancing end-to-end autonomous driving with latent world model

    Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.08481, 2024. 2

    2024 arXiv

  18. [26]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024. 3

    2024

  19. [27]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving

    Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139, 2024. 2

    2024 arXiv

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 3

    2023

  21. [29]

    Mtd-gpt: A multi-task decision-making gpt model for autonomous driving at unsignalized intersections

    Jiaqi Liu, Peng Hang, Xiao Qi, Jianqiang Wang, and Jian Sun. Mtd-gpt: A multi-task decision-making gpt model for autonomous driving at unsignalized intersections. In 2023 IEEE 26th International Conference on Intelligent Trans- portation Systems (ITSC), pages 5154–5161. IEEE, 2023. 3

    2023

  22. [30]

    Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving

    Yuhang Lu, Yichen Yao, Jiadong Tu, Jiangnan Shao, Yuexin Ma, and Xinge Zhu. Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 5838–5846, 2025. 3

    2025

  23. [31]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In ECCV, pages 403–420. Springer, 2024. 3

    2024

  24. [32]

    Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023. 3

    2023 arXiv

  25. [33]

    A language agent for autonomous driving

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. arXiv preprint arXiv:2311.10813, 2023. 1, 3

    2023 arXiv

  26. [34]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In CVPR, pages 14760–14769, 2024. 1, 3, 7

    2024

  27. [35]

    Bezier curve fitting

    Tim A Pastva. Bezier curve fitting . PhD thesis, Monterey, California. Naval Postgraduate School, 1998. 6, 12

    1998

  28. [36]

    Fasionad: Fast and slow fusion thinking systems for human-like autonomous driving with adaptive feedback

    Kangan Qian, Zhikun Ma, Yangfan He, Ziang Luo, Tianyu Shi, Tianze Zhu, Jiayin Li, Jianhui Wang, Ziyu Chen, Xiao He, et al. Fasionad: Fast and slow fusion thinking systems for human-like autonomous driving with adaptive feedback. arXiv preprint arXiv:2411.18013, 2024. 3

    2024 arXiv

  29. [37]

    Languagempc: Large language models as decision makers for autonomous driving

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving. arXiv preprint arXiv:2310.03026, 2023. 3

    arXiv

  30. [38]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, pages 15120–15130, 2024

    2024

  31. [39]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In ECCV, pages 256–274. Springer, 2024. 3

    2024

  32. [40]

    Sparsedrive: End-to-end au- tonomous driving via sparse scene representation

    Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Hao- ran Wu, and Sifa Zheng. Sparsedrive: End-to-end au- tonomous driving via sparse scene representation. arXiv preprint arXiv:2405.19620, 2024. 1, 2

    2024 arXiv

  33. [41]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 3

    2023 arXiv

  34. [42]

    Tokenize the world into object-level knowledge to address long-tail events in autonomous driving

    Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv preprint arXiv:2407.00959, 2024. 3

    2024 arXiv

  35. [43]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 1, 3, 7

    2024 arXiv

  36. [44]

    Scene as occupancy

    Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In ICCV, pages 8406–8415, 2023. 7

    2023

  37. [45]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024. 3

    2024 arXiv

  38. [46]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 1, 3

    2023 arXiv

  39. [47]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving. arXiv preprint arXiv:2312.09245, 2023. 1, 3

    2023

  40. [48]

    Dilu: A knowledge-driven approach to au- tonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to au- tonomous driving with large language models. arXiv preprint arXiv:2309.16292, 2023. 3

    2023 arXiv

  41. [49]

    Para-drive: Parallelized architecture for real- time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real- time autonomous driving. In CVPR, pages 15449–15458,

  42. [50]

    Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line. Advances in Neural Information Processing Systems , 35:6119–6132, 2022. 7

    2022

  43. [51]

    Openemma: Open-source multimodal model for end-to-end autonomous driving

    Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. arXiv preprint arXiv:2412.15208, 2024. 3

    2024 arXiv

  44. [52]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision

    Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446, 2024. 1, 3, 7

    2024 arXiv

  45. [53]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 2024. 1, 3

    2024

  46. [54]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 3

    2024 arXiv

  47. [55]

    Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024. 3, 6, 15

    2024

  48. [56]

    End-to-end inter- pretable neural motion planner

    Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end inter- pretable neural motion planner. In CVPR, pages 8660–8669,

  49. [57]

    Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes

    Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430, 2023. 7

    2023 arXiv

  50. [58]

    When driving in the current sce- nario

    Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end au- tonomous driving. In ECCV, pages 87–104. Springer, 2024. 1, 2, 5, 7 Appendix In the appendix, we first present additional model details, followed by analytical experiments to...

    2024

  51. [59]

    You must provide a probability distribution for each command category, ensuring that the probabilities for each category sum up to 1

  52. [60]

    Use format strictly as shown below without any extra text or explanations

  53. [61]

    Emergency Control

    Refer to the provided examples to guide your output format and content. Command Types Emergency Control • EMERGENCY BRAKE: Immediately apply emergency brakes to avoid a collision or respond to a sudden hazard. • PARK: Bring the vehicle to a complete stop and park, suitable for...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.