Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An end-to-end driving planner that casts trajectory prediction as language generation reports the lowest displacement error and collision rate on the nuScenes open-loop benchmark.

desk verdict A promising architecture undermined by an unresolved ego-status contradiction in the evaluation protocol; worth refereeing but not trustworthy as-is. read the letter →

arxiv 2501.08861 v1 pith:RXUERXTU submitted 2025-01-15 cs.CV

Tengpeng Li , Hanli Wang , Xianfei Li , Wenlong Liao , Tao He , Pai Peng This is my paper · ORCID
classification cs.CV
keywords autonomousdrivingvision-languagepre-trainingbird's-eyeviewtrajectoryplanningautoregressivegenerationend-to-endnuScenes3Dperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces GPVL, an end-to-end autonomous driving model that frames trajectory planning as a language generation problem. It argues that aligning bird's-eye-view perception features with text descriptions, then feeding them to a language model, lets the system produce safe future trajectories. On the nuScenes open-loop planning benchmark, GPVL reports the lowest L2 displacement error (0.21 m at 1 s, 0.43 m average) and lowest average collision rate (0.14%) among 13 compared methods, and it also shows better zero-shot transfer to a new city and stronger robustness to weather noise than UniAD and VAD. The intended value is that marrying 3D perception with linguistic reasoning improves both safety and generalization in end-to-end driving.

What carries the argument

The central mechanism is a group-wise vision-language alignment combined with an autoregressive trajectory generator. The alignment treats detection, motion, map, and global BEV features as visual 'groups' and matches each to a corresponding text description through a contrastive loss with a learnable temperature and similarity function. The trajectory generator then works like a captioning model: it takes the aligned BEV feature, a 2D scene caption, and a navigation instruction, and produces the future trajectory as a sequence of tokens in natural language, maximizing the likelihood of each token given the previous ones.

What would settle it

Run the authors' released code on the nuScenes val split with the same training protocol; if the reported 0.43 m average L2 and 0.14% collision rate are not reproduced, the central performance claim fails.

Watch

Extended reading notes

Core claim

The central claim is that planning in autonomous driving can be improved by formulating it as generative language modeling over 3D visual features. GPVL first extracts a bird's-eye-view feature map with a pretrained BEVformer and decodes detection, motion, and map features. A 3D-vision language pre-training module uses a group-wise contrastive alignment (detection, motion, map, and global groups) to project these visual features and environmental text descriptions into a shared space. A cross-modal language model, implemented with BERT, then receives the global visual feature, a 2D scene caption, and a navigation prompt, and autoregressively emits a trajectory caption, trained with a cross-entropy loss. The paper reports that this design yields the lowest L2 displacement error (0.21 m at 1 s, 0.43 m average) and lowest average collision rate (0.14%) on nuScenes open-loop planning, with better zero-shot city transfer and robustness to weather noise than UniAD and VAD.

Load-bearing premise

The paper's performance numbers rest on treating a bidirectional text-understanding model (BERT) as a next-token generator for trajectories, a capability the paper assumes but does not demonstrate.

Editorial extensions

If this is right

  • Trajectory planning becomes a sequence-generation task: the same training and inference machinery used for text generation (greedy search, beam search, prompt-conditioning) can be applied to driving.
  • The group-wise vision-language alignment can act as a pretraining objective for other end-to-end driving components, improving them even before the generative planner is trained.
  • Language-grounded planners are less prone to shortcut learning on the straight-road majority of nuScenes, as shown by their stronger metrics on turn-left and turn-right subsets.
  • The reported latency (198 ms) and throughput (5.1 FPS) are compatible with near-real-time deployment, a prerequisite for practical use in future autonomous driving systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If trajectory-as-text becomes an interchange format, plans could be inspected, amended, or constrained purely through natural-language prompts, which regression-head planners do not support.
  • The same group-wise alignment scheme could be reused in other embodied settings where 3D scene features and language coexist, such as indoor robot navigation or drone planning.
  • A natural follow-up is ablating the 2D scene captioner: the paper leaves open whether the aligned 3D features alone would carry the language grounding or whether the 2D caption is essential.
  • Testing the pipeline with a decoder-only causal language model would clarify how much of the reported performance depends on the specific bidirectional encoder, since causal generation is the standard way to evaluate autoregressive trajectory sampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes GPVL, an end-to-end autonomous driving framework that combines a BEVFormer-based perception backbone with a 3D-vision language pre-training module (group-wise contrastive alignment) and a cross-modal language model that generates driving decisions and trajectories in an autoregressive text format. Experiments on nuScenes report the lowest open-loop planning L2 errors (0.21 m at 1 s, 0.39 m at 2 s, 0.69 m at 3 s, 0.43 m average) and the lowest average collision rate (0.14%) among 13 baselines, along with improved zero-shot city transfer and robustness to image corruptions compared with UniAD and VAD. Ablations show that each proposed component contributes to the final performance.

Significance. If the results hold, GPVL would be a meaningful step toward integrating language-based reasoning with 3D visual perception for trajectory planning, and the paper provides a useful comparison across many non-autoregressive and LLM-injected baselines. The release of code is a positive feature. However, the central SOTA claim is currently not supported by a clearly specified experimental protocol: there is an unresolved contradiction about whether ego status is used, and the generative decoding mechanism is underspecified because BERT is treated as an autoregressive decoder without explaining how causal generation is implemented. These issues are load-bearing for the reported planning results and for reproducibility.

major comments (4)
  1. [§Planning via Cross-modal Language Model; Table 1 caption; §Comparison with State-of-the-art Methods] The paper directly contradicts itself on whether ego status information is used. The Nav template in the 'Planning via Cross-modal Language Model' section reads 'The box of ego-car is [cx, cy, cz, cw, ch, cl]' and is fed into the language model in Eq. (8). Yet the Experiment section and the Table 1 caption state that 'the ego status information of GPVL is deactivated' to prevent shortcut learning, citing BEV-Planner. The manuscript never specifies how deactivation is implemented (e.g., removing the box tokens, masking them, or replacing them with a placeholder) or whether the results in Tables 1-5 were obtained with or without the ego box. If the box remains, the comparison with UniAD and VAD is unfair and the very low L2/collision values may reflect the shortcut documented by BEV-Planner; if it was removed, the method description is inaccurate and the actual input format is undefined. This must be resolved before the central SOTA claim can be accepted.
  2. [§Planning via Cross-modal Language Model, Eqs. (8)-(10); §Implementation Details] The cross-modal language model is said to use the BERT structure, but BERT is a bidirectional encoder, not a causal decoder. Eq. (9) defines an autoregressive factorization p(w_t|w_{1:t-1}) and Eq. (10) trains with teacher forcing, yet the paper gives no information about causal masking, positional encodings, or how bidirectional attention is converted into sequential generation. In addition, the trajectory tokenization is completely unspecified: continuous future waypoints must be discretized into a vocabulary, but the manuscript never defines the coordinate frame, quantization step, vocabulary size, or the textual template of the reference caption G_t. Without these details, the generative planning mechanism is not reproducible and the reported L2 values cannot be independently checked.
  3. [Introduction; §Dataset; §Comparison with State-of-the-art Methods] The comparison with baselines may be confounded by additional data and supervision. The 2D scene captioning model is 'trained and tested on the Ominidrive-nuScenes dataset', and the 3D-vision language pre-training also uses nuScenes-derived data. Since UniAD, VAD, and the other baselines are trained without this captioning or VL-pretraining supervision, the gains shown in Tables 1, 4, and 5 cannot be attributed solely to the architectural contributions unless the paper specifies the exact train/val splits, confirms that no nuScenes validation scenes were used during captioner or VL pretraining, and ideally reports a variant that removes the additional data. This is a fairness and reproducibility issue for the SOTA claim.
  4. [Table 1; Table 4; Table 5] The headline results are reported as single runs without error bars or multiple seeds. For example, GPVL's average collision rate is 0.14% versus 0.22% for VAD and 0.31% for UniAD, and some zero-shot and robustness differences are only a few hundredths of a meter. Without variance estimates, it is not possible to determine whether these differences are statistically meaningful. The paper should report standard deviations over at least three seeds or otherwise quantify evaluation uncertainty.
minor comments (4)
  1. [Eqs. (5)-(6)] The similarity function s is used generically in Eq. (5) but is only fully defined for the detection group in Eq. (6); please provide the analogous definitions for the motion, map, and global groups and clarify the roles of W1 and W2.
  2. [References] The reference list contains LaTeX encoding artifacts, for example 'BEVformer: Learning bird¡¯s-eye-view representation from multi-camera images via spatiotemporal transformers'; these should be cleaned up before publication.
  3. [Table 1 caption] The caption states that 'GPVL achieves the highest score on most evaluation metrics' although lower L2 and collision values are better; please rephrase to avoid ambiguity.
  4. [§Planning via Cross-modal Language Model] The decoding and evaluation pipeline should be described end to end: greedy search with size 1 produces text, but the manuscript does not explain how the generated text is parsed back into continuous waypoints for L2 evaluation; this step is essential for reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No prediction in GPVL reduces by construction to a fitted input or to a self-cited theorem; the ego-box/evaluation contradiction is an experimental-validity concern, not a circular derivation.

full rationale

GPVL's derivation chain is self-contained. The perception features are produced by a pre-trained BEVformer and supervised detection, motion, and map transformers; the 3D-vision language alignment is trained with a contrastive loss on ground-truth labels but is evaluated on inferred perception features; and the trajectory is generated by a standard autoregressive cross-entropy loss on ground-truth trajectory captions. No predicted quantity is defined as a fitted value, and no load-bearing claim rests on a uniqueness theorem from the authors' prior work. The self-citations that appear (Li et al. 2022b, 2022c for captioning objectives) are routine methodological references and do not carry the central argument. The scene-captioning model is trained and tested on the OmniDrive-nuScenes dataset, which overlaps with the nuScenes evaluation distribution; this is a data-domain reuse and generalization concern, not a by-construction equivalence. There is also an unresolved protocol inconsistency: Eq. 8's Nav template explicitly contains 'The box of ego-car is [cx, cy, cz, cw, ch, cl]', while Table 1's caption states 'the ego status information of GPVL is deactivated for a fair comparison.' If the ego box remained in the prompt, the open-loop L2 and collision scores could be inflated by copying the current ego state, and if it was removed, the method description is inaccurate. This threatens the validity of the SOTA comparison, but it is an experimental-protocol flaw rather than a circular derivation: the paper's equations do not by themselves equate the predicted trajectory with the ego box, and the paper explicitly claims the ego status was deactivated. Accordingly, the circularity score is low, and the inconsistency should be weighed as a correctness and reproducibility risk rather than as a circular step.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

GPVL introduces no new physical entities or forces. All model components are known architectures (BEVformer, BERT, BLIP) combined in a new way. The main load-bearing assumptions are that BEV features contain enough 3D information, that GT-to-text alignment transfers to inference, and that BERT can be used as an autoregressive decoder, the latter being unsubstantiated.

free parameters (2)
  • Learnable temperature tau in contrastive loss = learned
    Eq. 5, standard in CLIP-style contrastive learning; trained with the model, not a hand-set constant.
  • Learnable weights W1 and W2 in group-wise similarity = learned
    Eq. 6, introduced ad hoc for the max-pooled similarity function; learned during training.
assumptions (5)
  • domain assumption InfoNCE-style contrastive loss aligns visual and textual features in a shared semantic space
    Eq. 5; the paper assumes this alignment transfers to planning.
  • domain assumption BEV features extracted by pre-trained BEVformer contain sufficient 3D spatial information for driving
    Used as the perception backbone; not re-derived.
  • domain assumption Ground-truth object, motion, and map labels can be faithfully converted to text prompts
    Eq. 3, the textual features are built from GT labels via templates.
  • ad hoc to paper BERT can act as an autoregressive trajectory generator when fed visual and prompt features
    Eqs. 8-10, no causal masking or architectural modification is described; a critical assumption.
  • ad hoc to paper Discretized trajectory-as-text with cross-entropy loss yields accurate continuous trajectories
    Eq. 10 and greedy search; vocabulary and decoding details are not given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving." pith.science (2026). https://pith.science/paper/RXUERXTU

@misc{pith2026250108861,
  author       = {Pith},
  title        = {Pith review of: Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RXUERXTU}},
  note         = {Machine review of arXiv:2501.08861}
}
read the original abstract

Autonomous driving is a challenging task that requires perceiving and understanding the surrounding environment for safe trajectory planning. While existing vision-based end-to-end models have achieved promising results, these methods are still facing the challenges of vision understanding, decision reasoning and scene generalization. To solve these issues, a generative planning with 3D-vision language pre-training model named GPVL is proposed for end-to-end autonomous driving. The proposed paradigm has two significant aspects. On one hand, a 3D-vision language pre-training module is designed to bridge the gap between visual perception and linguistic understanding in the bird's eye view. On the other hand, a cross-modal language model is introduced to generate holistic driving decisions and fine-grained trajectories with perception and navigation information in an auto-regressive manner. Experiments on the challenging nuScenes dataset demonstrate that the proposed scheme achieves excellent performances compared with state-of-the-art methods. Besides, the proposed GPVL presents strong generalization ability and real-time potential when handling high-level commands in various scenarios. It is believed that the effective, robust and efficient performance of GPVL is crucial for the practical application of future autonomous driving systems. Code is available at https://github.com/ltp1995/GPVL

Figures

Figures reproduced from arXiv: 2501.08861 by the authors.

Figure 1
Figure 1. (a) The existing end-to-end autonomous driving [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of GPVL for autonomous driving. The framework is divided into three parts: (1) the backbone includes a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. To provide a comprehensive understanding of driving [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Visualized comparison of the proposed GPVL, VAD and the ground-truth on the nuScenes dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.

Reference graph

Works this paper leans on

48 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  4. [4]

    Bai, Y.; Wu, D.; Liu, Y.; Jia, F.; Mao, W.; Zhang, Z.; Zhao, Y.; Shen, J.; Wei, X.; Wang, T.; et al. 2024. Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving? arXiv preprint arXiv:2405.18361

    arXiv 2024

  5. [5]

    M.; and Nasrabadi, N

    Bishop, C. M.; and Nasrabadi, N. M. 2006. Pattern recognition and machine learning, volume 4. Springer

    work page 2006

  6. [6]

    Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Chen, X.; Choromanski, K.; Ding, T.; Driess, D.; Dubey, A.; Finn, C.; et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818

    arXiv 2023

  7. [7]

    H.; Vora, S.; Liong, V

    Caesar, H.; Bankiti, V.; Lang, A. H.; Vora, S.; Liong, V. E.; Xu, Q.; Krishnan, A.; Pan, Y.; Baldan, G.; and Beijbom, O. 2020. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 11621--11631

    work page 2020

  8. [8]

    Chen, S.; Jiang, B.; Gao, H.; Liao, B.; Xu, Q.; Zhang, Q.; Huang, C.; Liu, W.; and Wang, X. 2024. VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning. arXiv preprint arXiv:2402.13243

    arXiv 2024

Show all 48 references
  1. [9]

    E.; et al

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6

    2023

  2. [10]

    M.; and Gaidon, A

    Codevilla, F.; Santana, E.; L \'o pez, A. M.; and Gaidon, A. 2019. Exploring the limitations of behavior cloning for autonomous driving. In ICCV, 9329--9338

    2019

  3. [11]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 4171--4186

    2019

  4. [12]

    Ding, X.; Han, J.; Xu, H.; Liang, X.; Zhang, W.; and Li, X. 2024. Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models. In CVPR, 13668--13677

    2024

  5. [13]

    Dosovitskiy, A.; Ros, G.; Codevilla, F.; Lopez, A.; and Koltun, V. 2017. CARLA: An open urban driving simulator. In CoRL, 1--16

    2017

  6. [14]

    S.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al

    Driess, D.; Xia, F.; Sajjadi, M. S.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378

    2023 arXiv

  7. [15]

    Gu, J.; Hu, C.; Zhang, T.; Chen, X.; Wang, Y.; Wang, Y.; and Zhao, H. 2023. Vip3d: End-to-end visual trajectory prediction via 3d agent queries. In CVPR, 5496--5506

    2023

  8. [16]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR, 770--778

    2016

  9. [17]

    Hu, P.; Huang, A.; Dolan, J.; Held, D.; and Ramanan, D. 2021. Safe local motion planning with self-supervised freespace forecasting. In CVPR, 12732--12741

    2021

  10. [18]

    Hu, S.; Chen, L.; Wu, P.; Li, H.; Yan, J.; and Tao, D. 2022. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 533--549

    2022

  11. [19]

    Hu, Y.; Yang, J.; Chen, L.; Li, K.; Sima, C.; Zhu, X.; Chai, S.; Du, S.; Lin, T.; Wang, W.; et al. 2023. Planning-oriented autonomous driving. In CVPR, 17853--17862

    2023

  12. [20]

    Jia, X.; Wu, P.; Chen, L.; Xie, J.; He, C.; Yan, J.; and Li, H. 2023. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In CVPR, 21983--21994

    2023

  13. [21]

    Jiang, B.; Chen, S.; Xu, Q.; Liao, B.; Chen, J.; Zhou, H.; Zhang, Q.; Liu, W.; Huang, C.; and Wang, X. 2023. Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, 8340--8350

    2023

  14. [22]

    LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature, 521(7553): 436--444

    2015

  15. [23]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 19730--19742

    2023

  16. [24]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022 a . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 12888--12900

    2022

  17. [25]

    Li, Q.; Li, T.; Wang, H.; and Chen, C. W. 2022 b . Taking an emotional look at video paragraph captioning. arXiv preprint arXiv:2203.06356

    2022 arXiv

  18. [26]

    Li, T.; Wang, H.; He, B.; and Chen, C. W. 2022 c . Knowledge-enriched attention network with group-wise semantic for visual storytelling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8634--8645

    2022

  19. [27]

    Li, Z.; Wang, W.; Li, H.; Xie, E.; Sima, C.; Lu, T.; Qiao, Y.; and Dai, J. 2022 d . Bevformer: Learning bird¡¯s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 1--18

    2022

  20. [28]

    Li, Z.; Yu, Z.; Lan, S.; Li, J.; Kautz, J.; Lu, T.; and Alvarez, J. M. 2024. Is ego status all you need for open-loop end-to-end autonomous driving? In CVPR, 14864--14873

    2024

  21. [29]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. In NeurIPS, 12021--12036

    2023

  22. [30]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  23. [31]

    Mao, J.; Qian, Y.; Zhao, H.; and Wang, Y. 2023. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415

    2023 arXiv

  24. [32]

    G.; Velipasalar, S.; and Ren, L

    Pan, C.; Yaman, B.; Nesti, T.; Mallik, A.; Allievi, A. G.; Velipasalar, S.; and Ren, L. 2024. VLP: Vision Language Planning for Autonomous Driving. In CVPR, 14760--14769

    2024

  25. [33]

    Pan, Y.; Yao, T.; Li, Y.; and Mei, T. 2020. X-linear attention networks for image captioning. In CVPR, 10971--10980

    2020

  26. [34]

    Prakash, A.; Chitta, K.; and Geiger, A. 2021. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 7077--7087

    2021

  27. [35]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748--8763

    2021

  28. [36]

    L.; Liu, Y.; and Li, H

    Shao, H.; Hu, Y.; Wang, L.; Song, G.; Waslander, S. L.; Liu, Y.; and Li, H. 2024. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, 15120--15130

    2024

  29. [37]

    Tian, X.; Gu, J.; Li, B.; Liu, Y.; Hu, C.; Wang, Y.; Zhan, K.; Jia, P.; Lang, X.; and Zhao, H. 2024. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models. arXiv preprint arXiv:2402.12289

    2024 arXiv

  30. [38]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  31. [39]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. In NeurIPS, 6000--6010

    2017

  32. [40]

    Wang, G.; Xie, Y.; Jiang, Y.; Mandlekar, A.; Xiao, C.; Zhu, Y.; Fan, L.; and Anandkumar, A. 2023 a . Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291

    2023 arXiv

  33. [41]

    Wang, S.; Yu, Z.; Jiang, X.; Lan, S.; Shi, M.; Chang, N.; Kautz, J.; Li, Y.; and Alvarez, J. M. 2024. OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning. arXiv preprint arXiv:2405.01533

    2024 arXiv

  34. [42]

    Wang, W.; Xie, J.; Hu, C.; Zou, H.; Fan, J.; Tong, W.; Wen, Y.; Wu, S.; Deng, H.; Li, Z.; et al. 2023 b . Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245

    2023

  35. [43]

    Wen, L.; Fu, D.; Li, X.; Cai, X.; Ma, T.; Cai, P.; Dou, M.; Shi, B.; He, L.; and Qiao, Y. 2023. Dilu: A knowledge-driven approach to autonomous driving with large language models. arXiv preprint arXiv:2309.16292

    2023 arXiv

  36. [44]

    Wu, P.; Jia, X.; Chen, L.; Yan, J.; Li, H.; and Qiao, Y. 2022. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. In NeurIPS, 6119--6132

    2022

  37. [45]

    K.; Li, Z.; and Zhao, H

    Xu, Z.; Zhang, Y.; Xie, E.; Zhao, Z.; Guo, Y.; Wong, K. K.; Li, Z.; and Zhao, H. 2023. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. arXiv preprint arXiv:2310.01412

    2023 arXiv

  38. [46]

    Yu, E.; Zhao, L.; Wei, Y.; Yang, J.; Wu, D.; Kong, L.; Wei, H.; Wang, T.; Ge, Z.; Zhang, X.; et al. 2023. Merlin: Empowering multimodal llms with foresight minds. arXiv preprint arXiv:2312.00589

    2023 arXiv

  39. [47]

    Zeng, W.; Luo, W.; Suo, S.; Sadat, A.; Yang, B.; Casas, S.; and Urtasun, R. 2019. End-to-end interpretable neural motion planner. In CVPR, 8660--8669

    2019

  40. [48]

    Zhai, J.-T.; Feng, Z.; Du, J.; Mao, Y.; Liu, J.-J.; Tan, Z.; Zhang, Y.; Ye, X.; and Wang, J. 2023. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.