Pith. sign in

REVIEW 3 major objections 6 minor 48 references

NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NetRoller lets a frozen vision-language model improve a real-time driving model without slowing it down.

desk verdict Useful engineering for asynchronous VLM-to-driving coupling, but the headline numbers are mis-stated and the paper's own ablation leaves the VLM's causal contribution unproven. read the letter →

arxiv 2506.14589 v2 pith:27DLF7JZ submitted 2025-06-17 cs.RO

Ren Xin , Hongji Liu , Xiaodong Mei , Wenru Liu , Maosheng Ye , Zhili Chen , Jun Ma This is my paper · ORCID
classification cs.RO
keywords end-to-endautonomousdrivingvision-languagemodelsasynchronousmodelintegrationadapterlatentfeaturecollectionqueryshiftnuScenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the hidden reasoning features of a large vision-language model can be harvested early, translated, and injected into a specialized end-to-end driving model so the driver gains human-like situational awareness while still running at its own native frequency. The reported result is a 16.71% reduction in average collision rate and a 12.46% improvement in trajectory similarity on nuScenes. If true, this offers a practical interface for combining slow general reasoning with fast specialized control, without waiting for the VLM to finish generating text.

What carries the argument

The load-bearing object is the Roller Embedding vector $E_R = \mathrm{Linear}(\mathrm{QFormer}([h^{-l}_{:i} + PE^{-l}_{layer}, E_N]))$, produced from the first $i$ tokens' last $l$ layer hidden states of the VLM, a layer positional embedding, and a learnable Nonsense Embedding $E_N$. This compressed representation is then added to or cross-attended with the specialized model's BEV queries or features. The Nonsense Embedding is designed to keep attention stable when the upstream VLM produces no output, and the layer positional embedding lets the translator know which transformer layer each collected vector came from.

What would settle it

Run closed-loop simulation with the same NetRoller configurations and compare collision rate and trajectory similarity against VAD base; if the 16.71% collision reduction or the 12.46% similarity gain disappears under rollout, the open-loop results do not establish the claimed real-time benefit. A separate probe would be to mask the VLM's visual input and test whether planning-relevant information remains decodable from the first-token features; if not, the translation premise fails.

Watch

Extended reading notes

Core claim

NetRoller's central claim is that a frozen VLM can improve a specialized driving model through a three-stage adapter: collect latent vectors from the first token across all transformer layers (early stop), translate them with a lightweight QFormer into low-dimensional 'Roller Embeddings', and distribute those embeddings into the SM either by shifting the BEV query (Query Shift, best for safety) or by shifting the BEV feature (Feature Shift, best for trajectory similarity). In the paper's own numbers, the specialized model achieves a 16.71% reduction in collision rate with Query Shift and a 12.46% improvement in trajectory similarity with Feature Shift on nuScenes, while the C1 collection mode cuts information-collection latency by roughly 98%.

Load-bearing premise

The argument assumes that the hidden-layer feature vectors of the first token genuinely encode task-relevant driving instructions that a lightweight QFormer can translate into the specialized model's BEV space; if those latents carry no usable driving signal, the adapter has nothing to transfer.

Editorial extensions

If this is right

  • A real-time driving model can receive LLM/VLM reasoning as a low-latency prior, so the slow model does not need to finish a sentence before the vehicle reacts.
  • Query Shift (adding the embedding to BEV queries) improves perception and safety metrics, while Feature Shift (transformer-based value shift on BEV features) improves human-like trajectory similarity, suggesting different distribution sites serve different driving subtasks.
  • The first-token/all-layer collection mode performs comparably to collecting all tokens across all layers, implying the first token carries much of the planning-relevant latent instruction.
  • The GM enhancement pays off even when the GM's visual input is masked, indicating the trained adapter can act as a stable conditional prior rather than a fragile online dependency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper only measures open-loop planning on nuScenes, the most direct test of the claim would be closed-loop simulation or on-vehicle evaluation; a safety gain that holds in closed loop would make the asynchronous GM-SM architecture a credible production pattern.
  • The QA-type analysis suggests GM instructions are not uniformly useful: planning, prediction, and behavior questions help while perception questions sometimes hurt, implying the interface could be improved by selecting or weighting GM prompts, a step the paper leaves implicit.
  • The same collection-translation-distribution pattern could transfer to other sensor modalities or other specialized real-time controllers, not just BEV-based driving, since the adapter only touches the SM's latent stream.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. NetRoller proposes a three-stage adapter for interfacing a general model (a DriveLM-fine-tuned LLaMA-2-7B/CLIP VLM) with a specialized end-to-end driving model (VAD). Stage 1 collects hidden-layer features from the first token with early stopping (C1), reducing latency relative to full token generation; Stage 2 translates these features into SM-space "Roller Embeddings" via a QFormer with learnable queries, a nonsensical embedding, and layer positional embeddings (Eq. 5); Stage 3 distributes the result by shifting the BEV query (Query Shift) or the BEV feature (Feature Shift), either through a transformer or by adding ER as a bias (Eqs. 6-8). The SM is fine-tuned for a few epochs. Experiments on nuScenes report that NetRoller-Bias-C1-PEl reduces the average collision rate by 16.71% and improves NDS/mAP, while NetRoller-TF-C0 improves trajectory L2 similarity by 12.46%, with a 98% reduction in information collection latency (Table IV). Ablations cover collection modes (C0/C1/C-1), layer embeddings, freezing strategies, QA types, and inference with masked GM inputs.

Significance. If the quantitative claims hold, NetRoller is a practical answer to the asynchronous GM-SM integration problem: it keeps the SM at its native frequency, collects only the first token's all-layer states, and shifts BEV queries/features with a lightweight module. The paper has clear strengths: the adapter is evaluated on the external nuScenes benchmark with an in-house rerun of the VAD baseline; the latency reduction (C1: 0.0667 s vs C0/C-1: about 3.5 s in Table IV) is machine-measured; the configurations are fully tabulated; the code is stated to be released; and the frame-wise and QA-type analyses are useful engineering evidence. The reported effect sizes are, however, overstated as written (see major comments), and the ablations do not yet separate the contribution of the VLM's driving-relevant content from the learned Bias plus SM fine-tuning. With corrected statistics and a proper control for the information content of the collected features, the significance of the contribution would be solid; as written, the causal claim is stronger than the evidence supports.

major comments (3)
  1. [Section V.B, Table III; Section VI] The two headline percentages do not match Table III under the standard definition of a relative reduction. For collision: VAD base* average is 0.1900 and NetRoller-Bias-C1-PEl is 0.1628, so the reduction is (0.1900-0.1628)/0.1900 = 14.3%, not the stated 16.71%; the printed value equals 0.1900/0.1628 - 1, which is a ratio of means, not a reduction. For trajectory L2: 0.7175 to 0.6380 is an 11.1% decrease, not the stated 12.46% (= 0.7175/0.6380 - 1). Section VI repeats both numbers. Please recompute the percentages, state clearly that the two numbers come from different configurations (Bias-C1-PEl and TF-C0, respectively), and note that TF-C0 slightly increases average collision (0.1923 vs 0.1900).
  2. [Section V.D.c, Fig. 6(b.3)/(c.3)] The masked-GM ablation does not secure the load-bearing premise that the first-token, all-layer hidden states of Eq. (5) carry driving-relevant instructions. When all input features to the GM are masked, the paper reports that the Bias configuration "maintains competitive performance across most metrics" (Fig. 6(b.3)); the offered explanation, that the GM allows the SM to focus on reasoning rather than irrelevant details, cannot follow from an experiment in which the GM is fed masked inputs, because the scene-specific visual information and hence any scene-dependent driving instruction are absent by construction. The competitive masked-GM result is equally consistent with the improvement coming from the learned Bias plus SM fine-tuning rather than from VLM latent content. The radar normalization of Eq. (10) also hides the absolute magnitudes. Please add a quantitative control, e.g., replace h^{-l}_{:i} with random or scrambled features of matched dimension, retrain under the same protocol, and report the scalar metrics for with-GM, masked-GM, and random-feature conditions.
  3. [Section V.B, Table III, Fig. 5] All performance tables lack error bars and per-seed variation, and the headline numbers are selected post hoc from six configurations (Bias/TF x C0/C1/C-1) with PEl toggled. The differences between neighboring configurations are small, e.g., TF-C0 L2 avg 0.6380 vs TF-C1-PEl 0.6599 vs TF-C-1 0.6556, and the frame-wise analysis in Fig. 5 shows that TF mode degrades some frames. Please report at least three seeds with standard deviations and state which pairwise differences are significant; without this, the "significant enhancement" wording in the Abstract and Section V.B is not supported.
minor comments (6)
  1. [Section IV.B, Eq. (5)] The indexing of h^{-l}_{:i} is ambiguous; please define the stacking order, what l and i are for each collection mode (C0/C1/C-1), how PE^{-l}_{layer} and EN are concatenated, and clarify that ER in Eqs. (6)-(8) is the output of Eq. (5).
  2. [Section V.A.d] Please specify the at-deployment protocol: whether the question category is randomly sampled at inference as in training, the distribution of the randomly sampled prior-frame annotation used to simulate GM latency, and how this relates to the GM update cadence assumed in Table IV. Fig. 7 suggests perception QA hurts performance, so a random draw of the QA category can inject harmful instructions.
  3. [Fig. 6 captions] The figure captions contain typos and likely mislabels: "(c.2) Effect of Gf/Sf for Bias" should presumably read "for TF", "(a.2) ... with and without le" has a broken phrase, and "(b.3) ... Biast C1" should be "Bias C1". Please proofread and verify that each panel matches its caption.
  4. [Section V.A.c] The metric called "Similarity" is an L2 distance in metres with lower-is-better semantics; the phrase "12.46% improvement in L2 similarity" is confusing. Rename it to "trajectory L2 error" and report percentage decreases instead of ratios.
  5. [Section V.B, Table IV] The "98% reduction in information collection latency" claim should be qualified: the 0.0667 s collection/translation cost for C1 is per GM invocation, not per SM frame. State the assumed GM update cadence and how the amortized overhead is computed.
  6. [Section VI] The self-citation to PlanScope [48] in the concluding paragraph is not load-bearing for the paper's claims; please either justify its role or remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NetRoller's adapter is trained end-to-end and evaluated on external nuScenes benchmarks; self-citations are not load-bearing.

full rationale

NetRoller's central claim is empirical: a trained QFormer adapter translates frozen-GM latent features into VAD's BEV query/feature space and is evaluated on nuScenes planning/perception metrics. Eq. (5) defines ER as a learned function of collected hidden states; it is not defined in terms of the downstream collision/L2 quantities it is claimed to improve, so there is no self-definitional reduction. The few-epoch fine-tuning of the SM with Query/Feature Shift is a fitted procedure, but the reported improvements are measured against external nuScenes metrics rather than on the training residuals, so the fitted-input-called-prediction pattern does not apply. The w/o-GM ablation (Section V.D.c, Fig. 6(b.3)) is the paper's own limitation statement: 'Surprisingly, the Bias configuration maintains competitive performance across most metrics, even without GM integration.' That weakens the causal attribution of the gains to GM content, but it is an isolation/attribution concern, not a circularity concern; the prediction is still evaluated against external ground truth. Self-citations are present ([23], [29], [48]) but none is load-bearing: PlanScope [48] appears only as a closing reference to 'coarse-to-fine planning paradigm' and does not justify the core premise that first-token all-layer VLM features carry driving instructions. That premise is motivated by Anthropic's circuit tracing [31], an external source. The paper does not invoke a same-author uniqueness theorem or smuggle an ansatz in via citation. The headline percentage issue (16.71% is computed as 0.1900/0.1628 - 1 rather than (0.1900-0.1628)/0.1900 = 14.3%) is an arithmetic presentation problem, not a derivation that reduces to its own input. Overall, the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on trained adapter components (roller queries, nonsense embedding, layer positional embedding, shift transformers and biases) and several domain assumptions: VLM inter-layer features carry useful driving instructions, DriveLM QA training is a sufficient GM interface, random sampling of prior-frame annotations simulates asynchronous latency, and adding ER to queries or features is enough to transfer knowledge. No new physical entities are introduced.

free parameters (6)
  • Roller Embeddings (learnable queries in QFormer) = 1x256 for Bias mode, 8x1024 for TF mode
    Learned during joint fine-tuning on nuScenes training split; they are the cross-modality translation queries in Eq. (5) and Section IV.B.
  • Nonsense Embedding (EN) = R^{1 x dim_GM}, initialized randomly
    Learned padding vector concatenated to GM latents to stabilize attention and serve as a fallback when the GM produces no output; described in Section IV.B.
  • Layer Positional Embedding (PE_layer) = R^{l x dim_GM}, initialized uniformly
    Learned embedding marking which VLM layer a feature came from; used in Eq. (5) and Section IV.B.
  • Query/Feature Shift parameters (Transformer or bias) = Q + ER or Transformer(tgt = Q/F, mem = ER); dimensions differ for Bias versus TF
    Learned transformations that inject ER into the SM; equations (6)-(8) and Section IV.C.
  • Early stopping collection point and layer count l = Not specified
    Chosen by hand; Section V.B says 'the reasoning process can be stopped as early as we get the designated iterations of essential latent variables', but the value of l is never reported.
  • SM fine-tuning updates = VAD base parameters updated during 5-epoch joint fine-tuning
    The specialized model is fine-tuned together with the adapter, so observed gains mix adapter and SM updates; Section V.D.c tries to address this but not with a matched-parameter baseline.
assumptions (5)
  • domain assumption Inter-layer latent variables of an LLM encode semantically rich, instruction-like information that can guide generation (circuit tracing [31]).
    Used to justify collecting first-token all-layer features; Section I and IV.A. This is an empirical finding about LMs, not proven for the specific VAD task.
  • standard math VLM inference can be modeled as autoregressive token generation with argmax over P(tau_i | tau_{:i}, V; theta), with O(n^2) latency.
    Eq. (1) and Section III; standard autoregressive LM formulation.
  • domain assumption DriveLM's QA annotations over a subset of nuScenes are a sufficient vehicle for the VLM to produce useful driving instructions.
    Section V.A.a; the VLM is fine-tuned on DriveLM and used as GM. The paper itself notes QA categories vary and perception QA can hurt performance.
  • ad hoc to paper Randomly sampling a prior-frame annotation during training simulates the asynchronous latency, omissions, and irrelevant outputs of a real slow GM.
    Section V.A.d; this is a training-simulation assumption with no real-time closed-loop validation.
  • domain assumption Adding ER as a bias to Q or F (or via a one-layer transformer) is sufficient to inject GM knowledge without retraining the SM architecture.
    Eqs. (6)-(8); this is the core design hypothesis, evaluated only on offline nuScenes benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving." pith.science (2026). https://pith.science/paper/27DLF7JZ

@misc{pith2026250614589,
  author       = {Pith},
  title        = {Pith review of: NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27DLF7JZ}},
  note         = {Machine review of arXiv:2506.14589}
}
read the original abstract

Integrating General Models (GMs) such as Large Language Models (LLMs), with Specialized Models (SMs) in autonomous driving tasks presents a promising approach to mitigating challenges in data diversity and model capacity of existing specialized driving models. However, this integration leads to problems of asynchronous systems, which arise from the distinct characteristics inherent in GMs and SMs. To tackle this challenge, we propose NetRoller, an adapter that incorporates a set of novel mechanisms to facilitate the seamless integration of GMs and specialized driving models. Specifically, our mechanisms for interfacing the asynchronous GMs and SMs are organized into three key stages. NetRoller first harvests semantically rich and computationally efficient representations from the reasoning processes of LLMs using an early stopping mechanism, which preserves critical insights on driving context while maintaining low overhead. It then applies learnable query embeddings, nonsensical embeddings, and positional layer embeddings to facilitate robust and efficient cross-modality translation. At last, it employs computationally efficient Query Shift and Feature Shift mechanisms to enhance the performance of SMs through few-epoch fine-tuning. Based on the mechanisms formalized in these three stages, NetRoller enables specialized driving models to operate at their native frequencies while maintaining situational awareness of the GM. Experiments conducted on the nuScenes dataset demonstrate that integrating GM through NetRoller significantly improves human similarity and safety in planning tasks, and it also achieves noticeable precision improvements in detection and mapping tasks for end-to-end autonomous driving. The code and models are available at https://github.com/Rex-sys-hk/NetRoller .

Figures

Figures reproduced from arXiv: 2506.14589 by the authors.

Figure 1
Figure 1. This diagram illustrates the evolution of the asynchronous frameworks [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. It depicts the input and output modalities of the GM-SM asynchronous system, where the VLM acts as the GM and the E2E-AD model acts as SM. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of GM-SM asynchronous framework with NetRoller. The VLM receives panoptic images and prompts as inputs at time step ti and generates answer tokens. NetRoller collects the latent instructions and conducts either Query or Feature Shift, instructing the detection and planning modules of the specialized model at the following time steps. more time, resulting in a new query vector. This process extracts importan… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons of perception and planning performance are presented, which evaluate the vanilla VAD Base against its GM-enhanced variants using the NetRoller in Bias mode and TF mode. With GM integrated in NetRoller Bias mode, previously undetected vehicles on…
Figure 5
Figure 5. Figure 5: Frame-wise analysis: (a) illustrates the increase in similarity across [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Results of the ablation studies are presented. The first row compares the collection mechanisms based on Bias, with and without le and TF modes. The [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Comparison of different QA types for Bias and TF modes. In each figure, planning related QA serves as the baseline and the axis represents the advantage value of other QA type. Shift as two strategies for distributing the information for downstream applications. Throug…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [1]

    Planning-oriented autonomous driving,

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang et al. , “Planning-oriented autonomous driving,” in Proceedings of the Conference on Computer Vision and Pattern Recognition , 2023, pp. 17 853–17 862

    work page 2023

  2. [2]

    SparseDrive: End-to-end autonomous driving via sparse scene representation,

    W. Sun, X. Lin, Y . Shi, C. Zhang, H. Wu, and S. Zheng, “SparseDrive: End-to-end autonomous driving via sparse scene representation,” arXiv preprint arXiv:2405.19620, 2024

    arXiv 2024

  3. [3]

    V AD: Vectorized scene representation for efficient autonomous driving,

    B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang, “V AD: Vectorized scene representation for efficient autonomous driving,” Proceedings of the International Conference on Computer Vision , 2023

    work page 2023

  4. [4]

    MultiRuler: A multi-dimensional resource modeling method for embedded intelligent systems of autonomous driving,

    Y . Xu, B. Li, Z. Zhu, W. Liu, G. Jia, G. Han, and X. Li, “MultiRuler: A multi-dimensional resource modeling method for embedded intelligent systems of autonomous driving,” IEEE Transactions on V ehicular Technology, vol. 73, no. 5, pp. 6212–6224, 2024

    work page 2024

  5. [5]

    Trajectory planning for autonomous driving in unstructured scenarios based on deep learning and quadratic optimization,

    H. Li, P. Chen, G. Yu, B. Zhou, Y . Li, and Y . Liao, “Trajectory planning for autonomous driving in unstructured scenarios based on deep learning and quadratic optimization,” IEEE Transactions on V ehicular Technology, vol. 73, no. 4, pp. 4886–4903, 2024

    work page 2024

  6. [6]

    Vision-based autonomous driving: A hierarchical reinforcement learning approach,

    J. Wang, H. Sun, and C. Zhu, “Vision-based autonomous driving: A hierarchical reinforcement learning approach,” IEEE Transactions on V ehicular Technology, vol. 72, no. 9, pp. 11 213–11 226, 2023

    work page 2023

  7. [7]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan et al. , “Language models are few-shot learners,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901

    work page 2020

  8. [8]

    A simple method for commonsense reasoning,

    T. H. Trinh and Q. V . Le, “A simple method for commonsense reasoning,” arXiv preprint arXiv:1806.02847 , 2018

    arXiv 2018

Show all 48 references
  1. [9]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  2. [10]

    LLaMA 3 model card,

    AI@Meta, “LLaMA 3 model card,” 2024

    2024

  3. [11]

    LLaMA 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “LLaMA 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  4. [12]

    A generalized ChatGPT-based collaborative multi-objective decision-making framework for robust vehicle platoon collision avoidance,

    H. Zhang, C. Hou, J. Chen, H. Zhang, and F.-Y . Wang, “A generalized ChatGPT-based collaborative multi-objective decision-making framework for robust vehicle platoon collision avoidance,” IEEE Transactions on V ehicular Technology, vol. 74, no. 5, pp. 7212–7225, 2025

    2025

  5. [13]

    LA VIS: A one-stop library for language-vision intelligence,

    D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “LA VIS: A one-stop library for language-vision intelligence,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics . Toronto, Canada: Association for Computational Linguistics, Jul. 202...

    2023

  6. [14]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in Proceedings of the International Conference on Machine Learning, 2023

    2023

  7. [15]

    Judging LLM-as-a-Judge with MT-Bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging LLM-as-a-Judge with MT-Bench and chatbot arena,” Advances in Neural Information Processing Systems , vol. 36, pp. 46 595–46 623, 2023

    2023

  8. [16]

    Senna: Bridging large vision-language models and end-to-end autonomous driving,

    B. Jiang, S. Chen, B. Liao, X. Zhang, W. Yin, Q. Zhang, C. Huang, W. Liu, and X. Wang, “Senna: Bridging large vision-language models and end-to-end autonomous driving,” arXiv preprint arXiv:2410.22313 , 2024

    2024 arXiv

  9. [17]

    EMMA: End-to-end multimodal model for autonomous driving,

    J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp et al. , “EMMA: End-to-end multimodal model for autonomous driving,” arXiv preprint arXiv:2410.23262 , 2024

    2024 arXiv

  10. [18]

    OmniDrive: A holistic LLM-agent framework for autonomous driving with 3D perception, reasoning and planning,

    S. Wang, Z. Yu, X. Jiang, S. Lan, M. Shi, N. Chang, J. Kautz, Y . Li, and J. M. Alvarez, “OmniDrive: A holistic LLM-agent framework for autonomous driving with 3D perception, reasoning and planning,” arXiv preprint arXiv:2405.01533, 2024

    2024 arXiv

  11. [19]

    AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning,

    B. Jiang, S. Chen, Q. Zhang, W. Liu, and X. Wang, “AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning,” https://arxiv.org/abs/2503.07608, 2025

    2025 arXiv

  12. [20]

    DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving,

    W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y . Wen, S. Wu, H. Deng, Z. Li et al., “DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving,” arXiv preprint arXiv:2312.09245, 2023. 13

    2023

  13. [21]

    DriveVLM: The convergence of autonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “DriveVLM: The convergence of autonomous driving and large vision-language models,” arXiv preprint arXiv:2402.12289 , 2024

    2024 arXiv

  14. [22]

    DriveGPT4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable end-to-end autonomous driving via large language model,” IEEE Robotics and Automation Letters , 2024

    2024

  15. [23]

    VLM-E2E: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,

    P. Liu, H. Liu, H. Liu, X. Liu, J. Ni, and J. Ma, “VLM-E2E: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,” arXiv preprint arXiv:2502.18042 , 2025

    2025

  16. [24]

    VLM-AD: End-to-end autonomous driving through vision-language model supervision,

    Y . Xu, Y . Hu, Z. Zhang, G. P. Meyer, S. K. Mustikovela, S. Srinivasa, E. M. Wolff, and X. Huang, “VLM-AD: End-to-end autonomous driving through vision-language model supervision,” arXiv preprint arXiv:2412.14446, 2024

    2024 arXiv

  17. [25]

    ORION: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,

    H. Fu, D. Zhang, Z. Zhao, J. Cui, D. Liang, C. Zhang, D. Zhang, H. Xie, B. Wang, and X. Bai, “ORION: A holistic end-to-end autonomous driving framework by vision-language instructed action generation,” arXiv:2503.19755, 2025

    2025 arXiv

  18. [26]

    LMDrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “LMDrive: Closed-loop end-to-end driving with large language models,” in Proceedings of the Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 120–15 130

    2024

  19. [27]

    Asynchronous large language model enhanced planner for autonomous driving,

    Y . Chen, Z.-h. Ding, Z. Wang, Y . Wang, L. Zhang, and S. Liu, “Asynchronous large language model enhanced planner for autonomous driving,” in Proceedings of the European Conference on Computer Vision . Springer, 2024, pp. 22–38

    2024

  20. [28]

    OpenDriveVLA: Towards end-to-end autonomous driving with large vision language action model,

    X. Zhou, X. Han, F. Yang, Y . Ma, and A. C. Knoll, “OpenDriveVLA: Towards end-to-end autonomous driving with large vision language action model,” 2025

    2025

  21. [29]

    DSDrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning,

    W. Liu, P. Liu, and J. Ma, “DSDrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning,” arXiv preprint arXiv:2505.05360 , 2025

    2025 arXiv

  22. [30]

    Sora: A review on background, technology, limitations, and opportunities of large vision models,

    Y . Liu, K. Zhang, Y . Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y . Huang, H. Sun, J. Gao et al. , “Sora: A review on background, technology, limitations, and opportunities of large vision models,” arXiv preprint arXiv:2402.17177, 2024

    2024 arXiv

  23. [31]

    Circuit tracing: Revealing computational graphs in language models,

    E. Ameisen, J. Lindsey, A. Pearce, W. Gurnee, N. L. Turner, B. Chen, C. Citro, D. Abrahams, S. Carter, B. Hosmer et al. , “Circuit tracing: Revealing computational graphs in language models,” Transformer Circuits Thread, 2025

    2025

  24. [32]

    nuScenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” in Proceedings of the Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 621–11 631

    2020

  25. [33]

    MP3: A unified model to map, perceive, predict and plan,

    S. Casas, A. Sadat, and R. Urtasun, “MP3: A unified model to map, perceive, predict and plan,” in Proceedings of the Conference on Computer Vision and Pattern Recognition , 2021, pp. 14 398–14 407

    2021

  26. [34]

    V ADv2: End-to-end vectorized autonomous driving via probabilistic planning,

    S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang, “V ADv2: End-to-end vectorized autonomous driving via probabilistic planning,” arXiv preprint arXiv:2402.13243 , 2024

    2024 arXiv

  27. [35]

    PLUTO: Pushing the limit of imitation learning-based planning for autonomous driving,

    J. Cheng, Y . Chen, and Q. Chen, “PLUTO: Pushing the limit of imitation learning-based planning for autonomous driving,” arXiv preprint arXiv:2404.14327, 2024

    2024 arXiv

  28. [36]

    DiffusionDrive: Truncated diffusion model for end-to-end autonomous driving,

    B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y . Zhang, Q. Zhang et al. , “DiffusionDrive: Truncated diffusion model for end-to-end autonomous driving,” arXiv preprint arXiv:2411.15139 , 2024

    2024 arXiv

  29. [37]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learning . PmLR, 2021, pp. 8748–8763

    2021

  30. [38]

    Applications of large language models and multimodal large models in autonomous driving: A comprehensive review,

    J. Li, J. Li, G. Yang, L. Yang, H. Chi, and L. Yang, “Applications of large language models and multimodal large models in autonomous driving: A comprehensive review,” Drones, vol. 9, no. 4, 2025

    2025

  31. [39]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics , 2019, pp. 4171–4186

    2019

  32. [40]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  33. [41]

    DriveLM: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with graph visual question answering,” arXiv preprint arXiv:2312.14150 , 2023

    2023 arXiv

  34. [42]

    Vision transformer with deformable attention,

    Z. Xia, X. Pan, S. Song, L. E. Li, and G. Huang, “Vision transformer with deformable attention,” in Proceedings of the Conference on Computer Vision and Pattern Recognition , 2022, pp. 4794–4803

    2022

  35. [43]

    BEVFormer: Learning bird‘s-eye-view representation from multi-camera images via spatiotemporal transformers,

    Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y . Qiao, and J. Dai, “BEVFormer: Learning bird‘s-eye-view representation from multi-camera images via spatiotemporal transformers,” arXiv preprint arXiv:2203.17270, 2022

    2022 arXiv

  36. [44]

    SmolVLM: Redefining small and efficient multimodal models,

    A. Marafioti, O. Zohar, M. Farré, M. Noyan, E. Bakouch, P. Cuenca, C. Zakka, L. B. Allal, A. Lozhkov, N. Tazi et al., “SmolVLM: Redefining small and efficient multimodal models,” arXiv preprint arXiv:2504.05299, 2025

    2025 arXiv

  37. [45]

    End-to-end interpretable neural motion planner,

    W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun, “End-to-end interpretable neural motion planner,” in Proceedings of the Conference on Computer Vision and Pattern Recognition , 2019, pp. 8660–8669

    2019

  38. [46]

    ST-P3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,

    S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao, “ST-P3: End-to-end vision-based autonomous driving via spatial-temporal feature learning,” in Proceedings of the European Conference on Computer Vision , 2022

    2022

  39. [47]

    Gpt-Driver: Learning to drive with GPT,

    J. Mao, Y . Qian, J. Ye, H. Zhao, and Y . Wang, “Gpt-Driver: Learning to drive with GPT,” arXiv preprint arXiv:2310.01415 , 2023

    2023 arXiv

  40. [48]

    PlanScope: Learning to plan within decision scope does matter,

    R. Xin, J. Cheng, H. Liu, and J. Ma, “PlanScope: Learning to plan within decision scope does matter,” arXiv preprint arXiv:2411.00476 , 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.