Pith. sign in

REVIEW 3 major objections 6 minor 49 references

ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By forecasting a large model's features from past frames and updating them with a small current-frame model, ETA reaches a 69.53 Bench2Drive driving score at 50 ms latency.

desk verdict The async forecasting idea is real and the ablations are careful, but the 8% SOTA claim is not load-bearing until the authors evaluate on the official Bench2Drive split. read the letter →

arxiv 2506.07725 v1 pith:ME4P5JKT submitted 2025-06-09 cs.CV cs.AI

Shadi Hamdan , Chonghao Sima , Zetong Yang , Hongyang Li , Fatma Güney This is my paper
classification cs.CVcs.AI
keywords autonomousdrivingdual-systemarchitecturelargevisionmodelsasynchronousinferencefeatureforecastingactionmaskBench2Driveclosed-loopevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on the speed-accuracy dilemma in self-driving: the most informative vision models are too slow for every control frame, while fast models miss the reasoning those large models provide. Its proposed solution, ETA, is an asynchronous dual system in which a large model processes an earlier frame, a lightweight forecasting module predicts what the large model's features would be at the current frame, and a small model supplies fresh information about the present. On the closed-loop Bench2Drive benchmark this reaches a driving score of 69.53 at 50 ms latency, an 8.2 percent improvement over the previous best, while the large-model-only base reaches 74.33 at 102 ms. If the comparison holds, the paper establishes that latency and large-model quality are not a strict trade-off, and its pattern is portable to other embodied systems that need slow, thoughtful perception on a fast control loop.

What carries the argument

The load-bearing mechanism is the pairing of a predictive large model with a small-model update on a two-frame asynchronous loop. At time $t-\Delta$ the large encoder extracts features; a two-layer forecasting transformer predicts the features at time $t$, conditioned on the predicted action, speed, and target waypoints; the small encoder processes the current image so that unpredictable events such as a red light change or a sudden brake enter the representation; and an action decoder fuses both feature streams. Supervising the forecast with the large model's true features, with gradients stopped, and supervising an action mask with projected expert trajectories is what makes the fused representation behave like a current-frame large-model representation. Batching the large-model inference across multiple frames is what converts the saved computation into a 50 ms control period.

What would settle it

Re-run the async model on the original Bench2Drive evaluation protocol's test routes, the same routes used for the published baseline scores, and compare driving scores; if the score no longer exceeds the previous best of 64.22, the claimed state-of-the-art improvement is an artifact of the custom split.

Watch

Extended reading notes

Core claim

The central claim is that asynchronous execution, with the slow model reading the past and the fast model reading the present, lets a driving policy use large-model features at every time step rather than only when the slow model happens to finish. The paper implements this as ETA: the large encoder processes frame $I_{t-\Delta}$, a forecasting transformer predicts the current-frame features $\hat{f}_t^l$ from those past features together with the past action and conditioning, and a small encoder (the first eight layers of the same vision transformer) processes the current frame $I_t$; an action decoder consumes both feature streams. The forecast is supervised during training by an $L_1$ loss against the large model's own features extracted at the current frame, and an action-mask loss aligns attention with the image patches the expert trajectory touches. On the Bench2Drive benchmark the resulting asynchronous model scores 69.53 at 50 ms, behind only the paper's own 102 ms base model (74.33) and ahead of all previously published results in the comparison table.

Load-bearing premise

The headline comparison assumes that the paper's custom 94%-3%-3% split of the Bench2Drive base set uses the same test routes as the evaluation protocol behind the published baseline scores; if the routes differ, the 8.2 percent improvement could reflect route difficulty rather than method quality.

Editorial extensions

If this is right

  • If the comparison is valid, ETA raises the best published Bench2Drive driving score by roughly 8 percent while cutting latency to 50 ms, so a 20 Hz control loop can carry large-model information.
  • The dual model closes most of the gap to the 102 ms base model, meaning the forecast-plus-small-model update captures nearly all of the benefit of running the large model on the current frame.
  • Ablations show the gain is not just parameter count: without forecasting supervision the score drops from 69.53 to 54.92, and without the small model it drops to 42.49, so both components are necessary.
  • With ground-truth features in place of forecasted ones the async model reaches 74.12, close to the base model's 74.33, indicating that better forecasting, not architectural redesign, is the path to parity.
  • The same batched predictive-inference pattern can be applied to any slow perception module whose output is needed at every control tick in an embodied system.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the action mask carries more weight in the async model than its name suggests, since removing it costs about 27 driving-score points, comparable to removing the small model; it may be a learned spatial-gating signal rather than a simple auxiliary loss.
  • Editorial inference: the custom 94%-3%-3% split leaves the headline 8.2 percent gain vulnerable to route-selection effects, so re-running on the original benchmark protocol's test routes would settle whether the improvement transfers.
  • Editorial inference: because the small model is the first eight layers of the same encoder, the design suggests a parameter-sharing recipe: one shared backbone split across time, with the slow deep path forecasting and the fast shallow path correcting.
  • Editorial inference: a natural testable extension is applying ETA to other latency-critical policies, such as robot manipulation or online mapping, where the slow model's features could be forecasted at the same 20 Hz cadence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes ETA, an asynchronous dual-system architecture for closed-loop autonomous driving in CARLA Bench2Drive. The base model uses a CLIP-ViT-L large encoder with a transformer decoder to predict driving actions. To reduce latency, the async model shifts the large model's computation to a previous frame, forecasts its features to the current frame, and combines them with features from a small 8-layer encoder, guided by an action-mask loss. On Bench2Drive, the async model reports a driving score of 69.53 at 50 ms latency, which the authors claim is an 8.2% improvement over the previous best, while the base model reaches 74.33 at 102 ms. Ablations show that removing forecasting, the small model, or the action mask degrades performance, and using ground-truth forecast features recovers base-model-level performance.

Significance. The core idea is timely and relevant: making large-model features available at every control frame through batched forecasting plus a small-model 'glance' is a plausible alternative to memory-bank retrieval or low-frequency large-model updates. The paper includes several good practices: three-run averaging with standard deviations for the main models, a ground-truth-forecasting upper-bound experiment that supports the design rather than an unfalsifiable claim, and per-ability diagnostics. The internal ablations are consistent with the proposed mechanism. However, the headline SOTA claim is only as strong as the evaluation protocol: the custom 94/3/3 split and unverified comparability to official Bench2Drive numbers, plus underspecified latency measurements, mean the quantitative claims are not yet established. With these addressed, the contribution would be significant to the autonomous-driving and efficient-foundation-model communities.

major comments (3)
  1. [Section 4.2, Table 1] The central claim that ETA advances SOTA by 8.2% (69.53 vs. 64.22) is not yet supported by the evaluation protocol. The authors evaluate on a custom 94%/3%/3% split of the 1000-route Bench2Drive base set, yielding approximately 30 test routes, while the prior scores in Table 1 were obtained under the benchmark's original protocol. The paper does not report the test route IDs, the distribution of abilities and weather in the test split, or any re-evaluation of the prior baselines on the same 30 routes. The 8.2% gap could be partly or wholly attributable to route-difficulty differences. To substantiate the SOTA claim, the authors should evaluate on the official Bench2Drive evaluation set or, failing that, re-run the competing methods on their custom split and report the same three-run statistics.
  2. [Section 4.2, Table 1] The efficiency claim ('near-real-time at 50 ms', '20 FPS') is reported without specifying the hardware, inference framework, batch size, or whether this is end-to-end latency or network-only latency on a single process. Prior methods' latencies in Table 1 are taken from their respective papers and may be measured on different GPUs and with different batching. A 50 ms number on an A100 is not directly comparable to 211.7 ms measured elsewhere. The authors should report the exact evaluation setup and, ideally, measure all baselines on the same hardware.
  3. [Section 4.2, Tables 3 and 4] With only about 30 routes in the test split, the per-ability scores (e.g., Merging, E. Brake, T. Sign) are computed over very few scenarios, making the per-ability conclusions in Section 4.4 fragile. The paper should report the number of routes per ability and provide confidence intervals or error bars for these scores, especially since the async model's advantage over prior methods is claimed to be concentrated in these ability categories.
minor comments (6)
  1. [Section 3.2, Eq. (2)] Equation (2) writes the action prediction as f_action(I_t, c_t), but the text states that the action model takes the large-model features f^l_t; this should be corrected to f_action(f^l_t, c_t) for consistency with the async model's Eq. (6).
  2. [Supplementary Table 6] Row C of Table 6 reports an SR of 87.26 for the base model without the action mask, which is inconsistent with the base model's SR of 48.33 in the same table and appears to be a typo (possibly 47.26); please correct it.
  3. [Section 4.2] The paper states that the data are divided into 94%/3%/3% splits but does not describe how the validation set is used or whether the reported results are on the test set; please clarify the checkpoint-selection protocol.
  4. [Figure 1] Figure 1 would benefit from error bars or confidence intervals on the proposed models, since the paper reports three-run standard deviations in Tables 1 and 2.
  5. [Supplementary Section 7 (Data Bucketing)] Several buckets (e.g., 'Light acceleration' and 'Medium acceleration') are described with the same phrase 'The agent is slightly accelerating'; the wording should distinguish the ranges, and the oversampling weights should be listed consistently in one place.
  6. [Section 8.3, Fig. 8 caption] The caption states that 'both models get a perfect driving score in this scenario,' which is unclear because Bench2Drive scores are route-level composites; please clarify whether the scenario is scored as 100 or some other value.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reasoning: the forecasting target is the large model's own features, a standard distillation objective, and the claimed driving score is an empirical measurement rather than an identity.

full rationale

The paper's derivation chain is self-contained and non-circular. The asynchronous model predicts current-frame features from previous-frame large-model features via Eq. (4): ˆf^l_t = f_forecast(f^l_{t−∆}, ˆa_{t−∆}, c_{t−∆}), and supervises this with L_forecast(f^l_t, ˆf^l_t) = |f^l_t − ˆf^l_t| in Eq. (8). The supervision target is the large model's own feature representation at the current time step, not the downstream driving score; predicting features from previous features is a standard distillation/forecasting setup and does not presuppose the final DS. The final action is produced independently from forecasted and small-model features via Eq. (6), and the driving score is obtained by closed-loop evaluation on Bench2Drive. Ablations (A–F) explicitly vary components and even show that using ground-truth forecast features at test time degrades performance (F: 60.72 vs. 69.53), which would be impossible if the result were forced by construction. There are self-citations to prior work by the authors (e.g., UniAD, DriveAdapter, DriveLM), but none is used as a load-bearing premise or as a 'uniqueness theorem' to exclude alternatives; they appear as baselines or related work. The one substantive concern is the custom 94%-3%-3% split in Section 4.2, which may make the SOTA comparison non-comparable to prior published numbers; however, that is a benchmark-protocol/validity issue, not circular reasoning. No equation or definition reduces the claimed improvement to its own inputs, so no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical entities or formal derivations. It relies on standard deep learning assumptions about benchmark validity, expert data quality, feature predictability, and transfer from pre-trained vision encoders. The free parameters are engineering choices that affect the final driving score but are not the focus of a formal derivation.

free parameters (5)
  • Delta (time offset) = 0.5 seconds
    Chosen by hand as the temporal gap between the large model's input frame and the current frame. It trades forecasting difficulty against available compute time.
  • Action mask loss weight (lambda_1) = 1/16
    Set to balance the action mask loss against the action loss in Eq. 7. This value is tuned and affects the final driving score.
  • Forecasting loss weight = 0.5
    Weight for the feature forecasting loss in the async model, chosen to balance imitation of large-model features against action quality.
  • Small model depth = 8 layers of CLIP-ViT-L-336px
    The first 8 layers of the large encoder are used as the small model to meet the 50 ms latency target; depth directly controls speed and accuracy.
  • Data bucket oversampling weights = Steering left/right: 3, acceleration buckets: 2, others: 1
    Hand-set sampling weights for data buckets to increase frequency of interesting scenarios during training; these affect the learned policy.
assumptions (4)
  • domain assumption Bench2Drive closed-loop scores are a meaningful proxy for real-world driving performance
    The central evaluation and all claims rest on the validity of the CARLA-based Bench2Drive benchmark as a measure of driving ability. This is not questioned in the paper.
  • domain assumption Think2Drive RL expert provides high-quality demonstrations
    Training data is collected by the Think2Drive agent; the paper assumes these demonstrations are reliable enough to learn good driving behavior.
  • domain assumption Features from a large model at t-delta can be forecasted to approximate features at t
    The core mechanism of the async model assumes that a lightweight transformer can predict large-model features 0.5 s ahead given conditioning inputs. The paper tests this empirically but does not derive it.
  • domain assumption CLIP-ViT pre-training transfers to driving action prediction
    The large encoder is CLIP-ViT-L-336px from LLaVA 1.6; the paper assumes its visual features are useful for predicting driving paths and waypoints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models." pith.science (2026). https://pith.science/paper/ME4P5JKT

@misc{pith2026250607725,
  author       = {Pith},
  title        = {Pith review of: ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ME4P5JKT}},
  note         = {Machine review of arXiv:2506.07725}
}
read the original abstract

How can we benefit from large models without sacrificing inference speed, a common dilemma in self-driving systems? A prevalent solution is a dual-system architecture, employing a small model for rapid, reactive decisions and a larger model for slower but more informative analyses. Existing dual-system designs often implement parallel architectures where inference is either directly conducted using the large model at each current frame or retrieved from previously stored inference results. However, these works still struggle to enable large models for a timely response to every online frame. Our key insight is to shift intensive computations of the current frame to previous time steps and perform a batch inference of multiple time steps to make large models respond promptly to each time step. To achieve the shifting, we introduce Efficiency through Thinking Ahead (ETA), an asynchronous system designed to: (1) propagate informative features from the past to the current frame using future predictions from the large model, (2) extract current frame features using a small model for real-time responsiveness, and (3) integrate these dual features via an action mask mechanism that emphasizes action-critical image regions. Evaluated on the Bench2Drive CARLA Leaderboard-v2 benchmark, ETA advances state-of-the-art performance by 8% with a driving score of 69.53 while maintaining a near-real-time inference speed at 50 ms.

Figures

Figures reproduced from arXiv: 2506.07725 by the authors.

Figure 1
Figure 1. Comparison on Bench2Drive [13]. Our Base model achieves the best performance across all metrics with a high la￾tency. By improving latency to 20 FPS, our Async model achieves the second best in all metrics except Traffic Sign handling. bilities [4, 9, 24, 38]. Due to the real-time demand of self￾driving, inference-time speed becomes a critical concern when using large models. For instance, the best-performing models… view at source ↗
Figure 2
Figure 2. ). One potential solution for integrating large models into every frame is to store the outputs of the large model for complex cases in memory and retrieve relevant instances as needed using the small model, as explored in prior re￾search [25]. However, managing an expanding memory bank in dynamic environments poses a challenge and can be difficult to generalize. In this paper, we manage to make the inference of lar… view at source ↗
Figure 3
Figure 3. Overview of the Asynchronous (Async) Model. Our model processes two frames, ∆ apart in time, using the large model flarge for the previous frame It−∆ and the small model fsmall for the current frame It. Based on the previous frame’s features from the large model, f l t−∆, along with conditioning inputs ct and aˆt, we predict the current time-step features, ˆf l t . The action aˆt is then predicted using the action m… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Action Mask. In the top row, we show two examples of the RGB image with the path (purple) and waypoints (blue) pro￾jected onto it. Patches containing a path or waypoint are marked as 1 (yellow), and all other patches are marked as 0 (purple), creating the binary mask, …
Figure 5
Figure 5. Figure 5: Hard Brake without Small Model. Left: The initial state of the scene where the yellow car ahead suddenly brakes, requiring the ego vehicle to perform a hard brake. Middle: The Async Model detects the hazard using the small model and stops in time. Right: The version wi…
Figure 6
Figure 6. Figure 6: Lane Change without Forecasting. Left: The initial state of the scene, where the ego vehicle is tasked with switching to the left lane. Middle: The Async Model successfully executes the lane change while avoiding collisions by leveraging forecasting. Right: The version…
Figure 7
Figure 7. Figure 7: Turn Failure. In the left column, we show the initial state of the scene. In this scenario, the agent is performing a right turn at an intersection. In the middle, the Async Model can maneuver properly, performing a correct turn. On the right, the model without the sma…
Figure 8
Figure 8. Figure 8: Lane Change - Base model vs. Async. In the left column, we show the initial state of the ego vehicle in traffic. In this scenario, the ego vehicle is required to switch the left lane. In the middle, the Base model is able to correctly make the lange change while avoidi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 35 canonical work pages

  1. [1]

    Validity Learning on Failures: Mitigating the Distribution Shift in Autonomous Vehicle Planning

    Fazel Arasteh, Mohammed Elmahgiubi, Behzad Khamidehi, Hamidreza Mirkhani, Weize Zhang, Cao Tongtong, and Kasra Rezaee. Validity learning on failures: Mitigating the distribution shift in autonomous vehicle planning.arXiv preprint arXiv:2406.01544, 2024. 3

    work page Pith review arXiv 2024

  2. [2]

    PDM-Lite: A rule-based planner for carla leaderboard 2.0

    Jens Beißwenger. PDM-Lite: A rule-based planner for carla leaderboard 2.0. 2024. 3

    work page 2024

  3. [3]

    Towards synergis- tic, generalized, and efficient dual-system for robotic manip- ulation.arXiv preprint arXiv:2410.08001, 2024

    Qingwen Bu, Hongyang Li, Li Chen, Jisong Cai, Jia Zeng, Heming Cui, Maoqing Yao, and Yu Qiao. Towards synergis- tic, generalized, and efficient dual-system for robotic manip- ulation.arXiv preprint arXiv:2410.08001, 2024. 1, 3

    arXiv 2024

  4. [4]

    End-to-end autonomous driving: Challenges and frontiers.IEEE TPAMI, 2024

    Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, An- dreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers.IEEE TPAMI, 2024. 1

    work page 2024

  5. [5]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InCVPR, 2024. 1

    work page 2024

  6. [6]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InICLR, 2021. 1

    work page 2021

  7. [7]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duck- worth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An embodie...

    work page 2023

  8. [8]

    Helix: A vision-language-action model for generalist humanoid control

    Figure. Helix: A vision-language-action model for generalist humanoid control. 2025. 1, 3

    work page 2025

Show all 49 references
  1. [9]

    Planning-oriented autonomous driv- ing

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driv- ing. InCVPR, 2023. 1, 6, 12

    2023

  2. [10]

    Hid- den biases of end-to-end driving models

    Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hid- den biases of end-to-end driving models. InICCV, 2023. 3

    2023

  3. [11]

    DriveAdapter: breaking the coupling barrier of perception and planning in end-to-end autonomous driving

    Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. DriveAdapter: breaking the coupling barrier of perception and planning in end-to-end autonomous driving. InICCV, 2023. 1, 6, 12

    2023

  4. [12]

    Think twice before driv- ing: Towards scalable decoders for end-to-end autonomous driving

    Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driv- ing: Towards scalable decoders for end-to-end autonomous driving. InCVPR, 2023. 6, 12

    2023

  5. [13]

    Bench2Drive: towards multi-ability bench- marking of closed-loop end-to-end autonomous driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2Drive: towards multi-ability bench- marking of closed-loop end-to-end autonomous driving. In NeurIPS 2024 Datasets and Benchmarks Track, 2024. 1, 3, 5

    2024

  6. [14]

    DriveTransformer: unified transformer for scalable end-to- end autonomous driving

    Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. DriveTransformer: unified transformer for scalable end-to- end autonomous driving. InICLR, 2025. 1, 3, 6, 12

    2025

  7. [15]

    V AD: vectorized scene representation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. V AD: vectorized scene representation for efficient autonomous driving. InICCV, 2023. 6, 12

    2023

  8. [16]

    Farrar, Straus and Giroux, 2011

    Daniel Kahneman.Thinking, fast and slow. Farrar, Straus and Giroux, 2011. 1

    2011

  9. [17]

    Integrat- ing end-to-end and modular driving approaches for online corner case detection in autonomous driving.arXiv preprint arXiv:2409.01178, 2024

    Gemb Kaljavesi, Xiyan Su, and Frank Diermeyer. Integrat- ing end-to-end and modular driving approaches for online corner case detection in autonomous driving.arXiv preprint arXiv:2409.01178, 2024. 3

    2024 arXiv

  10. [18]

    An energy and gpu-computation efficient backbone network for real-time object detection

    Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpu-computation efficient backbone network for real-time object detection. In CVPR Workshop, 2019. 1

    2019

  11. [19]

    Think2Drive: efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2)

    Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2Drive: efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2). InECCV, 2024. 3, 5

    2024

  12. [20]

    CogACT: a foundational vision-language- action model for synergizing cognition and action in robotic manipulation.arXiv preprint arXiv:2411.19650, 2024

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jian- min Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. CogACT: a foundational vision-language- action model f...

    2024 arXiv

  13. [21]

    Enhancing end-to-end autonomous driving with latent world model

    Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. InICLR, 2025. 5

    2025

  14. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 4 9

    2023

  15. [23]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR,

  16. [24]

    Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023. 1

    2023 arXiv

  17. [25]

    Continuously learning, adapting, and improving: A dual- process approach to autonomous driving

    Jianbiao Mei, Yukai Ma, Xuemeng Yang, Licheng Wen, Xinyu Cai, Xin Li, Daocheng Fu, Bo Zhang, Pinlong Cai, Min Dou, Botian Shi, Liang He, Yong Liu, and Yu Qiao. Continuously learning, adapting, and improving: A dual- process approach to autonomous driving. InNeurIPS, 2024. 1, 2, 3, 6

    2024

  18. [26]

    OpenAI: Introducing ChatGPT

    OpenAI. OpenAI: Introducing ChatGPT. 2022. 1

    2022

  19. [27]

    VLP: vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. VLP: vision language planning for autonomous driving. InCVPR, 2024. 11

    2024

  20. [28]

    CarLLaV A: Vi- sion language models for camera-only closed-loop driving

    Katrin Renz, Long Chen, Ana-Maria Marcu, Jan H¨unermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, and Oleg Sinavski. CarLLaV A: Vi- sion language models for camera-only closed-loop driving. arXiv preprint arXiv:2406.10165, 2024. 3, 4, 6, 11

    2024 arXiv

  21. [29]

    Waslander, Yu Liu, and Hongsheng Li

    Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, and Hongsheng Li. LMDrive: Closed-loop end-to- end driving with large language models. InCVPR, 2024. 3

    2024

  22. [30]

    From LLMs to Actions: Latent codes as bridges in hierarchical robot control.arXiv preprint arXiv:2405.04798,

    Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From LLMs to Actions: Latent codes as bridges in hierarchical robot control.arXiv preprint arXiv:2405.04798,

    arXiv

  23. [31]

    Hi Robot: Open-ended instruction following with hierarchical vision-language-action models.arXiv preprint arXiv:2502.19417, 2025

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyim- ing Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi Robot: Open-ended instruction following with hierarch...

    2025 arXiv

  24. [32]

    DriveLM: Driving with graph visual ques- tion answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with graph visual ques- tion answering. InECCV, 2024. 3, 11

    2024

  25. [33]

    Centaur: Robust end-to-end autonomous driving with test-time training.arXiv preprint arXiv:2503.11650, 2025

    Chonghao Sima, Kashyap Chitta, Zhiding Yu, Shiyi Lan, Ping Luo, Andreas Geiger, Hongyang Li, and Jose M Al- varez. Centaur: Robust end-to-end autonomous driving with test-time training.arXiv preprint arXiv:2503.11650, 2025. 1

    2025 arXiv

  26. [34]

    Probing multi- modal LLMs as world models for driving.arXiv preprint arXiv:2405.05956, 2024

    Shiva Sreeram, Tsun-Hsuan Wang, Alaa Maalouf, Guy Ros- man, Sertac Karaman, and Daniela Rus. Probing multi- modal LLMs as world models for driving.arXiv preprint arXiv:2405.05956, 2024. 11

    2024 arXiv

  27. [35]

    DriveVLM: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Zhiyong Zhao, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The convergence of autonomous driving and large vision-language models. InCoRL, 2024. 1, 2, 6, 11

    2024

  28. [36]

    DriveMLM: Aligning multi- modal large language models with behavioral planning states for autonomous driving.arXiv preprint arXiv:2312.09245,

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, Hao Tian, Lewei Lu, Xizhou Zhu, Xiaogang Wang, Yu Qiao, and Jifeng Dai. DriveMLM: Aligning multi- modal large language models with behavioral planning sta...

  29. [37]

    Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong base- line. InNeurIPS, 2022. 6, 12

    2022

  30. [38]

    DriveGPT4: Interpretable end-to-end autonomous driving via large language model.RA-L, 2024

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable end-to-end autonomous driving via large language model.RA-L, 2024. 1

    2024

  31. [39]

    Hard cases detection in motion prediction by vision-language foundation models

    Yi Yang, Qingwen Zhang, Kei Ikemura, Nazre Batool, and John Folkesson. Hard cases detection in motion prediction by vision-language foundation models. InIV, 2024. 11

    2024

  32. [40]

    Distilling system 2 into system 1.arXiv preprint arXiv:2407.06023,

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1.arXiv preprint arXiv:2407.06023,

    arXiv

  33. [41]

    Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023

    Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes.arXiv preprint arXiv:2305.10430, 2023. 2, 6, 12

    2023 arXiv

  34. [42]

    HiRT: En- hancing robotic control with hierarchical robot transformers

    Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. HiRT: En- hancing robotic control with hierarchical robot transformers. InCoRL, 2024. 1, 3

    2024

  35. [43]

    Feedback-guided autonomous driving

    Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedback-guided autonomous driving. InCVPR,

  36. [44]

    AD-H: autonomous driving with hierarchical agents.arXiv preprint arXiv:2406.03474, 2024

    Zaibin Zhang, Shiyu Tang, Yuanhang Zhang, Talas Fu, Yifan Wang, Yang Liu, Dong Wang, Jing Shao, Lijun Wang, and Huchuan Lu. AD-H: autonomous driving with hierarchical agents.arXiv preprint arXiv:2406.03474, 2024. 1, 2, 3, 6

    2024 arXiv

  37. [45]

    Hidden biases of end- to-end driving datasets.arXiv preprint arXiv:2412.09602,

    Julian Zimmerlin, Jens Beißwenger, Bernhard Jaeger, An- dreas Geiger, and Kashyap Chitta. Hidden biases of end- to-end driving datasets.arXiv preprint arXiv:2412.09602,

    arXiv

  38. [46]

    Sanketi, Grecia Salazar, Michael S

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryo...

    2023

  39. [47]

    A line of work, represented by approaches like DriveMLM [36], LLM4AD/CarLLaV A [28], and FeD [43], uses LLMs for initialization to benefit from their pre-training on large datasets

    Additional Related Work LLMs/VLMs for Driving:There is a large increase in the use of LLMs/VLMs for driving. A line of work, represented by approaches like DriveMLM [36], LLM4AD/CarLLaV A [28], and FeD [43], uses LLMs for initialization to benefit from their pre-training on la...

  40. [48]

    Small Model:For the Small Model, we use the first 8 layers of the same CLIP-ViT-L-336px encoder as the Base Large Model

    Experimental Details Base Large Model:For the Large Model, we use the CLIP-ViT-L-336px encoder from the checkpoint of LLaV A 1.6 [23] 7B Vicuna, which has 24 layers with a hidden di- mension size of 1024 and an input patch size of 14. Small Model:For the Small Model, we use th...

  41. [49]

    Ability Scores We include the per-ability scores for all models in Table 4

    Additional Experiments 8.1. Ability Scores We include the per-ability scores for all models in Table 4. 8.2. Additional Ablations We include two additional ablations in Table 6 in addition to the ablations inincluded in the main paper. Action Mask Loss:We ablate the inclusion ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.