Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Planning driving trajectories as pixels on the image, then lifting them to 3D only at inference, lets vision-language models scale across cameras and ground plans in vision rather than ego-status.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 16:00 UTC pith:KCGOQX4U

load-bearing objection Solid systems fix for sensor-entangled driving VLAs: learn 2D image-plane plans, lift with fixed geometry at inference; ego-status ablation is the real evidence, local-plane lift is the known hinge. the 3 major comments →

arxiv 2607.04637 v1 pith:KCGOQX4U submitted 2026-07-06 cs.CV

PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

Pin Tang , Guoqing Wang , Xiangxuan Ren , Zhongdao Wang , Guodongfang Zhao , Bailan , Chao Ma This is my paper
classification cs.CV
keywords vision-language-action modelsautonomous drivingdata scaling2D trajectory planningreinforcement learningend-to-end drivingsensor-agnostic planning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing vision-language-action models for autonomous driving predict metric 3D trajectories straight from camera images. That forces the network to absorb camera-specific geometry, which blocks mixing heterogeneous datasets and often lets the model cheat by extrapolating from speed and acceleration instead of looking at the scene. PixelPilot instead plans future vehicle positions as pixel coordinates entirely in the image plane, trains only in that sensor-agnostic 2D space, and converts the pixels to 3D world coordinates solely at test time using the vehicle’s known camera calibration. Dense intermediate rewards during reinforcement learning tie the final path to explicit object detections and high-level meta-actions, enforcing a causal chain from perception to planning. The result is state-of-the-art open- and closed-loop performance and clear evidence that the model now relies more on images than on ego-status.

Core claim

PixelPilot establishes that reformulating end-to-end driving as sensor-agnostic 2D-to-2D planning in the image plane, followed by deterministic geometric lifting to 3D only at inference, removes the camera-parameter entanglement that has limited data scaling and visual reasoning in prior VLAs. With dense intermediate rewards from perception through planning, this yields superior open-loop accuracy (0.30 m average L2) and closed-loop driving score (79.14) while shifting reliance from ego-status to visual cues.

What carries the argument

The decoupled planning-and-lifting paradigm: the model autoregressively outputs 2D bounding boxes, box-grounded reasoning, meta-actions, and image-plane waypoints (optimized only in 2D); a fixed ray–plane intersection then lifts those waypoints to metric 3D using the target vehicle’s camera parameters and a local-plane height assumption.

Load-bearing premise

The road surface a few seconds ahead can be treated as a flat plane of fixed height relative to the car, so every planned pixel maps uniquely to a 3D point by simple ray casting.

What would settle it

Evaluate open-loop L2 and closed-loop success on multi-level roads, steep grades, or bridges where the three-second height change greatly exceeds the 0.16 m average reported for nuScenes; if accurate 2D plans systematically produce invalid 3D trajectories after lifting, the geometric premise fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Heterogeneous calibrated driving datasets can be mixed for training without forcing the network to learn dataset-specific 2D-to-3D mappings.
  • A trained policy can be moved to a new calibrated vehicle by swapping only the camera parameters used in the final lifting step.
  • Dense intermediate rewards on verifiable perception and meta-action outputs reduce collapse to pure ego-status kinematics.
  • Short-horizon planning can remain inside native vision-language image entities (boxes and pixels) while still producing metric vehicle control.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same 2D-first pattern may transfer to other embodied vision-language agents (manipulation, drones) whose sensor geometry differs across platforms but whose visual grounding is shared.
  • If local-plane lifting is the main limit, residual height correction or multi-plane selection applied only at lift time could extend the planning horizon without re-entangling training.
  • Explicit box-grounded reasoning chains may make driving VLAs more auditable by humans than free-form text alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. PixelPilot proposes a Vision-Language-Action model for end-to-end autonomous driving that decouples sensor-agnostic 2D planning from sensor-specific 3D lifting. Instead of predicting 3D trajectories directly from images (which entangles the model with camera parameters and encourages ego-status shortcuts), the method formulates perception, reasoning with bounding boxes, meta-action prediction, and trajectory planning entirely in the image plane (Eq. 1), then deterministically lifts 2D waypoints to 3D at inference via ray–plane intersection under a local-plane assumption Z = −h (Eq. 2). Training uses multi-task SFT followed by GRPO with dense intermediate rewards on format, perception IoU, meta-action F1, and 2D trajectory metrics (Eq. 3), without semantic rewards on free-form reasoning. Experiments report strong open-loop results on nuScenes (avg L2 0.30 m), closed-loop gains on Bench2Drive (DS 79.14, SR 58.87%), competitive zero-shot Waymo transfer, and ablations showing greater reliance on images than ego-status (Table 9).

Significance. If the claims hold, the paper offers a practical and conceptually clean alternative to the dominant 2D-to-3D VLA training recipe in autonomous driving. Decoupling learnable policy from camera-specific geometry is a useful systems insight: it enables training on heterogeneous calibrated datasets without baking sensor mappings into network weights, and the ego-status vs. image ablation (Table 9) provides concrete evidence that the design reduces kinematic shortcuts. The knowledge-instilled pipeline—explicit 2D boxes as reasoning anchors plus dense intermediate GRPO rewards—is a transferable recipe for long-horizon VLM policies. Strengths include multi-setting evaluation (perception, meta-action, open-loop, closed-loop, cross-dataset), component ablations (Tables 5–7), and an explicit scope discussion of the local-plane limitation. Absolute SOTA margins are modest, but the paradigm and the vision-vs-ego evidence are the more durable contributions.

major comments (3)
  1. [Sec. 3.1, Eq. (2), Table 10] Sec. 3.1 and Eq. (2): the deterministic lift assumes a fixed-height local plane Z = −h. The geometric justification (0.16 m average 3 s height change on nuScenes; bijective projective mapping under that approximation) is reasonable for short-horizon ego-centric planning, but Table 10 only shows that open-loop L2 remains comparable to Imprompt-VLA on high-height-change bins—it does not verify that the lifted 3D trajectory is geometrically correct when the plane is wrong. Because closed-loop control depends on the lift, the manuscript should report (i) sensitivity of Bench2Drive metrics to h, (ii) residual lift error vs. GT 3D on non-planar subsets, or (iii) a clearer bound on when the approximation fails. The conclusion already flags multi-level roads; this should be quantified rather than left as a qualitative caveat.
  2. [Abstract; Table 3; Table 4] Table 3 and the SOTA claim: PixelPilot’s open-loop avg L2 (0.30 m) matches Imprompt-VLA and is worse than several grayed or competing entries (FSDrive 0.28, AutoDrive-R2 0.20). Collision (0.25%) is not best-in-table. The closed-loop gains on Bench2Drive (Table 4) are clearer but still incremental over AutoVLA. The central contribution is better framed as a scalable, vision-grounded paradigm with competitive performance, not as unambiguous open-loop SOTA. Softening the abstract/intro SOTA language and emphasizing the ego-status ablation and cross-dataset scaling would better match the evidence.
  3. [Sec. 3.2, Stage 1] Sec. 3.2 (Reasoning with Bounding Boxes data generation): synthetic CoT traces from Qwen-VL-Max are retained only when they improve action/trajectory on the training set. This selection can bias the SFT prior toward cases where language helps and may overstate the causal role of the generated reasoning. A control that uses unfiltered or randomly retained reasoning traces (or reports how many candidates were discarded) would strengthen the claim that dense intermediate rewards enforce a perception→planning chain rather than post-hoc rationalization.
minor comments (5)
  1. [Fig. 5; Table 6] Fig. 5 and the ego-centric stitching discussion are important for multi-view continuity but are only lightly quantified (Table 6). A short qualitative failure case for simplest stitching vs. 180° rotation would help readers.
  2. [Eq. (2)–(3); Sec. 4.1] Reward weights λ_fmt, λ_percep, λ_action, λ_traj and the fixed plane height h are free parameters (Eq. 3; Eq. 2) but not listed with values or sensitivity in the main text. Please report them (or point to a specific supplementary table).
  3. [Abstract; Table 3] Table 3 grays FSDrive and AutoDrive-R2 for private data / kinematics; the caption is clear, but the abstract’s unqualified “state-of-the-art” still reads as absolute. Align wording.
  4. [Fig. 1; throughout] Typographical / formatting: occasional missing spaces after commas in figure captions and “2D-to-3D” hyphenation inconsistency; “unscalable” in Fig. 1 is slightly awkward—consider “sensor-specific / non-scalable.”
  5. [Table 11] Latency trade-off (Table 11) is useful; please state whether lifting time is included and whether multi-view collage resolution matches training.

Circularity Check

1 steps flagged

Empirical systems paper; no load-bearing circular derivation. Only a mild self-reinforcing filter on synthetic CoT training data.

specific steps
  1. other [Sec. 3.2 Stage 1, Reasoning with Bounding Boxes paragraph]
    "we first evaluate the baseline SFT model on the training set to identify failure cases. For these cases, we employ the Qwen-VL-Max model and provide it with a structured prompt containing the input image I, ground-truth action A, and the 2D bounding boxes B2D of surrounding agents to synthesize a causal reasoning chain... A sample is strictly selected only if the inclusion of reasoning leads to a tangible improvement, i.e., correcting the predicted action and reducing the trajectory L2 error."

    Synthetic CoT traces are kept only when they improve the same SFT model’s action and trajectory metrics on the training set. This is a mild self-reinforcing data filter: the CoT corpus is conditioned on reducing the target error of the model that will later be trained on it. It does not force the open-loop/closed-loop SOTA or the ego-status ablation by construction, but it can inflate the apparent value of the CoT chain relative to an independently curated reasoning set.

full rationale

PixelPilot is an empirical VLA systems paper, not a first-principles derivation. The core claims (sensor-agnostic 2D planning + deterministic 3D lift; dense GRPO rewards; better vision-vs-ego-status reliance; open/closed-loop SOTA) are tested against held-out nuScenes, Waymo, and Bench2Drive metrics and do not reduce by construction to fitted constants or self-cited uniqueness theorems. 2D box/trajectory labels are standard projections of existing 3D GT; evaluation L2 is measured after lift against real 3D GT under a stated local-plane approximation whose limitations the paper itself bounds (0.16 m avg height change; Table 10; Conclusion). The local-plane height Z=−h is an explicit modeling assumption, not a fitted parameter renamed as a prediction. Citations for the plane approximation ([30,35]) and GRPO ([45]) are external. The only circularity-adjacent practice is the SFT reasoning-data filter: synthetic CoT traces are retained only when they improve the same model’s action/trajectory metrics on the training set. That can mildly reinforce the SFT prior for the CoT component, but it is data curation rather than a claimed prediction forced by a fit, and it is not load-bearing for the main scalability or vision-over-ego-status results. Score 1 reflects that minor filter only; central claims remain independently falsifiable.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on geometric and training assumptions rather than free physical constants. Load-bearing premises are the local-plane bijective map, availability of camera parameters and ego-motion at inference, and the reward/format design that ties CoT to verifiable boxes and 2D waypoints. Free parameters are engineering choices (plane height, reward weights, pixel L1 threshold, GRPO group size). Invented entities are methodological constructs, not new physical objects.

free parameters (5)
  • local plane height h (Z = −h)
    Fixed ego-relative plane height used for ray–plane lifting; chosen empirically rather than estimated per scene.
  • trajectory L1 success threshold (10 pixels)
    Binary component of R_traj-L1; hand-chosen cutoff for dense RL reward.
  • reward weights λ_fmt, λ_percep, λ_action, λ_traj
    Linear combination of dense rewards in Eq. 3; not derived, must be set for GRPO.
  • GRPO completions per prompt (8)
    Group size for relative advantages; training hyperparameter.
  • decoding temperature / top-p / top-k (1.0 / 0.5 / 20)
    Inference sampling settings that affect reported trajectories.
axioms (5)
  • domain assumption For ~3 s ego-centric planning the road is approximately a local plane, so image waypoints and 3D road points are related by a bijective projective map.
    Sec. 3.1 theoretical justification; supported by 0.16 m average height change on nuScenes but not generally true for multi-level or steep roads.
  • domain assumption Avoiding 2D image-space overlaps of visible agents is a conservative cue for avoiding visible 3D collisions under accurate perception and the local-plane assumption.
    C-space / I-space obstacle analogy in Sec. 3.1; fails under occlusion or non-planar geometry.
  • domain assumption Target vehicle camera intrinsics/extrinsics and ego-motion are known and accurate at inference for deterministic lifting.
    Stated scope: weakly calibrated or uncalibrated 3D planning is out of scope (Sec. 3.1, Conclusion).
  • domain assumption Projecting existing 3D GT boxes/trajectories to 2D yields valid supervision for a policy that will never need 3D labels at deployment.
    Sec. 3.2 label construction; standard but couples training labels to 3D annotation pipelines.
  • ad hoc to paper Dense rewards on format, IoU, meta-action F1, and 2D trajectory metrics (without semantic reward on free-form think text) enforce a causal perception→planning chain.
    Core of knowledge-instilled GRPO design; plausible but not independently proven beyond ablations.
invented entities (3)
  • Decoupled planning and lifting paradigm no independent evidence
    purpose: Make the learnable VLA policy sensor-agnostic (2D-to-2D) while confining camera-specific geometry to non-learned inference.
    Central methodological construct; evidence is empirical performance and ego-status ablations, not an external physical measurement.
  • Reasoning with bounding boxes no independent evidence
    purpose: Force free-form CoT to reference explicit 2D RoIs before meta-action and trajectory.
    Training/data-generation construct; selected when it improves SFT metrics.
  • Knowledge-instilled policy learning (multi-task SFT + dense intermediate GRPO) no independent evidence
    purpose: Instill perception–action–planning structure then refine with verifiable intermediate rewards.
    Training recipe; supported by ablations removing reward terms or multi-task SFT.

pith-pipeline@v1.1.0-grok45 · 23031 in / 3941 out tokens · 34351 ms · 2026-07-11T16:00:13.909230+00:00 · methodology

0 comments
read the original abstract

Vision-Language-Action Models (VLAs), which leverage the advanced reasoning capabilities of Vision-Language Models (VLMs), show promising generalization in complex autonomous driving scenarios. Existing VLAs typically predict and optimize 3D trajectories from 2D images. While intuitive, this 2D-to-3D prediction is inherently entangled with camera parameters, leading to limited data scalability across heterogeneous driving datasets. Moreover, directly optimizing in 3D space induces severe convergence to trivial solutions, where VLAs rely on ego-status rather than visual scene understanding. To address these issues, we propose PixelPilot, a novel VLA featuring a decoupled planning and lifting paradigm. In the planning phase, PixelPilot reformulates scene understanding and trajectory prediction as sensor-agnostic 2D-to-2D tasks in the image plane, thereby facilitating scalable training across diverse datasets. The planned 2D trajectories are then deterministically lifted to 3D only during inference, ensuring the full exploitation of visual cues and generalization across different vehicles. To realize this paradigm, we propose a knowledge-instilled policy learning strategy that applies dense, intermediate rewards via Group Relative Policy Optimization (GRPO) to enforce a rigorous causal chain from visual perception to spatial planning. Extensive experiments demonstrate that PixelPilot achieves state-of-the-art performance in both open-loop and closed-loop settings, validating its superior scalability and visual reasoning capabilities.

Figures

Figures reproduced from arXiv: 2607.04637 by Bailan, Chao Ma, Guodongfang Zhao, Guoqing Wang, Pin Tang, Xiangxuan Ren, Zhongdao Wang.

Figure 1
Figure 1. Figure 1: (a) Previous VLAs directly predict 3D trajectories from 2D images which in￾volves learning sensor-specific 2D-to-3D mapping, leading to limited data scalability. (b) Our PixelPilot features a decoupled planning and lifting paradigm. It plans and optimizes 2D trajectories within the image plane, facilitating data scaling and visual reasoning, while unscalable 2D-to-3D trajectory lifting occurs only during i… view at source ↗
Figure 2
Figure 2. Figure 2: Decoupled planning and lifting paradigm. The sensor-agnostic planning phase is scalable and introduces reasoning with bounding boxes to explore visual cues, while the sensor-specific lifting is unscalable and is performed only at inference. 3 Method 3.1 Decoupled Planning and Lifting Paradigm Previous Vision-Language-Action Models (VLAs) are typically fine-tuned to pre￾dict 3D trajectories directly. While … view at source ↗
Figure 3
Figure 3. Figure 3: Justification for decoupled planning and lifting paradigm. (a) The bijective trajectory pairs demonstrate trajectory feasibility under a local-plane assumption. (b) Planning in image space provides a conservative cue for avoiding visible collision risks. (2) Interaction Safety. The safety motivation of our 2D planning approach is justified through an analogy to Configuration Space (C-Space) planning [6,38]… view at source ↗
Figure 4
Figure 4. Figure 4: Training pipeline of PixelPilot. We propose a knowledge-instilled policy learning strategy. The first stage instills foundational driving knowledge into the VLA by multi￾task SFT. The second stage assigns rewards on the holistic VLA pipeline for coherence. (a) Simplest Stitching Front Back Front Back (b) Back Views Horizontally Flipped Stitching Front Left Front Front Right Back Left Back Back Right (c) Ba… view at source ↗
Figure 5
Figure 5. Figure 5: Different image preprocessing methods for VLAs. separate rows, introducing severe egocentric inconsistency, such as lane mark￾ing misalignment when using the simplest stitching ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of 2D trajectories across Qwen2.5-VL-7B, OmniDrive, and our PixelPilot on the nuScenes validation dataset. The predicted and ground-truth trajec￾tories are depicted in blue and green, respectively [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 20 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  3. [3]

    NeurIPS pp

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS pp. 1877–1901 (2020)

    1901

  4. [4]

    In: CVPR

    Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: CVPR. pp. 11618–11628 (2020)

    2020

  5. [5]

    In: AAAI

    Cai, T., Liu, Y., Zhou, Z., Ma, H., Zhao, S.Z., Wu, Z., Ma, J.: Driving with regula- tion: Trustworthy and interpretable decision-making for autonomous driving with retrieval-augmented reasoning. In: AAAI. pp. 38287–38295 (2026)

    2026

  6. [6]

    In: ICCV

    Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: Learning affordance for direct perception in autonomous driving. In: ICCV. pp. 2722–2730 (2015)

    2015

  7. [7]

    IEEE TPAMI pp

    Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., Li, H.: End-to-end autonomous driving: Challenges and frontiers. IEEE TPAMI pp. 10164–10183 (2024)

    2024

  8. [8]

    arXiv preprint arXiv:2402.13243 (2024)

    Chen,S.,Jiang,B.,Gao,H.,Liao,B.,Xu,Q.,Zhang,Q.,Huang,C.,Liu,W.,Wang, X.: Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243 (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    In: CVPR

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: CVPR. pp. 24185–24198 (2024)

    2024

  10. [10]

    NeurIPS (2026)

    Chi, H., Gao, H.a., Liu, Z., Liu, J., Liu, C., Li, J., Yang, K., Yu, Y., Wang, Z., Li, W., et al.: Impromptu vla: Open weights and open data for driving vision- language-action models. NeurIPS (2026)

    2026

  11. [11]

    arXiv preprint arXiv:2312.16886 (2023)

    Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al.: Mobilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 (2023)

    Pith/arXiv arXiv 2023

  12. [12]

    NeurIPS pp

    Dauner, D., Hallgarten, M., Li, T., Weng, X., Huang, Z., Yang, Z., Li, H., Gilitschenski, I., Ivanovic, B., Pavone, M., et al.: Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking. NeurIPS pp. 28706– 28719 (2024)

    2024

  13. [13]

    In: ICLR (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)

    2021

  14. [14]

    In: ICCV

    Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C.R., Zhou, Y., et al.: Large scale interactive motion forecasting for au- tonomous driving: The waymo open motion dataset. In: ICCV. pp. 9690–9699 (2021)

    2021

  15. [15]

    In: WACVW

    Fu, D., Li, X., Wen, L., Dou, M., Cai, P., Shi, B., Qiao, Y.: Drive like a human: Rethinking autonomous driving with large language models. In: WACVW. pp. 910–919 (2024)

    2024

  16. [16]

    In: ICCV

    Fu, H., Zhang, D., Zhao, Z., Cui, J., Liang, D., Zhang, C., Zhang, D., Xie, H., Wang, B., Bai, X.: Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation. In: ICCV. pp. 24823–24834 (2025) PixelPilot 17

    2025

  17. [17]

    Nature645(8081), 633–638 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al.: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (2025)

    2025

  18. [18]

    In: AAAI

    Han, W., Guo, D., Xu, C.Z., Shen, J.: Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving. In: AAAI. pp. 3347–3355 (2025)

    2025

  19. [19]

    In: ECCV

    Hu, S., Chen, L., Wu, P., Li, H., Yan, J., Tao, D.: St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In: ECCV. pp. 533–549 (2022)

    2022

  20. [20]

    In: CVPR

    Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., Lu, L., Jia, X., Liu, Q., Dai, J., Qiao, Y., Li, H.: Planning-oriented autonomous driving. In: CVPR. pp. 17853–17862 (2023)

    2023

  21. [21]

    arXiv preprint arXiv:2112.11790 (2021)

    Huang, J., Huang, G., Zhu, Z., Du, D.: Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790 (2021)

    Pith/arXiv arXiv 2021

  22. [22]

    arXiv preprint arXiv:2412.07689 (2024)

    Huang, Z., Fen, C., Yan, F., Xiao, B., Jie, Z., Zhong, Y., Liang, X., Ma, L.: Drivemm: All-in-one large multimodal model for autonomous driving. arXiv preprint arXiv:2412.07689 (2024)

    Pith/arXiv arXiv 2024

  23. [23]

    arXiv preprint arXiv:2410.23262 (2024)

    Hwang, J.J., Xu, R., Lin, H., Hung, W.C., Ji, J., Choi, K., Huang, D., He, T., Cov- ington, P., Sapp, B., et al.: Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262 (2024)

    Pith/arXiv arXiv 2024

  24. [24]

    In: ICCV

    Jia, X., Gao, Y., Chen, L., Yan, J., Liu, P.L., Li, H.: Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In: ICCV. pp. 7919–7929 (2023)

    2023

  25. [25]

    NeurIPS pp

    Jia, X., Yang, Z., Li, Q., Zhang, Z., Yan, J.: Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. NeurIPS pp. 819–844 (2024)

    2024

  26. [26]

    In: ICCV

    Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang,C.,Wang,X.:Vad:Vectorizedscenerepresentationforefficientautonomous driving. In: ICCV. pp. 8306–8316 (2023)

    2023

  27. [27]

    arXiv preprint arXiv:2503.07608 (2025)

    Jiang, B., Chen, S., Zhang, Q., Liu, W., Wang, X.: Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    In: CVPR

    Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR. pp. 9579–9589 (2024)

    2024

  29. [29]

    In: ICLR (2025)

    Lan, M., Chen, C., Zhou, Y., Xu, J., Ke, Y., Wang, X., Feng, L., Zhang, W.: Text4seg: Reimagining image segmentation as text generation. In: ICLR (2025)

    2025

  30. [30]

    In: IROS

    Lee, S., Lim, H., Myung, H.: Patchwork++: Fast and robust ground segmentation solving partial under-segmentation using 3d point cloud. In: IROS. pp. 13276– 13283 (2022)

    2022

  31. [31]

    In: CVPR

    Li, B., Wang, Y., Mao, J., Ivanovic, B., Veer, S., Leung, K., Pavone, M.: Driving everywhere with large language model policy adaptation. In: CVPR. pp. 14948– 14957 (2024)

    2024

  32. [32]

    In: ICML

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: ICML. pp. 19730–19742 (2023)

    2023

  33. [33]

    In: ECCV

    Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Yu, Q., Dai, J.: Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotem- poral transformers. In: ECCV. pp. 1–18 (2022)

    2022

  34. [34]

    Li, Z., Yu, Z., Lan, S., Li, J., Kautz, J., Lu, T., Alvarez, J.M.: Is ego status all you need for open-loop end-to-end autonomous driving? In: CVPR. pp. 14864–14873 (June 2024) 18 P. Tang et al

    2024

  35. [35]

    IEEE RA- L pp

    Lim, H., Oh, M., Myung, H.: Patchwork: Concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor. IEEE RA- L pp. 6458–6465 (2021)

    2021

  36. [36]

    In: NeurIPS

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS. pp. 34892–34916 (2023)

    2023

  37. [37]

    arXiv preprint arXiv:2512.12799 (2025)

    Liu, Z., Huang, R., Yang, R., Yan, S., Wang, Z., Hou, L., Lin, D., Bai, X., Zhao, H.: Drivepi: Spatial-aware 4d mllm for unified autonomous driving understanding, perception, prediction and planning. arXiv preprint arXiv:2512.12799 (2025)

    arXiv 2025

  38. [38]

    IEEE TC pp

    Lozano-Perez, T.: Spatial planning: A configuration space approach. IEEE TC pp. 108–120 (1983)

    1983

  39. [39]

    In: ECCV

    Ma, Y., Cao, Y., Sun, J., Pavone, M., Xiao, C.: Dolphins: Multimodal language model for driving. In: ECCV. pp. 403–420 (2024)

    2024

  40. [40]

    In: ECCV

    Marcu, A.M., Chen, L., Hünermann, J., Karnsund, A., Hanotte, B., Chidananda, P., Nair, S., Badrinarayanan, V., Kendall, A., Shotton, J., et al.: Lingoqa: Visual question answering for autonomous driving. In: ECCV. pp. 252–269 (2024)

    2024

  41. [41]

    In: Findings of EMNLP

    Qian, K., Jiang, S., Zhong, Y., Luo, Z., Huang, Z., Zhu, T., Jiang, K., Yang, M., Fu, Z., Miao, J., et al.: Agentthink: A unified framework for tool-augmented chain-of- thought reasoning in vision-language models for autonomous driving. In: Findings of EMNLP. pp. 10663–10682 (2025)

    2025

  42. [42]

    In: ICML

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)

    2021

  43. [43]

    arXiv preprint arXiv:2512.10554 (2025)

    Ren, X., Wang, Z., Hou, L., Tang, P., Wang, G., Ma, C.: Grounding everything in tokens for multimodal large language models. arXiv preprint arXiv:2512.10554 (2025)

    Pith/arXiv arXiv 2025

  44. [44]

    In: CVPR

    Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S.L., Liu, Y., Li, H.: Lmdrive: Closed-loop end-to-end driving with large language models. In: CVPR. pp. 15120– 15130 (2024)

    2024

  45. [45]

    arXiv preprint arXiv:2402.03300 (2024)

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  46. [46]

    In: ECCV

    Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: Drivelm: Driving with graph visual question answering. In: ECCV. pp. 256–274 (2024)

    2024

  47. [47]

    arXiv preprint arXiv:2503.13047 (2025)

    Song, R., Guo, X., Wu, H., Wei, Q., Chen, L.: Insightdrive: Insight scene rep- resentation for end-to-end autonomous driving. arXiv preprint arXiv:2503.13047 (2025)

    arXiv 2025

  48. [48]

    In: CVPR

    Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: CVPR. pp. 2443–2451 (2020)

    2020

  49. [49]

    In: ICRA

    Sun, W., Lin, X., Shi, Y., Zhang, C., Wu, H., Zheng, S.: Sparsedrive: End-to- end autonomous driving via sparse scene representation. In: ICRA. pp. 8795–8801 (2025)

    2025

  50. [50]

    In: ECCV

    Tang, Y., Meng, Z., Chen, G., Cheng, E.: Simpb: A single model for 2d and 3d object detection from multiple cameras. In: ECCV. pp. 1–17 (2024)

    2024

  51. [51]

    arXiv preprint arXiv:2402.12289 (2024)

    Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., Zhao, H.: Drivevlm: The convergence of autonomous driving and large vision- language models. arXiv preprint arXiv:2402.12289 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    arXiv preprint arXiv:2302.13971 (2023) PixelPilot 19

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) PixelPilot 19

    Pith/arXiv arXiv 2023

  53. [53]

    In: ICCV

    Wang, S., Liu, Y., Wang, T., Li, Y., Zhang, X.: Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In: ICCV. pp. 3598–3608 (2023)

    2023

  54. [54]

    In: CVPR

    Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., Kautz, J., Li, Y., Alvarez, J.M.: Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In: CVPR. pp. 22442–22452 (2025)

    2025

  55. [55]

    arXiv preprint arXiv:2403.16996 (2024)

    Wang, T., Xie, E., Chu, R., Li, Z., Luo, P.: Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint arXiv:2403.16996 (2024)

    Pith/arXiv arXiv 2024

  56. [56]

    Visual Intelligence3(1) (2025)

    Wang, W., Xie, J., Hu, C., Zou, H., Fan, J., Tong, W., Wen, Y., Wu, S., Deng, H., Li, Z., et al.: Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. Visual Intelligence3(1) (2025)

    2025

  57. [57]

    In: ICCV

    Wang, X., Zhu, Z., Xu, W., Zhang, Y., Wei, Y., Chi, X., Ye, Y., Du, D., Lu, J., Wang, X.: Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In: ICCV. pp. 17804–17813 (2023)

    2023

  58. [58]

    In: CoRL

    Wang, Y., Guizilini, V.C., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In: CoRL. pp. 180–191 (2022)

    2022

  59. [59]

    In: CVPR

    Wang, Y., He, J., Fan, L., Li, H., Chen, Y., Zhang, Z.: Driving into the future: Mul- tiview visual forecasting and planning with world model for autonomous driving. In: CVPR. pp. 14749–14759 (2024)

    2024

  60. [60]

    In: ICCV

    Wang, Z., Huang, Z., Fu, J., Wang, N., Liu, S.: Object as query: Lifting any 2d object detector to 3d detection. In: ICCV. pp. 3768–3777 (2023)

    2023

  61. [61]

    NeurIPS pp

    Wu, P., Jia, X., Chen, L., Yan, J., Li, H., Qiao, Y.: Trajectory-guided control pre- diction for end-to-end autonomous driving: A simple yet strong baseline. NeurIPS pp. 6119–6132 (2022)

    2022

  62. [62]

    IEEE RA-L pp

    Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.Y.K., Li, Z., Zhao, H.: Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE RA-L pp. 8186–8193 (2024)

    2024

  63. [63]

    In: CVPR

    Yang,L.,Kang,B.,Huang,Z.,Xu,X.,Feng,J.,Zhao,H.:Depthanything:Unleash- ing the power of large-scale unlabeled data. In: CVPR. pp. 10371–10381 (2024)

    2024

  64. [64]

    arXiv preprint arXiv:2505.16278 (2025)

    Yang, Z., Chai, Y., Jia, X., Li, Q., Shao, Y., Zhu, X., Su, H., Yan, J.: Drivemoe: Mixture-of-experts for vision-language-action model in end-to-end autonomous driving. arXiv preprint arXiv:2505.16278 (2025)

    Pith/arXiv arXiv 2025

  65. [65]

    arXiv preprint arXiv:2308.01006 (2023)

    Ye, T., Jing, W., Hu, C., Huang, S., Gao, L., Li, F., Wang, J., Guo, K., Xiao, W., Mao, W., et al.: Multi-modality fusion for prediction and planning tasks of autonomous driving. arXiv preprint arXiv:2308.01006 (2023)

    Pith/arXiv arXiv 2023

  66. [66]

    IEEE Access12, 54608–54649 (2024)

    Yenduri, G., Srivastava, G., Maddikunta, P.K.R., Jhaveri, R.H., Wang, W., Vasi- lakos, A.V., Gadekallu, T.R., et al.: Gpt (generative pre-trained transformer): A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. IEEE Access12, 54608–54649 (2024)

    2024

  67. [67]

    arXiv preprint arXiv:2509.01944 (2025)

    Yuan, Z., Tang, J., Luo, J., Chen, R., Qian, C., Sun, L., Chu, X., Cai, Y., Zhang, D., Li, S.: Autodrive-r2: Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving. arXiv preprint arXiv:2509.01944 (2025)

    Pith/arXiv arXiv 2025

  68. [68]

    arXiv preprint arXiv:2505.17685 (2025)

    Zeng, S., Chang, X., Xie, M., Liu, X., Bai, Y., Pan, Z., Xu, M., Wei, X.: Future- sightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505.17685 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    arXiv preprint arXiv:2305.10430 (2023) 20 P

    Zhai, J.T., Feng, Z., Du, J., Mao, Y., Liu, J.J., Tan, Z., Zhang, Y., Ye, X., Wang, J.: Rethinkingtheopen-loopevaluationofend-to-endautonomousdrivinginnuscenes. arXiv preprint arXiv:2305.10430 (2023) 20 P. Tang et al

    Pith/arXiv arXiv 2023

  70. [70]

    In: CVPR

    Zhang, B., Song, N., Jin, X., Zhang, L.: Bridging past and future: End-to-end autonomous driving with historical prediction and planning. In: CVPR. pp. 6854– 6863 (2025)

    2025

  71. [71]

    arXiv preprint arXiv:2507.20879 (2025)

    Zheng, W., Mao, X., Ye, N., Li, P., Zhan, K., Lang, X., Zhao, H.: Driveagent-r1: Advancing vlm-based autonomous driving with hybrid thinking and active percep- tion. arXiv preprint arXiv:2507.20879 (2025)

    Pith/arXiv arXiv 2025

  72. [72]

    In: AAAI

    Zhou, X., Han, X., Yang, F., Ma, Y., Knoll, A.C.: Opendrivevla: Towards end-to- end autonomous driving with large vision language action model. In: AAAI. pp. 13782–13790 (2026)

    2026

  73. [73]

    In: NeurIPS (2025)

    Zhou, Z., Cai, T., Zhao, S.Z., Zhang, Y., Huang, Z., Zhou, B., Ma, J.: Autovla: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In: NeurIPS (2025)

    2025

  74. [74]

    In: ICLR (2024)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. In: ICLR (2024)

    2024

  75. [75]

    arXiv preprint arXiv:2504.10479 (2025)

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025)

    Pith/arXiv arXiv 2025

  76. [76]

    In: ICLR (2021)

    Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. In: ICLR (2021)

    2021