{"id":"e0c60db0-22e2-488a-be8d-fd659dcf93b3","arxiv_id":"2501.08861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 3D-vision-language pre-training model with group-wise contrastive alignment generates driving trajectories as text and reports state-of-the-art open-loop planning results on nuScenes.","lead":"GPVL is an end-to-end driving model that aligns 3D bird's-eye-view perception with language features and generates driving trajectories as text. It reports state-of-the-art open-loop planning results on nuScenes, with lower displacement error and collision rate than strong baselines like VAD and UniAD.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Nav prompt in Eq. (8) includes the ego car's current 3D box, while the text and Table 1 claim ego status is excluded; this unresolved contradiction may enable shortcut planning and undermine the SOTA comparison.","rationale":"The reader's weakest assumption was that BERT, a bidirectional encoder, cannot straightforwardly compute the autoregressive distribution in Eq. (9). That is a legitimate reproducibility concern, but it is secondary: even if the BERT issue were resolved, the reported numbers could still be invalid if the ego box was present during evaluation. The paper's own text creates a direct contradiction between the method section (which provides the ego box in the Nav prompt) and the evaluation section (which claims ego status is deactivated). BEV-Planner (Li et al., CVPR 2024) is cited precisely because ego status enables shortcut learning in open-loop planning, so this is not an abstract worry. The central claim of state-of-the-art planning performance depends entirely on whether the prompt leakage is real. Consequently, the stress-test should shift attention from the architectural detail to the validity of the experimental protocol. The verdict remains CONDITIONAL because the concern is addressable: the authors can clarify the prompt used, rerun the experiment without the ego box, and verify whether the improvements survive. If the improvement collapses, the paper's main contribution would be unsupported. This is a more load-bearing concern than the BERT masking question, which is why agreement with the reader is 'disagree' rather than 'agree' or 'partial'.","tokens_in":12640,"tokens_out":7174,"duration_ms":75262,"concrete_test":"Obtain the released code and the exact prompt strings used for training and for open-loop evaluation. Then compare two variants on the nuScenes val set: (A) the full prompt including the ego box, and (B) the same prompt with '[cx, cy, cz, cw, ch, cl]' replaced by a constant marker. If variant B's average L2 rises substantially (e.g., from 0.43 to above 0.7 m) or its collision rate rises to the level of UniAD/VAD, the reported superiority is largely an artifact of ego-box leakage. Also report an identity baseline that repeats the current ego pose for all future timestamps; if its L2 is close to GPVL's 0.21/0.39/0.69, the shortcut is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is the open-loop nuScenes SOTA in Table 1. Two parts of the paper conflict on a critical input. Section 'Planning via Cross-modal Language Model' defines the Nav prompt as 'The box of ego-car is [cx, cy, cz, cw, ch, cl].' That is explicit ego status. The Experiment section and the Table 1 caption instead state that 'the ego status information of GPVL is deactivated' and cite BEV-Planner, which demonstrated that ego-state inputs let a planner copy the ego's current kinematic state and report artificially low L2. The paper never specifies whether the ego box was removed from the prompt during the reported open-loop evaluations, nor how 'deactivated' was implemented. If the box remained in the prompt, the comparison with UniAD/VAD is unfair and the very low L2/collision scores (0.43 m / 0.14%) may be inflated by this shortcut. If it was removed, the method description is inaccurate and the model's exact inputs are undefined. Either way, the central claim is not supported by a clearly specified, valid experimental protocol.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GPVL, an end-to-end autonomous driving framework that combines a BEVFormer-based perception backbone with a 3D-vision language pre-training module (group-wise contrastive alignment) and a cross-modal language model that generates driving decisions and trajectories in an autoregressive text format. Experiments on nuScenes report the lowest open-loop planning L2 errors (0.21 m at 1 s, 0.39 m at 2 s, 0.69 m at 3 s, 0.43 m average) and the lowest average collision rate (0.14%) among 13 baselines, along with improved zero-shot city transfer and robustness to image corruptions compared with UniAD and VAD. Ablations show that each proposed component contributes to the final performance.","tokens_in":12808,"tokens_out":6688,"duration_ms":67998,"significance":"If the results hold, GPVL would be a meaningful step toward integrating language-based reasoning with 3D visual perception for trajectory planning, and the paper provides a useful comparison across many non-autoregressive and LLM-injected baselines. The release of code is a positive feature. However, the central SOTA claim is currently not supported by a clearly specified experimental protocol: there is an unresolved contradiction about whether ego status is used, and the generative decoding mechanism is underspecified because BERT is treated as an autoregressive decoder without explaining how causal generation is implemented. These issues are load-bearing for the reported planning results and for reproducibility.","major_comments":[{"comment":"The paper directly contradicts itself on whether ego status information is used. The Nav template in the 'Planning via Cross-modal Language Model' section reads 'The box of ego-car is [cx, cy, cz, cw, ch, cl]' and is fed into the language model in Eq. (8). Yet the Experiment section and the Table 1 caption state that 'the ego status information of GPVL is deactivated' to prevent shortcut learning, citing BEV-Planner. The manuscript never specifies how deactivation is implemented (e.g., removing the box tokens, masking them, or replacing them with a placeholder) or whether the results in Tables 1-5 were obtained with or without the ego box. If the box remains, the comparison with UniAD and VAD is unfair and the very low L2/collision values may reflect the shortcut documented by BEV-Planner; if it was removed, the method description is inaccurate and the actual input format is undefined. This must be resolved before the central SOTA claim can be accepted.","section":"§Planning via Cross-modal Language Model; Table 1 caption; §Comparison with State-of-the-art Methods"},{"comment":"The cross-modal language model is said to use the BERT structure, but BERT is a bidirectional encoder, not a causal decoder. Eq. (9) defines an autoregressive factorization p(w_t|w_{1:t-1}) and Eq. (10) trains with teacher forcing, yet the paper gives no information about causal masking, positional encodings, or how bidirectional attention is converted into sequential generation. In addition, the trajectory tokenization is completely unspecified: continuous future waypoints must be discretized into a vocabulary, but the manuscript never defines the coordinate frame, quantization step, vocabulary size, or the textual template of the reference caption G_t. Without these details, the generative planning mechanism is not reproducible and the reported L2 values cannot be independently checked.","section":"§Planning via Cross-modal Language Model, Eqs. (8)-(10); §Implementation Details"},{"comment":"The comparison with baselines may be confounded by additional data and supervision. The 2D scene captioning model is 'trained and tested on the Ominidrive-nuScenes dataset', and the 3D-vision language pre-training also uses nuScenes-derived data. Since UniAD, VAD, and the other baselines are trained without this captioning or VL-pretraining supervision, the gains shown in Tables 1, 4, and 5 cannot be attributed solely to the architectural contributions unless the paper specifies the exact train/val splits, confirms that no nuScenes validation scenes were used during captioner or VL pretraining, and ideally reports a variant that removes the additional data. This is a fairness and reproducibility issue for the SOTA claim.","section":"Introduction; §Dataset; §Comparison with State-of-the-art Methods"},{"comment":"The headline results are reported as single runs without error bars or multiple seeds. For example, GPVL's average collision rate is 0.14% versus 0.22% for VAD and 0.31% for UniAD, and some zero-shot and robustness differences are only a few hundredths of a meter. Without variance estimates, it is not possible to determine whether these differences are statistically meaningful. The paper should report standard deviations over at least three seeds or otherwise quantify evaluation uncertainty.","section":"Table 1; Table 4; Table 5"}],"minor_comments":[{"comment":"The similarity function s is used generically in Eq. (5) but is only fully defined for the detection group in Eq. (6); please provide the analogous definitions for the motion, map, and global groups and clarify the roles of W1 and W2.","section":"Eqs. (5)-(6)"},{"comment":"The reference list contains LaTeX encoding artifacts, for example 'BEVformer: Learning bird¡¯s-eye-view representation from multi-camera images via spatiotemporal transformers'; these should be cleaned up before publication.","section":"References"},{"comment":"The caption states that 'GPVL achieves the highest score on most evaluation metrics' although lower L2 and collision values are better; please rephrase to avoid ambiguity.","section":"Table 1 caption"},{"comment":"The decoding and evaluation pipeline should be described end to end: greedy search with size 1 produces text, but the manuscript does not explain how the generated text is parsed back into continuous waypoints for L2 evaluation; this step is essential for reproducibility.","section":"§Planning via Cross-modal Language Model"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially interesting, but the ego-status contradiction and the unspecified BERT autoregressive adaptation are not merely stylistic issues; they bear directly on whether the results in Table 1 are valid and reproducible. I would ask the editor to have the authors confirm that the ego box was removed in all reported open-loop evaluations and to provide the masking, tokenization, and vocabulary details. If the ego box was included in the reported numbers, the comparison with BEV-Planner, UniAD, and VAD would need to be redone; if it was deactivated, the method description needs correction. The code release is helpful, and I would also request verification that the released code matches the 'deactivated' configuration described in the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2501.08861.\n\nThe thing to know up front: the paper's central open-loop claim is clouded by a direct contradiction in the method description. In Eq. (8) the Nav prompt is given as \"The box of ego-car is [cx, cy, cz, cw, ch, cl] ...\", which is explicit ego status. The experiments section and Table 1 caption say ego status is \"deactivated\" to prevent shortcut learning, citing BEV-Planner. There is no statement of what \"deactivated\" means—whether the box tokens were removed from the prompt, zeroed, or the language model's attention to them was masked. If the box stayed in, the comparison with UniAD/VAD is unfair and the 0.21 m / 0.14% numbers could be shortcut-driven. If it was removed, the method description is inaccurate and readers don't know the actual input. Either way, the strongest empirical claim is not backed by a well-specified protocol. This is the first thing a reviewer should push on.\n\nWhat is genuinely new: the group-wise 3D vision-language alignment over detection, motion, map, and global groups, combined with autoregressive trajectory generation in a language model. I don't know of an exact prior combination. The ablations show each piece contributes, the zero-shot city transfer and robustness-to-noise tables are useful extras, and the authors at least cite BEV-Planner's warning even if they don't clearly honor it.\n\nOther soft spots: the BERT-as-autoregressive-decoder is odd. Eq. (9) writes p(w_t|w_1:t-1), but BERT is bidirectional; the paper never says how the mask is applied or how the trajectory tokens are tokenized. That's a reproducibility gap. No error bars or seeds anywhere. 5.1 FPS is not \"real-time potential\" for a production planner. And the code link is unverifiable from the paper.\n\nNet: I'd send this to peer review—the core idea is worth referee time, and the flaws are addressable with a careful revision. But I would not trust the numbers until the ego-status question is answered and the language-model details are nailed down. It's a conditional, not a reject, and not a pass either.","headline":"A promising architecture undermined by an unresolved ego-status contradiction in the evaluation protocol; worth refereeing but not trustworthy as-is.","tokens_in":13415,"tokens_out":2448,"would_cite":false,"duration_ms":23652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end driving planner that casts trajectory prediction as language generation reports the lowest displacement error and collision rate on the nuScenes open-loop benchmark.","keywords":["autonomous driving","vision-language pre-training","bird's-eye view","trajectory planning","autoregressive generation","end-to-end planning","nuScenes","3D perception"],"falsifier":"Run the authors' released code on the nuScenes val split with the same training protocol; if the reported 0.43 m average L2 and 0.14% collision rate are not reproduced, the central performance claim fails.","tokens_in":12361,"feed_emoji":"🚗","tokens_out":9476,"duration_ms":87560,"temperature":0.7,"pith_summary":"The paper introduces GPVL, an end-to-end autonomous driving model that frames trajectory planning as a language generation problem. It argues that aligning bird's-eye-view perception features with text descriptions, then feeding them to a language model, lets the system produce safe future trajectories. On the nuScenes open-loop planning benchmark, GPVL reports the lowest L2 displacement error (0.21 m at 1 s, 0.43 m average) and lowest average collision rate (0.14%) among 13 compared methods, and it also shows better zero-shot transfer to a new city and stronger robustness to weather noise than UniAD and VAD. The intended value is that marrying 3D perception with linguistic reasoning improves both safety and generalization in end-to-end driving.","feed_headline":"GPVL writes trajectories as text and beats 13 planners on nuScenes","feed_subtitle":"3D-vision language pre-training on bird's-eye views cuts 3-second error to 0.69 m and collision rate to 0.27%.","key_machinery":"The central mechanism is a group-wise vision-language alignment combined with an autoregressive trajectory generator. The alignment treats detection, motion, map, and global BEV features as visual 'groups' and matches each to a corresponding text description through a contrastive loss with a learnable temperature and similarity function. The trajectory generator then works like a captioning model: it takes the aligned BEV feature, a 2D scene caption, and a navigation instruction, and produces the future trajectory as a sequence of tokens in natural language, maximizing the likelihood of each token given the previous ones.","core_discovery":"The central claim is that planning in autonomous driving can be improved by formulating it as generative language modeling over 3D visual features. GPVL first extracts a bird's-eye-view feature map with a pretrained BEVformer and decodes detection, motion, and map features. A 3D-vision language pre-training module uses a group-wise contrastive alignment (detection, motion, map, and global groups) to project these visual features and environmental text descriptions into a shared space. A cross-modal language model, implemented with BERT, then receives the global visual feature, a 2D scene caption, and a navigation prompt, and autoregressively emits a trajectory caption, trained with a cross-entropy loss. The paper reports that this design yields the lowest L2 displacement error (0.21 m at 1 s, 0.43 m average) and lowest average collision rate (0.14%) on nuScenes open-loop planning, with better zero-shot city transfer and robustness to weather noise than UniAD and VAD.","pith_inferences":["If trajectory-as-text becomes an interchange format, plans could be inspected, amended, or constrained purely through natural-language prompts, which regression-head planners do not support.","The same group-wise alignment scheme could be reused in other embodied settings where 3D scene features and language coexist, such as indoor robot navigation or drone planning.","A natural follow-up is ablating the 2D scene captioner: the paper leaves open whether the aligned 3D features alone would carry the language grounding or whether the 2D caption is essential.","Testing the pipeline with a decoder-only causal language model would clarify how much of the reported performance depends on the specific bidirectional encoder, since causal generation is the standard way to evaluate autoregressive trajectory sampling."],"forward_implications":["Trajectory planning becomes a sequence-generation task: the same training and inference machinery used for text generation (greedy search, beam search, prompt-conditioning) can be applied to driving.","The group-wise vision-language alignment can act as a pretraining objective for other end-to-end driving components, improving them even before the generative planner is trained.","Language-grounded planners are less prone to shortcut learning on the straight-road majority of nuScenes, as shown by their stronger metrics on turn-left and turn-right subsets.","The reported latency (198 ms) and throughput (5.1 FPS) are compatible with near-real-time deployment, a prerequisite for practical use in future autonomous driving systems."],"supporting_citations":[{"why":"Extracts the bird's-eye-view feature map from multi-view cameras on which all perception heads run.","marker":"Li et al. 2022d"},{"why":"Supplies the vectorized detection, motion, and mapping transformers that produce the 3D perception features.","marker":"Jiang et al. 2023"},{"why":"Provides the textual attention block and the language model used for both alignment and trajectory generation.","marker":"Devlin et al. 2019"},{"why":"The contrastive learning formulation the group-wise alignment loss is modeled after.","marker":"Radford et al. 2021"},{"why":"The pretrained model used to produce the 2D scene captions that feed the planner.","marker":"Li et al. 2022a"},{"why":"The source of the Ominidrive-nuScenes captioning dataset for training the scene captioning model.","marker":"Wang et al. 2024"},{"why":"The dataset and benchmark all open-loop planning metrics are computed on.","marker":"Caesar et al. 2020"},{"why":"The planning-oriented end-to-end baseline that GPVL is compared against and improves upon.","marker":"Hu et al. 2023"},{"why":"The prior work used to justify excluding ego status to avoid shortcut learning in open-loop evaluation.","marker":"Li et al. 2024"}],"fun_headline_variants":["GPVL turns driving into language, tops nuScenes planning","Language-guided planning: GPVL beats 13 baselines on nuScenes","Write the trajectory, drive the car: GPVL's generative planning","3D vision-language pre-training drives GPVL to lowest errors","Autonomous driving via language: GPVL tops nuScenes with 0.14% collision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's performance numbers rest on treating a bidirectional text-understanding model (BERT) as a next-token generator for trajectories, a capability the paper assumes but does not demonstrate.","fun_headline_variants_meta":{"raw":{"variants":["GPVL turns driving into language, tops nuScenes planning","Language-guided planning: GPVL beats 13 baselines on nuScenes","Write the trajectory, drive the car: GPVL's generative planning","3D vision-language pre-training drives GPVL to lowest errors","Autonomous driving via language: GPVL tops nuScenes with 0.14% collision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000948,"raw_usage":{"total_tokens":4066,"prompt_tokens":986,"completion_tokens":3080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2992}},"tokens_in":602,"tokens_out":3080,"duration_ms":19829,"temperature":1.0,"reasoning_tokens":2992,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:15:22.997529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the authors' released code on the nuScenes val split with the same training protocol; if the reported 0.43 m average L2 and 0.14% collision rate are not reproduced, the central performance claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the vectorized detection, motion, and mapping transformers that produce the 3D perception features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the textual attention block and the language model used for both alignment and trajectory generation."},{"cited_title":"H.; Vora, S.; Liong, V","cited_arxiv_id":null,"evidence_quote":"The dataset and benchmark all open-loop planning metrics are computed on."}],"review_version":1}