Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DriveQA: Passing the Driving Knowledge Test

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DriveQA builds a 474K-question driving-knowledge benchmark and shows that current LLMs and multimodal LLMs pass basic traffic rules but systematically fail numerical limits, right-of-way ordering, sign variants, and spatial layouts—while tr

desk verdict DriveQA is a genuinely useful driving-knowledge benchmark with a strong synthetic visual component, but the text-QA validation process and fine-tuning details need more documentation before the claims fully land. read the letter →

arxiv 2508.21824 v1 pith:X6VARXWS submitted 2025-08-29 cs.CV

Maolin Wei , Wanzhou Liu , Eshed Ohn-Bar This is my paper
classification cs.CV
keywords drivingknowledgetestmultimodallargelanguagemodelstrafficsignrecognitionright-of-wayreasoningvision-languagequestionansweringsim-to-realtransferrule-basedfine-tuningbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether LLMs and multimodal LLMs could pass the written driving-knowledge test humans must pass to get a license. To answer it, it builds DriveQA, a US-centric benchmark of 26K text questions generated from 51 state driver handbooks and 448K visual QA pairs from a driving simulator plus real street images, covering 220 traffic signs and 19 question categories. Evaluation shows that current models handle basic rules but systematically fail numerical limits, complex right-of-way ordering, sign variations, and spatial layouts, with the strongest model dropping from 91.96% accuracy to 78.91% on a harder distractor set. Fine-tuning on DriveQA raises accuracy across categories, and pretraining on it improves trajectory prediction on nuScenes and action reasoning on BDD-OIA, evidence that rule knowledge learned from text and simulation transfers to real-world driving tasks.

What carries the argument

The machinery is the paired dataset: DriveQA-T, 26K text QA pairs generated by prompting a strong LLM with 51 official state driver handbooks and then manually verified, and DriveQA-V, 448K image-QA pairs from a driving simulator with 220 inserted sign models and right-of-way intersection setups, plus 1,303 real street images. Four design choices carry the argument: every question has an explanation, enabling reasoning-quality checks beyond answer accuracy; synthetic generation records ground-truth metadata such as camera perspective, time of day, weather, distance, and sign type, making controlled variations possible; hard distractors are constructed from same-category signs or numeric valu

What would settle it

Draw a stratified random sample of about 500 DriveQA-T questions, strip the multiple-choice options, and have a panel of licensed driving instructors answer them from the handbooks; if their agreement with the dataset labels is far below the claimed 100% human ceiling, the benchmark is measuring generated-label noise. A second check is to build a small human-written gold set from the same handbooks and see whether model rankings on DriveQA match rankings on that gold set; if not, the observed weaknesses are benchmark artifacts.

Watch

Extended reading notes

Core claim

DriveQA's central discovery is a mismatch: modern LLMs and multimodal LLMs carry substantial general knowledge but not the specialized, precise traffic-rule reasoning required for driving. On the text portion, off-the-shelf models average roughly 44–71% depending on size and prompting, with the strongest closed model at 91.96%; the hardest categories are speed and distance limits, parking, and intersection right-of-way. On the visual portion, off-the-shelf multimodal models often score near or below the 25% random baseline on intersection questions, which the paper attributes largely to visual-encoder failures rather than language reasoning alone. The paper's second claim is that this gap is

Load-bearing premise

The benchmark's ground-truth answers are trustworthy: the generated questions were checked by people, and the checks caught any errors in the traffic-law content.

Editorial extensions

If this is right

  • Fine-tuned open models reach 81–87% on the text benchmark and can exceed the strongest closed model on reasoning-quality metrics, showing that a relatively small, rule-focused dataset closes much of the knowledge gap.
  • Right-of-way and intersection questions remain the hardest category even after fine-tuning, making DriveQA a targeted stress test for spatial rule reasoning rather than a generic QA measure.
  • Pretraining on DriveQA improves downstream nuScenes trajectory L2 error and BDD-OIA action and explanation F1, implying rule knowledge learned from text and synthetic images transfers to real-world driving decision tasks.
  • Retrieval-augmented generation from driver handbooks and chain-of-thought prompting both help but do not fix the weaknesses, implying the bottleneck is applying retrieved rules to visual scenarios, not retrieving them.
  • Humans can be trained to 100% on DriveQA, so the remaining model errors reflect model limitations rather than an unresolvable test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's diagnostic numbers depend on the quality of the manual verification of the generated questions; an independent human-written gold set would be the cleanest check on whether the measured weaknesses are genuine.
  • If the transfer finding holds, the recipe—generate rule QA from handbooks, render controlled visual variations, fine-tune—could be applied to other codified decision domains such as medical triage or legal compliance, where text rules are explicit and annotated images are scarce.
  • The paper's own limitation section notes the benchmark is static and US-centric, so the transfer claims should be read as evidence about structured rule knowledge, not about interactive, time-sensitive behaviors like emergency-vehicle encounters or pedestrian intent.
  • Because the hardest visual errors track visual-encoder limits, improving the visual encoder may move benchmark scores more than further language training; this prediction is testable by swapping encoders while keeping the QA head fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DriveQA, a multimodal driving-knowledge benchmark with a text component (DriveQA-T, 26,143 QA pairs) generated by prompting GPT-4o with 51 U.S. state driver handbooks and a vision component (DriveQA-V, 448K VQA pairs) built from CARLA simulation and Mapillary images. The authors evaluate a range of open-source and closed-source LLMs/MLLMs, reporting that models are strong on basic traffic rules but weak on numerical questions, right-of-way scenarios, rare sign types, and spatial/visual reasoning. They further report that LoRA fine-tuning on DriveQA improves in-domain accuracy and that DriveQA fine-tuning transfers to real-world tasks, with improved L2 error on nuScenes (OpenEMMA) and improved F1 on BDD-OIA. The central claimed contributions are a large benchmark for traffic-rule knowledge and evidence that synthetic/text knowledge can transfer to downstream driving tasks.

Significance. If the DriveQA labels are trustworthy and the transfer results are reproducible, the benchmark fills a real gap: most driving VQA benchmarks focus on perception and planning, while DriveQA systematically targets traffic regulations, right-of-way, and sign knowledge, with controlled variations in perspective, weather, distance, and time of day. The inclusion of explanations for ground-truth answers and the large number of sign types are also strengths. The core diagnostic finding—that off-the-shelf MLLMs are near or below chance on several DriveQA-V intersection categories while GPT-4o is much stronger—is potentially valuable for the community. However, the utility of the entire benchmark hinges on validation of the automatically generated text labels, and the current manuscript does not supply sufficient evidence for that validation, nor for the statistical robustness of the transfer claims.

major comments (4)
  1. [Sec. 3, DriveQA-T curation] The 26,143 ground-truth labels are generated by prompting GPT-4o with driver handbooks and then passing through 'manual quality verification' (Sec. 3). The paper reports no number of verifiers, no annotation protocol, no inter-annotator agreement, no post-cleaning error rate, and no examples of discarded ambiguous cases. The assertion that 'humans, once trained, can obtain 100%' is unsupported without a formal evaluation. Because the questions are drawn from 51 state handbooks with differing traffic laws, the paper must also state whether each question is tagged with its source state or screened to be universal. As written, a model that correctly follows one state's right-of-way rule can be marked wrong under another state's rule, which would bias every accuracy number and corrupt the fine-tuning signal. This issue is load-bearing for the benchmark's validity.
  2. [Sec. 5.2, Tables 2 and 4] The fine-tuning experiments lack essential protocol details: no train/validation split, no number of epochs, no LoRA rank or learning rate, and no discussion of whether the reported DriveQA-T and DriveQA-V accuracies are on held-out samples from the same distribution as the training data. If the evaluation is on the same benchmark used for training, the large gains (e.g., Llama-3.1 from 60.79 to 87.62 in Table 2) may largely reflect memorization rather than improved rule understanding. The authors should report a clear split, and ideally test on held-out sign types, perspectives, or question templates that were not seen during fine-tuning.
  3. [Sec. 5.2, Tables 8 and 9] The claim that 'pretraining on DriveQA enhances downstream driving task performance' is supported by small improvements with no statistical significance or repeated runs. In Table 8, LLaVA-1.6 improves by only 0.07 m on average L2 (2.98 to 2.91), while InternVL-2.5 improves by 0.35 m (3.06 to 2.71); no confidence intervals are given. Table 9 shows a larger action mF1 improvement for InternVL-2.5 (0.2951 to 0.4911), but again no error bars or ablations. The authors should provide multiple seeds, confidence intervals, and controlled comparisons (e.g., fine-tuning on the same amount of generic QA data) to rule out that the gains come from additional training rather than from driving-specific knowledge.
  4. [Sec. 3 and Table 2] GPT-4o is used to generate DriveQA-T and is also the top-scoring evaluated model (91.96% average with CoT+RAG). This creates a circularity risk: question wording may be biased toward GPT-4o's output distribution, inflating its score relative to other models. The paper does not analyze this possibility or validate a sample of questions with an independent generator or human annotators. At minimum, the authors should report a human-verified subset and compare performance on questions generated by another model, or discuss how the manual verification process removes GPT-4o-specific artifacts.
minor comments (6)
  1. [Fig. 1 caption] Typo: 'Ths figure' should be 'This figure'.
  2. [Tables 4 and 5] Model name is printed inconsistently as 'LLA V A-1.5' and 'LLaV A-1.5'; please unify. Also, the sign-type names in Table 5 contain spacing artifacts (e.g., 'Playground Center', 'Trauma Center'); align columns and verify types.
  3. [Sec. 5.2] Terminology is inconsistent: the abstract and conclusion say 'pretraining on DriveQA,' but Tables 8 and 9 are fine-tuning experiments. Clarify whether the models are fine-tuned or pretrained and use consistent wording.
  4. [Table 7] The Mapillary test set is described as 'curated by us from Mapillary (1303 annotated images, including 166 sign types),' but the annotation protocol is not described. Specify who labeled these images and how label consistency was ensured.
  5. [Sec. 3] The paper repeatedly refers to 'additional details in the supplementary' for verification and category definitions, but the supplementary is not part of the arXiv manuscript. Please include the relevant details in the main text or ensure they are publicly available with the submission.
  6. [Sec. 5.2, Table 6] The 'hard subset' results would benefit from a human-performance baseline or a chance-level reference, so the reported accuracy degradation can be interpreted as a measure of question difficulty rather than just model sensitivity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DriveQA is an empirical benchmark paper whose claims are evaluated on held-out and external datasets; no claim reduces to its own inputs.

full rationale

DriveQA is an empirical benchmark paper rather than a derivation, so there is no equation-level chain in which an output is equivalent to an input by construction. The text QA set is grounded in 51 official U.S. state driver handbooks, with GPT-4o used only to draft questions that are then manually verified against the handbooks; the visual QA set is procedurally generated in CARLA with explicit simulator state and manually annotated Mapillary data. The central findings—weaknesses in numerical/right-of-way reasoning, improvements from fine-tuning, and transfer to nuScenes and BDD-OIA—are measured on held-out benchmark splits and on external real-world datasets, not on the same fitted values used to construct the benchmark. No parameter is fitted to a subset and then renamed as a prediction. The paper's self-citations to prior work by the same group appear in related-work context and are not load-bearing for the benchmark's validity. The use of GPT-4o both to generate questions and as an evaluated model is a potential bias, and the manuscript gives little detail on the manual verification protocol or the assertion that humans can obtain 100%; however, these are data-quality and validity concerns, not circular reasoning. The paper is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical benchmark and transfer study, so it introduces no derived quantities or newly postulated physical entities. Its claims rest on dataset-construction assumptions: that LLM-generated, manually verified questions are valid ground truth, that synthetic images represent real-world sign difficulty, and that the fine-tuning results are measured on held-out data.

assumptions (4)
  • domain assumption GPT-4o-generated questions from state driver handbooks, after manual verification, are accurate representations of driving knowledge test questions.
    Sec. 3 describes automatic generation with GPT-4o followed by manual quality verification; the benchmark's validity as ground truth depends on this.
  • domain assumption Manual verifiers are reliable and consistent across rounds.
    Section 3 mentions quality checks in rounds where ambiguous/inconsistent cases were discarded, but inter-annotator agreement is not reported.
  • domain assumption Synthetic CARLA images with inserted 3D sign assets capture task-relevant variation for real-world sign recognition.
    DriveQA-V images are procedural; sim-to-real transfer is partially validated by Mapillary, but the synthetic distribution is the primary training/eval domain.
  • domain assumption Fine-tuning evaluation on DriveQA uses a held-out split of the dataset.
    Necessary to interpret the large fine-tuning gains in Tables 2 and 4 as generalization rather than memorization; the paper never states the split.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DriveQA: Passing the Driving Knowledge Test." pith.science (2026). https://pith.science/paper/X6VARXWS

@misc{pith2026250821824,
  author       = {Pith},
  title        = {Pith review of: DriveQA: Passing the Driving Knowledge Test},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X6VARXWS}},
  note         = {Machine review of arXiv:2508.21824}
}
read the original abstract

If a Large Language Model (LLM) were to take a driving knowledge test today, would it pass? Beyond standard spatial and visual question-answering (QA) tasks on current autonomous driving benchmarks, driving knowledge tests require a complete understanding of all traffic rules, signage, and right-of-way principles. To pass this test, human drivers must discern various edge cases that rarely appear in real-world datasets. In this work, we present DriveQA, an extensive open-source text and vision-based benchmark that exhaustively covers traffic regulations and scenarios. Through our experiments using DriveQA, we show that (1) state-of-the-art LLMs and Multimodal LLMs (MLLMs) perform well on basic traffic rules but exhibit significant weaknesses in numerical reasoning and complex right-of-way scenarios, traffic sign variations, and spatial layouts, (2) fine-tuning on DriveQA improves accuracy across multiple categories, particularly in regulatory sign recognition and intersection decision-making, (3) controlled variations in DriveQA-V provide insights into model sensitivity to environmental factors such as lighting, perspective, distance, and weather conditions, and (4) pretraining on DriveQA enhances downstream driving task performance, leading to improved results on real-world datasets such as nuScenes and BDD, while also demonstrating that models can internalize text and synthetic traffic knowledge to generalize effectively across downstream QA tasks.

Figures

Figures reproduced from arXiv: 2508.21824 by the authors.

Figure 1
Figure 1. Can LLMs Pass a Driving Knowledge Test? We in￾troduce a comprehensive multimodal dataset to evaluate the traffic rule-following capabilities of MLLMs. While most question an￾swering (QA) benchmarks in autonomous driving focus on spatial understanding and common planning tasks, our DriveQA dataset assesses broad driving knowledge. The challenging benchmark comprises text-based questions derived from various U.S. stat… view at source ↗
Figure 2
Figure 2. Example Questions and Answers of DriveQA Dataset. We introduce a text and vision-based benchmark for extensively validating driving knowledge with question type categorization, answer explanation, and environmental information ground truth (GT). align with text through extensive pre-training on large-scale image-text pairs. The modality projector aligns encoder outputs with the language model, enabling integration o… view at source ↗
Figure 4
Figure 4. Word Cloud of Questions in DriveQA. The fig￾ure statistically summarizes the language terms in the introduced DriveQA benchmark. in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Scene Priors: Fine-Grained Traffic Scene Reasoning with Benchmarking and Query-Guided Small-Object Focus

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A query-guided sparse residual module (TG-SOF) plus a new distractor-heavy traffic MCQ benchmark lifts a 4B MLLM by 2.1 points on fine-grained local-evidence questions.

Reference graph

Works this paper leans on

99 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://leaderboard.carla.org/, 2022

    Carla autonomous driving leaderboard. https://leaderboard.carla.org/, 2022. 3

    2022

  2. [2]

    In CVPR, 2024

    Eyes wide shut? exploring the visual shortcomings of multi- modal llms. In CVPR, 2024. 2, 3, 6, 8

    2024

  3. [3]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219, 2024. 1, 6

    arXiv 2024

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023. 3

    arXiv 2023

  5. [5]

    Covla: Comprehensive vision-language-action dataset for autonomous driving

    Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, and Issei Yamamoto. Covla: Comprehensive vision-language-action dataset for autonomous driving. arXiv:2408.10845, 2024. 3

    arXiv 2024

  6. [6]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In CVPR, 2020. 2, 3

    2020

  7. [7]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 3

    2020

  8. [8]

    Argoverse: 3d tracking and forecasting with rich maps

    Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jag- jeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019. 3

    2019

Show all 99 references
  1. [9]

    SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties. In CVPR, 2024. 2

    2024

  2. [10]

    X-llm: Bootstrapping ad- vanced large language models by treating multi-modalities as foreign languages

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping ad- vanced large language models by treating multi-modalities as foreign languages. arXiv:2305.04160, 2023. 3

    2023 arXiv

  3. [11]

    Driving with LLMs: Fusing object- level vector modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karn- sund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with LLMs: Fusing object- level vector modality for explainable autonomous driving. arXiv:2310.01957, 2023. 1

    2023 arXiv

  4. [12]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In ICRA,

  5. [13]

    Pix2seq: A language modeling framework for object detection

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Ge- offrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv:2109.10852, 2021. 3

    2021 arXiv

  6. [14]

    Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles. arXiv:2309.10228, 2023. 1

    2023 arXiv

  7. [15]

    Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles. In WACV, 2024. 3

    2024

  8. [16]

    Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE ITS Magazine, 2024. 3

    2024

  9. [17]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In WACV, 2024. 2

    2024

  10. [18]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023. 3

    2023

  11. [19]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. In CVPR, 2018. 2

    2018

  12. [20]

    Parting with misconceptions about learning- based vehicle motion planning

    Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learning- based vehicle motion planning. CoRL, 2023. 3

    2023

  13. [21]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017. 2, 3, 4

    2017

  14. [22]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017. 2

    2017

  15. [23]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  16. [24]

    What ’no entry’ sign? https://x.com/ FSDdreams/status/1781134085471048060, 2024

    FSD Dreams. What ’no entry’ sign? https://x.com/ FSDdreams/status/1781134085471048060, 2024. 2

    2024

  17. [25]

    Fsd does not see or display ’no entry’ signs

    FSD Dreams. Fsd does not see or display ’no entry’ signs. hhttps://x.com/FSDdreams/status/1845900114335793288,

  18. [26]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv:2407.21783, 2024. 1, 6, 8

    2024 arXiv

  19. [27]

    Motsynth: How can synthetic data help pedestrian detection and tracking? In ICCV, 2021

    Matteo Fabbri, Guillem Bras ´o, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljo ˇsa Oˇsep, Simone Calder- ara, Laura Leal-Taix´e, and Rita Cucchiara. Motsynth: How can synthetic data help pedestrian detection and tracking? In ICCV, 2021. 3

    2021

  20. [28]

    Drive like a human: Rethinking autonomous driving with large language models

    Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In WACV,

  21. [29]

    Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% perfor- mance. Visual Intelligence, 2024. 7, 8 9

    2024

  22. [30]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 3

    2012

  23. [31]

    Keybert: Minimal keyword extrac- tion with bert., 2020

    Maarten Grootendorst. Keybert: Minimal keyword extrac- tion with bert., 2020. 5

    2020

  24. [32]

    https://www.dmv.ca.gov/portal/handbook/california-driver- handbook/, 2025

    California Driver’s Handbook. https://www.dmv.ca.gov/portal/handbook/california-driver- handbook/, 2025. 1

    2025

  25. [33]

    Lora: Low-rank adaptation of large language mod- els

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. arXiv:2106.09685, 2021. 5

    2021 arXiv

  26. [34]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023. 3

    2023

  27. [35]

    Hid- den biases of end-to-end driving models

    Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hid- den biases of end-to-end driving models. ICCV, 2023. 3

    2023

  28. [36]

    Missed do not enter sign

    KCEHO. Missed do not enter sign. https : / / x . com / KCEHO2025/status/1894844753348563046, 2025. 2

    2025

  29. [37]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In NAACL-HLT, 2019. 5

    2019

  30. [38]

    Text to blind motion

    Hee Jae Kim, Kathakoli Sengupta, Masaki Kuribayashi, Hernisa Kacorri, and Eshed Ohn-Bar. Text to blind motion. NeurIPS, 2024. 2

    2024

  31. [39]

    Uncertainty-guided never-ending learning to drive

    Lei Lai, Eshed Ohn-Bar, Sanjay Arora, and John Seon Keun Yi. Uncertainty-guided never-ending learning to drive. CVPR, 2024. 1

    2024

  32. [40]

    ZeroVO: Visual odometry with minimal assumptions

    Lei Lai, Zekai Yin, and Eshed Ohn-Bar. ZeroVO: Visual odometry with minimal assumptions. In CVPR, 2025. 3

    2025

  33. [41]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeurIPS, 2020. 4

    2020

  34. [42]

    Driving everywhere with large language model policy adaptation

    Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In CVPR,

  35. [43]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023. 3

    2023 arXiv

  36. [44]

    Is ego status all you need for open- loop end-to-end autonomous driving? In CVPR, 2024

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving? In CVPR, 2024. 2, 3

    2024

  37. [45]

    KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d

    Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. PAMI, 2022. 3

    2022

  38. [46]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 2004. 5

    2004

  39. [47]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024. 1, 7, 8

    2024

  40. [48]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  41. [49]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 7, 8

    2024

  42. [50]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024. 3

    2024

  43. [51]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In ECCV, 2024. 1, 3

    2024

  44. [52]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In CVPR, 2024. 2

    2024

  45. [53]

    Gpt- driver: Learning to drive with gpt

    Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang. Gpt- driver: Learning to drive with gpt. arXiv:2310.01415, 2023. 1, 3

    2023 arXiv

  46. [54]

    A language agent for autonomous driving

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. In COLM,

  47. [55]

    Lingoqa: Video question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Video question answering for autonomous driving. arXiv:2312.14115, 2023. 2, 3

    2023 arXiv

  48. [56]

    https://driving-tests.org/, 2025

    America’s most trusted driver’s license test prep. https://driving-tests.org/, 2025. 1, 4

    2025

  49. [57]

    The mapillary vistas dataset for semantic understanding of street scenes

    Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 2, 3, 7, 8

    2017

  50. [58]

    Hierarchical clustering

    Frank Nielsen and Frank Nielsen. Hierarchical clustering. Introduction to HPC with MPI for Data Science, 2016. 5

    2016

  51. [59]

    Hello gpt-4o — openai

    OpenAI. Hello gpt-4o — openai. Retrieved in November 14, 2024 from https://openai.com/index/hello-gpt-4o/, 2024. 4, 6, 7, 8

    2024

  52. [60]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. 5

    2002

  53. [61]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shao- han Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824, 2023. 2

    2023 arXiv

  54. [62]

    Detgpt: Detect what you need via reasoning

    Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, et al. Detgpt: Detect what you need via reasoning. arXiv:2305.14167, 2023. 3

    2023 arXiv

  55. [63]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario. arXiv:2305.14836, 2023. 2

    2023 arXiv

  56. [64]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In AAAI, 2024. 3 10

    2024

  57. [65]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2

    2021

  58. [66]

    Sophia Koepke, Zeynep Akata, and Andreas Geiger

    Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata, and Andreas Geiger. Plant: Explainable planning transformers via object-level representations. In CoRL, 2022. 3

    2022

  59. [67]

    CarLLaV A: Vi- sion language models for camera-only closed-loop driving

    Katrin Renz, Long Chen, Ana-Maria Marcu, Jan H¨unermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, and Oleg Sinavski. CarLLaV A: Vi- sion language models for camera-only closed-loop driving. arXiv preprint arXiv:2406.10165, 2024. 3

    2024 arXiv

  60. [68]

    Playing for benchmarks

    Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, 2017. 3

    2017

  61. [69]

    Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning

    Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, and Behzad Dariush. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. In WACV, 2024. 3

    2024

  62. [70]

    Languagempc: Large lan- guage models as decision makers for autonomous driving

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large lan- guage models as decision makers for autonomous driving. arXiv:2310.03026, 2023. 1, 3

    2023 arXiv

  63. [71]

    Reasonnet: End-to-end driving with temporal and global reasoning

    Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In CVPR, 2023. 3

    2023

  64. [72]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In ECCV, 2024. 2, 3

    2024

  65. [73]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv:2305.16355, 2023. 3

    2023 arXiv

  66. [74]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR,

  67. [75]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118, 2024. 1, 6

    2024 arXiv

  68. [76]

    https://www.driverknowledgetests.com/,

    Driver Knowledge Test. https://www.driverknowledgetests.com/,

  69. [77]

    Tokenize the world into object-level knowledge to address long-tail events in autonomous driving

    Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv:2407.00959,

    arXiv

  70. [78]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. CoRL, 2024. 2, 3

    2024

  71. [79]

    Failed to recognize the ”do not enter” sign

    TT. Failed to recognize the ”do not enter” sign. https://x. com/CocJii/status/1896302421862985951, 2025. 2

    2025

  72. [80]

    OmniDrive: A holistic llm-agent framework for au- tonomous driving with 3d perception, reasoning and plan- ning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Al- varez. OmniDrive: A holistic llm-agent framework for au- tonomous driving with 3d perception, reasoning and plan- ning. arXiv:2405.01533, 2024. 2, 3

    2024 arXiv

  73. [81]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. NeurIPS, 2022. 4

    2022

  74. [82]

    Dilu: A knowledge-driven approach to autonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. arXiv:2309.16292, 2023. 3

    2023 arXiv

  75. [83]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lam- bert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Rat- nesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv:2301.00493, 2023. 3

    2023 arXiv

  76. [84]

    Language prompt for autonomous driving

    Dongming Wu, Wencheng Han, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. arXiv:2309.04379, 2023. 1

    2023 arXiv

  77. [85]

    Building generalizable agents with a realistic and rich 3d en- vironment

    Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d en- vironment. arXiv:1801.02209, 2018. 2

    2018 arXiv

  78. [86]

    Are vlms ready for autonomous driving? an empiri- cal study from the reliability, data, and metric perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empiri- cal study from the reliability, data, and metric perspectives. arXiv:2501.04003, 2025. 2

    2025 arXiv

  79. [87]

    Openemma: Open-source multimodal model for end-to-end autonomous driving

    Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. In WACV-LLVM-AD, 2025. 2, 7, 8

    2025

  80. [88]

    Explainable object-induced action decision for autonomous vehicles

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz- Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In CVPR, 2020. 2, 8

    2020

  81. [89]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. RA-L, 2024. 1, 3

    2024

  82. [90]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv:2111.11432, 2021. 3

    2021 arXiv

  83. [91]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022

    2022

  84. [92]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv:2306.02858, 2023. 3

    2023 arXiv

  85. [93]

    SelfD: self-learning large-scale driving policies from the web

    Jimuyang Zhang, Ruizhao Zhu, and Eshed Ohn-Bar. SelfD: self-learning large-scale driving policies from the web. In CVPR, 2022. 3 11

    2022

  86. [94]

    Coaching a teachable student

    Jimuyang Zhang, Zanming Huang, and Eshed Ohn-Bar. Coaching a teachable student. In CVPR, 2023. 3

    2023

  87. [95]

    Feedback-guided autonomous driving

    Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedback-guided autonomous driving. In CVPR,

  88. [96]

    End-to-end urban driving by imitating a reinforcement learning coach

    Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021. 3

    2021

  89. [97]

    Tinyllava: A framework of small-scale large multimodal models

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv:2402.14289,

    arXiv

  90. [98]

    Embodied understanding of driving scenarios

    Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. ECCV, 2024. 1, 3, 4

    2024

  91. [99]

    Learning to drive anywhere

    Ruizhao Zhu, Peng Huang, Eshed Ohn-Bar, and Venkatesh Saligrama. Learning to drive anywhere. CoRL, 2023. 3 12

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.