World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

Cheng Li; Ji Tao; Junyu Han; Mingliang Zhai; Ningrui Yang; Sanyuan Zhao; Xiameng Qin; Yunde Jia; Yuwei Wu; Zengyuan Guo

REVIEW 3 major objections 5 minor 49 references

World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that selecting only the instruction-relevant visual tokens before they reach the language model lets an autonomous-driving system reason about occluded hazards, and reports state-of-the-art results on several benchmarks.

desk verdict Useful token-reduction module for driving MLLMs, but the ORA benchmark does not support the world-knowledge claim as designed. read the letter →

arxiv 2412.06324 v3 pith:P3L6TEO2 submitted 2024-12-09 cs.CV

Mingliang Zhai , Cheng Li , Zengyuan Guo , Ningrui Yang , Xiameng Qin , Sanyuan Zhao , Junyu Han , Ji Tao

show 2 more authors

Yuwei Wu Yunde Jia

This is my paper · ORCID

classification cs.CV

keywords autonomousdrivingmulti-modallargelanguagemodelsworldknowledgeoccludedobjectreasoninginstruction-guidedinteractorvisualtokenselectionobject-levelriskassessmentopen-loopplanning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autonomous-driving language models answer questions well when everything is visible, but they struggle when a bus or a building hides a pedestrian, because the long, redundant streams of tokens from six cameras crowd out the world knowledge already stored in the language model. The paper claims that this can be fixed at the interface: an instruction-guided interactor scores every visual and bird's-eye-view token against the user's question, keeps only the most relevant $k$ tokens, and fuses them back with the original features so detail survives compression. Around this module the authors build a three-stage training pipeline, a refined set of roughly 2 million QA pairs and 1.7 million grounding samples, and a new 200K-pair object-level risk assessment benchmark generated from nuScenes with GPT models. They report state-of-the-art or near-state-of-the-art results on six-view QA, captioning, grounding, VQA, risk assessment, and open-loop planning, including a 10.6% accuracy gain on NuScenes-MQA, a 51.4% CIDEr gain on OmniDrive-NuScenes, and the best collision rate in planning. The pith, stated sympathetically, is that instruction-relevance is the right pressure to put on multi-view video: compressing it that way lets a frozen LLM actually use its world knowledge for driving.

What carries the argument

Instruction-guided interactor: a plug-and-play module that selects and pre-fuses multimodal tokens before they enter the LLM. It uses the instruction token sequence as a query to score all multi-view image tokens and BEV tokens by cosine similarity, keeps only the top $k$ (image $k=90$, BEV $k=300$), and runs two-layer cross-attention between those selected tokens and the full original features, producing a compact instruction-relevant representation for the language model. The three-stage training pipeline (single-view visual-language alignment, multi-view alignment with frozen BEV encoder, then task-specific LoRA fine-tuning) and the curated datasets supply the alignment and supervision; the interactor is the load-bearing mechanism that makes multi-view input tractable and, on the paper's account, lets world knowledge enter the reasoning.

What would settle it

Have human annotators verify a held-out subset of ORA where the risky object is fully occluded in every camera view, and compare the model's accuracy against a no-image text baseline on that subset; if the model does not beat the baseline, the claim that it reasons about perception-limited regions is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in perception-limited driving is not the visual encoder but the token sequence handed to the LLM: when six camera views and BEV features are concatenated directly, the input is too long and too undifferentiated for the model's world knowledge to engage, especially for occluded hazards. Its proposed fix is the instruction-guided interactor, a plug-and-play module that computes cosine similarity between instruction tokens and every visual token, selects the top $k$ most similar tokens ($k=90$ for images, $k=300$ for BEV), and then lets those selected tokens attend to the full feature map through cross-attention so that dropped details are recoverable. The authors argue that this 'pre-fusion' produces highly aggregated visual representations that fit within LLM sequence limits while remaining responsive to the query, and they support the claim with a three-stage training strategy and a large multi-view dataset. On their reported experiments, the complete model improves over prior driving MLLMs on NuScenes-MQA, OmniDrive-NuScenes, NuInstruct, NuScenes-QA, the new ORA benchmark, and open-loop planning, where it reaches the state of the art in collision rate.

Load-bearing premise

The load-bearing assumption is that the GPT-4o-generated risk labels on the ORA benchmark measure a model's ability to reason about occluded objects, even though those labels are built from ground-truth object lists the model never sees at inference.

Editorial extensions

If this is right

With the interactor, a frozen 8-billion-parameter LLM can take six-view video plus BEV as input without exceeding sequence limits, so multi-view reasoning becomes a token-selection problem rather than a scaling problem.
The reported gains on NuScenes-MQA (+10.6% ACC), OmniDrive-NuScenes (+51.4% CIDEr), and NuInstruct (+98.44% BLUE4) indicate the module transfers across QA, captioning, grounding, and reasoning tasks, not just one benchmark.
The ORA dataset, with its four object-level risk categories (view obstruction, collision possibility, traffic-rule violation, potential risk) and six question types, gives the driving-MLLM community a way to train and measure perception-limited reasoning separately from normal perception.
In open-loop planning the method reaches the best collision rate even without ego status and high-level commands, which the authors attribute to better scene understanding rather than to trajectory bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The interactor's recipe (score tokens by instruction relevance, keep top-$k$, re-attend to the full feature set) is modality-agnostic, so the same module could consume lidar, radar, or map tokens; the paper only demonstrates camera and BEV inputs, leaving that extension as a testable prediction.
Because ORA labels are generated by GPT-4o from ground-truth object lists that include occluded objects, part of the reported risk-assessment performance may reflect imitation of GPT's annotation style rather than visual inference; a human-labeled subset would separate the two effects.
If instruction-relevance is the correct compression criterion, the same idea could serve as a queryable memory system for real-time driving, materializing only the scene tokens a current decision needs instead of processing all frames.
The collision-rate gains without ego status raise the possibility that token selection acts as a structural regularizer on the planner, forcing it to rely on scene layout; the paper does not test this hypothesis directly.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a solid engineering paper with a modest but useful module, and a central claim that outruns its evidence. The instruction-guided interactor—top-k selection by similarity to the instruction, then cross-attention with the original features—is a clean, plausible extension of Q-Former-style token reduction to multi-view driving inputs. The ablations show consistent gains on NuScenes-MQA, OmniDrive, and NuInstruct, and the planning results (collision rate 0.18 avg vs 0.30 for the best comparable) are interesting. I also give them credit for retraining SparseBEV without future frames or validation data and for the sky-masking redundancy experiment; that is the kind of honest checking that makes me trust the public-benchmark numbers.

The soft spot is the ORA dataset, and it is load-bearing. The stress-test note is right: GPT-4o was given the full nuScenes object list, including occluded objects, and asked to enumerate risks. The model at inference only sees images. So a high ORA score can come from imitating GPT-4o's answer style and exploiting common risk priors, not from making inferences about occluded regions. The paper's key phrase 'perception-limited reasoning' is exactly what this benchmark does not isolate. That is not a minor metric issue—it is a construct-validity problem for the paper's central thesis.

Some smaller issues: Table 4 compares only against OPT baselines, not current driving MLLMs, so the 10.6% SoTA claim is overstated. No error bars anywhere. No code or dataset release, which makes the ORA evaluation impossible to scrutinize. The writing has typos and some sections are hard to follow (e.g., the arrow in Table 8 suggests mAP should have an up arrow, and Table 10 has odd metric shifts).

Who should read this: anyone working on token-efficient vision-language models for driving, or on the methodology of GPT-generated driving benchmarks. I would bring it to a reading group as a case study in benchmark validity, but I would not cite the ORA benchmark in my own work.

Recommendation: send it to peer review. The module deserves scrutiny, and the authors should be pushed to either validate ORA against human judgments or a visible-object-only baseline, release the data and code, and soften the 'world knowledge' claim until the benchmark actually supports it. As is, it is a conditional accept at best.

Referee Report

3 major / 5 minor

Summary. The paper proposes a multi-modal large language model framework for autonomous driving that centers on an instruction-guided interactor module. The interactor selects the top-k visual tokens most relevant to the language instruction and then applies cross-attention between those selected tokens and the full visual feature set, thereby pre-fusing multi-view images and BEV features before they enter the LLM. The authors also present a three-stage training pipeline (single-view alignment, multi-view alignment, task-specific instruction tuning) and collect a large fine-tuning corpus of roughly 2M QA pairs and 1.7M grounding samples. To evaluate reasoning under perception-limited conditions, they introduce an object-level risk assessment (ORA) dataset of 200K QA pairs generated with GPT-4o and GPT-4o-mini. The reported experiments show improvements over OPT-based baselines on NuScenes-MQA, over OmniDrive on OmniDrive-NuScenes, and strong results on NuInstruct and open-loop planning, including a new state-of-the-art collision rate.

Significance. If the results hold, the instruction-guided interactor is a simple, plug-and-play component that could make multi-view and high-resolution driving inputs tractable for MLLMs without sacrificing reasoning quality, which would be broadly useful. The scale of the collected dataset and the public-benchmark improvements provide a useful engineering contribution. However, the central claim that the method integrates world knowledge to reason under perception-limited conditions rests almost entirely on the ORA benchmark, whose construction includes privileged object information not available at inference. The public-benchmark comparisons also do not yet establish the claimed state of the art because the baseline sets are narrow and backbone choices are not controlled. The paper includes no error bars, and the ORA labels are not human-validated, so the central empirical claims need additional support.

major comments (3)

[Object-level Risks Assessment (ORA), Tables 14-15] The ORA benchmark, which is the only dedicated evaluation of the paper's central claim of world-knowledge-enhanced reasoning in perception-limited regions, is constructed with privileged object information that the evaluated model never receives at inference. The generation prompt in Table 14 gives GPT-4o the full nuScenes object list, including objects that may be occluded in the camera views, together with their categories, directions, and distances, and asks for per-object risk enumeration; Table 15 then converts these into QA pairs used as both training and test labels. At test time the model receives only multi-view video and an instruction (Eq. 1, Figure 2), never the object list. Consequently, a model could score well by imitating GPT-4o's answer distribution and by exploiting generic risk priors (e.g., a close car ahead implies collision possibility) without performing any image-based inference about occluded areas. The paper provides no evidence that ORA actually measures occlusion-driven reasoning: there is no human validation of GPT-generated labels, no control condition that ablatesthe object-list input, and no analysis of whether the test questions depend on objects that are invisible in the provided images. Because the ORA results (Table 3) underpin the paper's key contribution, this construct-validity issue is load-bearing and must be addressed, for instance with a human-annotated subset, a comparison of models with and without the privileged object list, and an analysis of how performance varies with the degree of occlusion.
[Tables 4 and 5] The claims of "10.6% improvement over previous SoTA" on NuScenes-MQA and "51.4% improvement in CIDEr" on OmniDrive-NuScenes are not supported by the baseline choices. Table 4 compares only against OPT-1.3B and OPT-6.7B, which are small, dated language models and are not representative of current driving MLLMs; none of the recent SoTA driving models (e.g., OmniDrive variants, BEV-InMLLM, DriveLM, DriveMLLM) are included. Table 5 compares only against OmniDrive, and the comparison does not control for backbone differences: the paper uses LLaMA3-8B and EVA-02, while OmniDrive uses different settings. The reported gains could be largely attributable to the choice of base LLM and visual encoder rather than to the proposed interactor. The authors should either add strong, code-available baselines with matched backbones and training data or substantially qualify the SoTA statements.
[Tables 3, 8, 10] No error bars, confidence intervals, or statistical significance tests are reported anywhere in the paper, even though several headline improvements are small. For example, Table 8 shows ACC improving from 74.1 to 74.4 on NuScenes-MQA and CIDEr from 101.2 to 103.9 on OmniDrive-NuScenes, and Table 3 contains differences under 2 points across many metrics. LLM fine-tuning is stochastic, and these differences may fall within run-to-run variability. The authors should report the mean and standard deviation over at least three independent training runs for the main tables, or otherwise demonstrate that the observed differences are statistically reliable.

minor comments (5)

[General] There are several typographical errors: "Matrics" should be "Metrics", "Comparsions" should be "Comparisons", "open-loos" should be "open-loop", "commends" should be "commands", and in Figure 2c "ego cat" should likely be "ego car".
[Eq. 2 and Implementation] The similarity operator ℕ in Eq. 2 is not defined at its first use; although Section "Implementation" states that cosine similarity is used, this should be stated in the method section. The top-k operator K is also not formally defined; please provide precise definitions for both.
[Table 3] The baseline "Bunny-Llama3" is not described anywhere in the text. Please provide its configuration, training setup, or a citation so that readers can understand what it represents.
[Abstract and Section 'Dataset Construction'] The abstract states the model uses "2 million natural language QA pairs, 1.7 million grounding task data", but Table 1 reports 1.7M grounding and 100K caption pre-training samples, while Table 2 lists about 2.3M fine-tuning pairs. The relationship between these numbers should be clarified and made consistent.
[Discussion and Conclusion] The Discussion section honestly notes that closed-loop evaluation and 3D grounding are not performed, yet the Conclusion states that "Extensive experiments across tasks such as VQA, open-loop driving, and detection demonstrate the effectiveness". Given that open-loop L2 error is comparable but not superior to baselines, the conclusion should be more measured.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains rest on held-out public benchmarks and standard supervised fine-tuning, not on equation-level self-reference or fitted parameters renamed as predictions.

full rationale

This paper does not exhibit circular reasoning in the sense of a claimed derivation reducing to its own inputs. The core architecture (Eqs. 1-3) defines the instruction-guided interactor as top-k visual-token selection followed by cross-attention with the original visual features; this is a constructive mechanism, and the paper does not define the interactor's outputs in terms of the target benchmark scores. The three-stage training procedure (single-view pretraining, multi-view alignment, task-specific fine-tuning) is a standard supervised pipeline: all benchmark results in Tables 4-9 are evaluated on held-out test splits of public datasets such as NuScenes-MQA, OmniDrive-NuScenes, NuInstruct, and NuScenes-QA, and compared against external baselines, so the reported improvements are not forced by construction. The self-built ORA dataset is generated by GPT-4o and GPT-4o-mini from object lists and then split into train and test; the model is fine-tuned on the train split and evaluated on the test split, which is normal supervised evaluation rather than fitting a parameter and renaming it a prediction. The top-k hyperparameter (k=90) is tuned and reported in Table 10, but it is presented as a parameter analysis, not as a predictive claim. No load-bearing argument rests on a self-citation: references to BLIP-2, SparseBEV, BEV-Planner, OmniDrive, and NuInstruct are external prior work, and the authors do not invoke their own previous results to justify the architecture. The manuscript itself notes limitations, including lack of closed-loop testing on CARLA or NuPlan and lack of 3D grounding validation; these concern generalization and scope, not circularity. A construct-validity question could be raised about whether ORA scores measure perception-limited reasoning when GPT-4o was given privileged object lists, but that is a measurement-validity concern, not a circular derivation of the paper's stated results.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claims rest on the validity of the GPT-generated risk benchmark, the availability of ground-truth object lists in nuScenes, and the assumption that LLaMA3's pretrained knowledge suffices for driving risk reasoning. The top-k hyperparameters are tuned on the reported datasets.

free parameters (3)

top-k image token count = 90
Chosen by parameter analysis on OmniDrive-NuScenes, NuScenes-QA, and NuInstruct (Table 10); k=90 gives the best BLUE4/ACC trade-off.
top-k BEV token count = 300
Set to 300 without a reported sensitivity study; affects input length and performance.
cross-attention layers in interactor = 2
Architecture choice; no ablation on this depth is reported.

assumptions (3)

domain assumption nuScenes ground-truth object lists, including occluded objects, are available for ORA generation.
The ORA construction feeds GPT-4o a list of objects with category, direction, and distance; this list likely comes from nuScenes labels, which include occluded objects. The paper does not state how occlusion labels are used.
ad hoc to paper GPT-4o and GPT-4o-mini produce reliable risk annotations without human verification.
The two-step generation in 'Object-level Risks Assessment' relies entirely on GPT models; no human quality control or inter-annotator agreement is reported.
domain assumption The pretrained LLaMA3-8B contains sufficient world knowledge for driving risk reasoning.
The method claims world knowledge enhancement but only fine-tunes the LLM; no external knowledge base or explicit world-knowledge injection is used.

invented entities (1)

Object-level Risk Assessment (ORA) dataset
purpose: Benchmark for evaluating reasoning in perception-limited regions
Constructed by GPT-4o/mini from nuScenes objects; not released and labels not human-verified, so no external benchmark can independently confirm it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving." pith.science (2026). https://pith.science/paper/P3L6TEO2

@misc{pith2026241206324,
  author       = {Pith},
  title        = {Pith review of: World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3L6TEO2}},
  note         = {Machine review of arXiv:2412.06324}
}

read the original abstract

The Multi-modal Large Language Models (MLLMs) with extensive world knowledge have revitalized autonomous driving, particularly in reasoning tasks within perceivable regions. However, when faced with perception-limited areas (dynamic or static occlusion regions), MLLMs struggle to effectively integrate perception ability with world knowledge for reasoning. These perception-limited regions can conceal crucial safety information, especially for vulnerable road users. In this paper, we propose a framework, which aims to improve autonomous driving performance under perceptionlimited conditions by enhancing the integration of perception capabilities and world knowledge. Specifically, we propose a plug-and-play instruction-guided interaction module that bridges modality gaps and significantly reduces the input sequence length, allowing it to adapt effectively to multi-view video inputs. Furthermore, to better integrate world knowledge with driving-related tasks, we have collected and refined a large-scale multi-modal dataset that includes 2 million natural language QA pairs, 1.7 million grounding task data. To evaluate the model's utilization of world knowledge, we introduce an object-level risk assessment dataset comprising 200K QA pairs, where the questions necessitate multi-step reasoning leveraging world knowledge for resolution. Extensive experiments validate the effectiveness of our proposed method.

Figures

Figures reproduced from arXiv: 2412.06324 by the authors.

**Figure 1.** Examples of dynamic and static environments risks. (a) The bus in motion severely obstructs the line of sight, resulting in the black sedan being hidden, which significantly increases the risk of a traffic accident in an unprotected scenario. (b) Buildings in static scenes can also become occluding objects. For example, in a construction site scene, the construction gate blocks the workers behind the gate. concea… view at source ↗

**Figure 2.** Overall of our architecture. (a) Task-specific instructions. (b) A multi-modal large language model equipped with an interactor, which can select important tokens and perform pre-fusion of these tokens before inputting multi-view and multimodal information into the LLM. (c) Decoding results and visualization of tokens output by LLM. achieve comprehensive perception of the current scene but also to integrate world k… view at source ↗

**Figure 3.** Interactor Module. N represents similarity operator. K represents the top-k operator. task. Moreover, as the number of input images or modalities increases, the excessively long input sequences can lead to computational demands that are unacceptably high. This issue is particularly prominent in autonomous driving systems, which require inputs from multiple perspectives and modalities. To address this issue, we pr… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Training pipeline of our method. SV means single [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Qualitative results with planning. The red line represents the ground truth path, while the blue line indicates the path predicted by our method. These results were obtained without ego status. method also achieves SoTA performance in collision rate and demonstrates c…

**Figure 6.** Figure 6: Bad Case Visualization [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 10 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[4]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

2022
[5]

Awadalla, A.; Gao, I.; Gardner, J.; Hessel, J.; Hanafy, Y.; Zhu, W.; Marathe, K.; Bitton, Y.; Gadre, S.; Sagawa, S.; et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

arXiv 2023
[6]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023
[7]

Bai, Y.; Wu, D.; Liu, Y.; Jia, F.; Mao, W.; Zhang, Z.; Zhao, Y.; Shen, J.; Wei, X.; Wang, T.; et al. 2024. Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving? arXiv preprint arXiv:2405.18361

arXiv 2024
[8]

Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Chen, X.; Choromanski, K.; Ding, T.; Driess, D.; Dubey, A.; Finn, C.; et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818

arXiv 2023

Show all 49 references

[9]

H.; Vora, S.; Liong, V

Caesar, H.; Bankiti, V.; Lang, A. H.; Vora, S.; Liong, V. E.; Xu, Q.; Krishnan, A.; Pan, Y.; Baldan, G.; and Beijbom, O. 2019. nuScenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027

2019 arXiv
[10]

J.; Birch, D.; Maund, D.; and Shotton, J

Chen, L.; Sinavski, O.; Hünermann, J.; Karnsund, A.; Willmott, A. J.; Birch, D.; Maund, D.; and Shotton, J. 2024 a . Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA)

2024
[11]

Chen, X.; Wang, X.; Changpinyo, S.; Piergiovanni, A.; Padlewski, P.; Salz, D.; Goodman, S.; Grycner, A.; Mustafa, B.; Beyer, L.; et al. 2022. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794

2022 arXiv
[12]

Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

2024
[13]

Cohen, G. H. 1997. ALIGN: a program to superimpose protein coordinates, accounting for insertions and deletions. Journal of applied crystallography, 30(6): 1160--1161

1997
[14]

Cui, Y.; Huang, S.; Zhong, J.; Liu, Z.; Wang, Y.; Sun, C.; Li, B.; Wang, X.; and Khajepour, A. 2023. Drivellm: Charting the path toward full autonomous driving with large language models. IEEE Transactions on Intelligent Vehicles

2023
[15]

K.; Srivastava, S.; Jatavallabhula, K

Dewangan, V.; Choudhary, T.; Chandhok, S.; Priyadarshan, S.; Jain, A.; Singh, A. K.; Srivastava, S.; Jatavallabhula, K. M.; and Krishna, K. M. 2023. Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving. arXiv preprint arXiv:2310.02251

2023 arXiv
[16]

Ding, X.; Han, J.; Xu, H.; Liang, X.; Zhang, W.; and Li, X. 2024. Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13668--13677

2024
[17]

Ding, X.; Han, J.; Xu, H.; Zhang, W.; and Li, X. 2023. Hilm-d: Towards high-resolution understanding in multimodal large language models for autonomous driving. arXiv preprint arXiv:2309.05186

2023 arXiv
[18]

Dosovitskiy, A.; Ros, G.; Codevilla, F.; Lopez, A.; and Koltun, V. 2017. CARLA : An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning, 1--16

2017
[19]

S.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al

Driess, D.; Xia, F.; Sajjadi, M. S.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378

2023 arXiv
[20]

Fang, Y.; Sun, Q.; Wang, X.; Huang, T.; Wang, X.; and Cao, Y. 2023. EVA-02: A Visual Representation for Neon Genesis. arXiv preprint arXiv:2303.11331

2023 arXiv
[21]

Fu, D.; Li, X.; Wen, L.; Dou, M.; Cai, P.; Shi, B.; and Qiao, Y. 2024. Drive like a human: Rethinking autonomous driving with large language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 910--919

2024
[22]

GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Rojas, D.; Feng, G.; Zhao, H.; Lai, H.; Yu, H.; Wang, H.; Sun, J.; Zhang, J.; Cheng, J.; Gui, J.; Tang, J.; Zhang, J.; Li, J.; Zhao, L.; Wu, L.; Zhong, L.; Liu, M.; Huang, M.; Zhang, P.; Zheng, Q.; Lu, R.; Duan, S.; Zhan...

2024 arXiv
[23]

Caesar, K

H. Caesar, K. T. e. a., J. Kabzan. 2021. NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop

2021
[24]

Inoue, Y.; Yada, Y.; Tanahashi, K.; and Yamaguchi, Y. 2024. Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 930--938

2024
[25]

Jiang, B.; Chen, S.; Xu, Q.; Liao, B.; Chen, J.; Zhou, H.; Zhang, Q.; Liu, W.; Huang, C.; and Wang, X. 2023. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 8340--8350

2023
[26]

Jiang, D.; He, X.; Zeng, H.; Wei, C.; Ku, M.; Liu, Q.; and Chen, W. 2024. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483

2024 arXiv
[27]

Lauren c on, H.; Tronchon, L.; Cord, M.; and Sanh, V. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246

2024 arXiv
[28]

Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

2023
[29]

Li, Z.; Yu, Z.; Lan, S.; Li, J.; Kautz, J.; Lu, T.; and Alvarez, J. M. 2024. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14864--14873

2024
[30]

Lin, J.; Yin, H.; Ping, W.; Molchanov, P.; Shoeybi, M.; and Han, S. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26689--26699

2024
[31]

Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024 a . LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

2024
[32]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

2024
[33]

Liu, H.; Teng, Y.; Lu, T.; Wang, H.; and Wang, L. 2023 a . Sparsebev: High-performance sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 18580--18590

2023
[34]

Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Li, C.; Yang, J.; Su, H.; Zhu, J.; et al. 2023 b . Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499

2023 arXiv
[35]

Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

2017 arXiv
[36]

Ma, Y.; Cao, Y.; Sun, J.; Pavone, M.; and Xiao, C. 2023. Dolphins: Multimodal Language Model for Driving. arXiv prepreint arXiv:2312.00438

2023 arXiv
[37]

Mei, J.; Ma, Y.; Yang, X.; Wen, L.; Cai, X.; Li, X.; Fu, D.; Zhang, B.; Cai, P.; Dou, M.; et al. 2024. Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving. arXiv preprint arXiv:2405.15324

2024 arXiv
[38]

Qian, T.; Chen, J.; Zhuo, L.; Jiao, Y.; and Jiang, Y.-G. 2024. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 4542--4550

2024
[39]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

2021
[40]

Sima, C.; Renz, K.; Chitta, K.; Chen, L.; Zhang, H.; Xie, C.; Luo, P.; Geiger, A.; and Li, H. 2023. DriveLM: Driving with Graph Visual Question Answering. arXiv preprint arXiv:2312.14150

2023 arXiv
[41]

Sun, Q.; Cui, Y.; Zhang, X.; Zhang, F.; Yu, Q.; Wang, Y.; Rao, Y.; Liu, J.; Huang, T.; and Wang, X. 2024. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14398--14409

2024
[42]

Tian, R.; Li, B.; Weng, X.; Chen, Y.; Schmerling, E.; Wang, Y.; Ivanovic, B.; and Pavone, M. 2024 a . Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving. arXiv preprint arXiv:2407.00959

2024 arXiv
[43]

Tian, X.; Gu, J.; Li, B.; Liu, Y.; Zhao, Z.; Wang, Y.; Zhan, K.; Jia, P.; Lang, X.; and Zhao, H. 2024 b . DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models. arXiv preprint arXiv:2402.12289

2024 arXiv
[44]

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[45]

Wang, S.; Yu, Z.; Jiang, X.; Lan, S.; Shi, M.; Chang, N.; Kautz, J.; Li, Y.; and Alvarez, J. M. 2024 a . OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning. arXiv preprint arXiv:2405.01533

2024 arXiv
[46]

Wang, T.; Xie, E.; Chu, R.; Li, Z.; and Luo, P. 2024 b . Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint arXiv:2403.16996

2024 arXiv
[47]

Wang, W.; Xie, J.; Hu, C.; Zou, H.; Fan, J.; Tong, W.; Wen, Y.; Wu, S.; Deng, H.; Li, Z.; et al. 2023. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245

2023
[48]

Wen, L.; Fu, D.; Li, X.; Cai, X.; Ma, T.; Cai, P.; Dou, M.; Shi, B.; He, L.; and Qiao, Y. 2023. Dilu: A knowledge-driven approach to autonomous driving with large language models. arXiv preprint arXiv:2309.16292

2023 arXiv
[49]

Yu, J.; Wang, X.; Tu, S.; Cao, S.; Zhang-Li, D.; Lv, X.; Peng, H.; Yao, Z.; Zhang, X.; Li, H.; et al. 2024. KoLA: Carefully Benchmarking World Knowledge of Large Language Models. In The Twelfth International Conference on Learning Representations

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[4] [4]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

2022

[5] [5]

Awadalla, A.; Gao, I.; Gardner, J.; Hessel, J.; Hanafy, Y.; Zhu, W.; Marathe, K.; Bitton, Y.; Gadre, S.; Sagawa, S.; et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

arXiv 2023

[6] [6]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023

[7] [7]

Bai, Y.; Wu, D.; Liu, Y.; Jia, F.; Mao, W.; Zhang, Z.; Zhao, Y.; Shen, J.; Wei, X.; Wang, T.; et al. 2024. Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving? arXiv preprint arXiv:2405.18361

arXiv 2024

[8] [8]

Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Chen, X.; Choromanski, K.; Ding, T.; Driess, D.; Dubey, A.; Finn, C.; et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818

arXiv 2023