Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A compact vision-language model running on a vehicle's onboard computer translates spoken commands and camera images into controller parameters, cutting driver takeover rates by up to 76.9%.

desk verdict Real-vehicle VLM-to-controller system that is credible and clever, but its personalization claim is under-evidenced by the reported analysis. read the letter →

arxiv 2411.11913 v1 pith:WUKIWMCT submitted 2024-11-17 cs.AI cs.RO

Can Cui , Zichong Yang , Yupeng Zhou , Juntong Peng , Sung-Yeon Park , Cong Zhang , Yunsheng Ma , Xu Cao
show 6 more authors
Wenqian Ye Yiheng Feng Jitesh Panchal Lingxi Li Yaobin Chen Ziran Wang
This is my paper · ORCID
classification cs.AIcs.RO
keywords personalizedautonomousdrivingvision-languagemodelsmotioncontrolretrieval-augmentedgenerationmodelpredictivePIDhuman-vehicleinteractionon-boarddeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Personalized driving means an autonomous vehicle adjusts its behavior to match a particular driver's preferences. This paper claims that a 9-billion-parameter vision-language model running on the vehicle's onboard computer can do that by turning a spoken command plus the current camera view into a small matrix of controller parameters: PID gains for speed control and MPC weights for steering. A retrieval-augmented memory stores each user's past instructions, scene descriptions, actions, and feedback, and injects the most relevant history into the model's prompt, so the vehicle keeps adapting without retraining. Real-vehicle experiments with seven participants report lower takeover rates than a fixed-parameter baseline across all instruction types, with a 76.9% reduction for indirect commands, and the system runs at about 1.6 seconds latency and under 16 GB of GPU memory, which the authors say makes on-board deployment commercially feasible.

What carries the argument

The action matrix $\mathbf{P} = \begin{bmatrix} K_p & K_i & K_d \\ W_l & W_h & W_s \end{bmatrix}$ is the central object: it is the VLM's only output and the direct input to the vehicle's longitudinal PID and lateral MPC controllers. The approach follows a prior idea of prompting a language model to produce controller parameter matrices rather than raw commands or trajectories. The second load-bearing mechanism is the retrieval-augmented memory module: a per-user vector database that stores tuples of (instruction, scene description, policy, feedback) and retrieves the top-k most similar past interactions by instruction similarity, feeding that history into the prompt so the model can honor stable preferences without retraining.

What would settle it

Re-run the human-in-the-loop study with a new group of participants on the same track, keeping the Level-3 (indirect) instructions; if the takeover rate does not drop from the baseline's 36.11% toward the claimed 8.33%, the personalization claim fails. Independently, simulate the closed-loop PID and MPC controllers across a grid of the action matrices the model generates; any meaningful fraction of unstable or overly aggressive outputs would show that trusting the VLM without a safety envelope is unsafe.

Watch

Extended reading notes

Core claim

The central claim is that the gap between high-level human instructions and low-level vehicle control can be closed by a compact VLM that outputs a two-row action matrix: the first row holds PID gains ($K_p$, $K_i$, $K_d$) for longitudinal acceleration, and the second holds MPC weights ($W_l$, $W_h$, $W_s$) for lateral steering. The model is fine-tuned on about 1,200 human-annotated image-instruction pairs labeled with such matrices, then compressed with low-bit weight quantization to run on a vehicle GPU. Each user has a per-user memory database built from retrieved copies of past interactions; when a new command arrives, the system retrieves the most similar past situations and appends them to the prompt, which the authors argue is what enables continuous personalization. The paper validates this in a real drive-by-wire vehicle with seven participants, reporting that takeover rates fall from 19.44% to 5.56% for direct commands and from 36.11% to 8.33% for indirect commands, while an ablation shows the memory module accounts for most of the improvement. To the best of the authors' knowledge, this is the first end-to-end VLM-based motion control system tested in real-world autonomous vehicles.

Load-bearing premise

The load-bearing premise is that a person's driving preferences can be captured by expert-defined parameter ranges for three coarse styles (aggressive, moderate, conservative), and that the fine-tuned vision-language model will output stable and safe controller parameters without any explicit safety envelope.

Editorial extensions

If this is right

  • If the central claim holds, personalized motion control no longer requires a cloud-hosted large model or per-user fine-tuning; a moderately sized on-board model with a personal memory store is enough to adapt driving style.
  • Because the memory module is per-user and grows with feedback, the system's personalization should improve over repeated trips, and new users should see immediate benefit from the pretrained knowledge plus a fresh, empty memory.
  • The decoupled PID-plus-MPC design means the approach can be grafted onto existing modular autonomous-driving stacks without replacing perception or planning.
  • The reported ablation, where removing the memory module raises the average takeover rate from 6.67% to 24.44%, implies that the memory, not just the VLM's reasoning, is the main driver of the takeover reduction.
  • The reported 1.6-second latency and under-16-GB memory footprint set a concrete operating budget for on-board multimodal reasoning that other vehicle control systems could be held to.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's labeling scheme forces preferences onto a coarse aggressive/moderate/conservative grid; a natural extension is to regress controller parameters continuously from scene embeddings, which could capture preferences the grid cannot express.
  • No safety envelope is placed on the VLM's parameter outputs; adding a runtime safety filter or clamping the generated gains and weights would make the approach more suitable for public roads.
  • Retrieval is keyed on instruction text only, so scenes that look different but carry the same command may retrieve the wrong memory; combining visual and textual embeddings for retrieval is a plausible improvement.
  • Training data drawn from an open driving dataset may limit how well the model transfers to unfamiliar road geometry or weather; a domain-shift evaluation would test whether the parameter choices remain sensible outside the training distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes an on-board Vision-Language Model (VLM) system for personalized autonomous vehicle motion control. The system fine-tunes a Qwen-VL-based 9B model with LoRA, quantizes to INT4 via AWQ, and augments it with a RAG-based per-user memory module. The VLM takes a camera image, a verbal instruction, a system message, and retrieved history as input, and outputs PID and MPC parameter matrices that are executed by the vehicle's drive-by-wire controllers. The authors validate the system in real-world experiments with seven participants, reporting improved driving scores, reduced takeover rates by up to 76.9%, higher survey ratings than a fixed-parameter baseline, and an ablation showing the memory module reduces takeover rates. The paper claims this is the first end-to-end VLM-based motion control system deployed in a real autonomous vehicle.

Significance. If the claims are substantiated, this is a valuable engineering contribution: it demonstrates that a lightweight, quantized VLM can run on-board with 1.6-2.0 s latency and provide sensible low-level controller parameters in a real vehicle. The real-world deployment, the detailed latency and GPU-memory numbers, and the memory-module ablation are concrete strengths. However, the central "personalized" claim is not yet established by the reported evidence: all performance results are aggregated over participants, with no demonstration that the system's outputs or takeover improvements vary as a function of the individual user. The work is therefore of interest to the autonomous-driving and human-AI-interaction communities, but the evidence as presented does not fully support the personalization narrative.

major comments (5)
  1. [Sec. 4.3, Fig. 4, Sec. 3.4] The paper's central claim is personalization, but no analysis shows that the system behaves differently for different users given the same instruction and scene. The RAG memory is per-user, yet retrieval is keyed only by the instruction text I, and the reported takeover rates and driving scores are aggregated over all seven participants. To support the personalization claim, the authors should show, for example, that the same instruction-image pair produces different output parameter matrices (or different takeover rates) for different participants, and/or fit a mixed-effects model with participant as a random effect. Without such evidence, the 76.9% takeover reduction could be explained by generic improvement in instruction- and scene-following rather than by adaptation to individual preferences.
  2. [Sec. 4.3, Fig. 4] Takeover rates are reported as percentages without denominators, confidence intervals, or a statistical test. With seven participants and roughly five instructions per scenario per system, the Level-1 comparison (19.44% vs. 5.56%) corresponds to small counts (e.g., 7 vs. 2 takeovers out of 36 trips); such a difference is well within sampling noise under a paired or binomial model. The paper should report raw counts and per-participant breakdowns, and use an appropriate paired test (e.g., McNemar's test) to support the takeover-reduction claim. The Wilcoxon p-values in Sec. 4.4 apply to survey ratings, not to takeover rates, so they do not address this gap.
  3. [Sec. 4.2, Eqs. (6) and (7)] The driving score S in Eq. (6) uses weights that are "empirically tuned for each driving scenario." This is a potential confound when comparing the proposed system against the baseline and GPT-4o, because the weights may have been selected to favor the authors' system. The paper should report the actual weight values, and/or present the per-metric scores in Table 1 instead of only the weighted sum. In addition, the Command Alignment score in Eq. (7) is computed using parameter ranges (Kp,lower, Kp,upper) determined by the same three expert-defined styles used to label the VLM's training set; the score is therefore partly a measure of how consistently the VLM reproduces its own training labels, not an independent measure of preference matching.
  4. [Sec. 3.5] The text states that "lateral control is handled by MPC calculating the longitudinal acceleration α, while longitudinal control is managed through a PID controller calculating the front steering angle δf." This is reversed: Eq. (4) shows the PID computing longitudinal acceleration from velocity error, and Eq. (5) shows the MPC regulating steering via the front steering angle. This contradiction could mislead readers about which controller acts on which axis and should be corrected.
  5. [Sec. 3.3-3.5] The VLM directly outputs PID and MPC parameter values with no safety envelope, bounds-checking, or stability validation. The paper reports that a human sits in the driver's seat to monitor the vehicle, but that is a testing safeguard, not a system-level safety mechanism. Given the claim of "safe" and "reliable" real-world performance, the authors should specify whether the generated gains are clamped or validated against a safe operating range, or provide evidence that the trained model's outputs remain within a stable controller region across the tested scenarios.
minor comments (6)
  1. [Sec. 3.6] The dataset size is inconsistent: the first sentence of Sec. 3.6 says "a dataset of 10,000 image-instruction pairs," but the Dataset Collection paragraph says "consisting of 1,200 semi-human-annotated image-text pairs." Please clarify the relationship between these two numbers.
  2. [Sec. 3.4 and elsewhere] The model is referred to as both a "9B-parameter VLM" (abstract and Sec. 3.3) and an "8B-parameter VLM" (Sec. 3.4). Please use a single, consistent size.
  3. [Abstract and Sec. 5] The abstract reports a takeover reduction of 76.9%, while the conclusion reports 76.8%. Please reconcile these numbers.
  4. [Sec. 3.4] The memory module stores scene descriptions D and history tuples, but retrieval is described as "the instruction I is used for similarity matching." If scene information is not used in retrieval, the claim of retrieving "similar driving scenarios" is overstated; please clarify the retrieval key and consider including visual/scene features.
  5. [Sec. 4.4, Fig. 5] The Wilcoxon signed-rank tests are performed for four metrics across three scenarios without correction for multiple comparisons; please state the total number of tests and adjust p-values accordingly, or report them only as exploratory.
  6. [Sec. 4.1] The participant demographics say "61.4% male and 28.6% female drivers." These percentages do not sum to 100% for a sample of seven participants; please check the reported proportions.

Circularity Check

1 steps flagged · score 2.0 of 10

Only one auxiliary metric (Command Alignment) is partly built from the same expert labels used to train the model; the central takeover-rate and safety results are external behavioral measurements and are not circular.

  1. fitted input called prediction [Sec. 4.2, Eq. (7), with training-data construction in Sec. 3.6]
    "where Kp,min and Kp,max are the minimum and maximum overall parameter range obtained through experiments, while Kp,lower and Kp,upper are determined by the command intention labeled by human experts. ... we annotate each image with three different instruction types—aggressive, moderate, and conservative—each paired with a corresponding action."

    The Command Alignment metric in Eq. (7) rewards outputs that fall inside parameter ranges whose endpoints are 'determined by the command intention labeled by human experts.' The training set in Sec. 3.6 is built from the same human-expert annotation scheme: each image is paired with aggressive/moderate/conservative instruction types and a corresponding action matrix. Thus a high Command Alignment score partly measures whether the fine-tuned VLM reproduces the label distribution it was trained on, rather than independently confirming that true per-user personalization is occurring.

full rationale

The paper's central claims—76.9% takeover reduction, safety/comfort metrics, and low on-board latency—are obtained from real-vehicle experiments against a fixed baseline and a GPT-4o comparison system. Those measurements do not reduce to the model's training labels or to any self-citation. The only partially circular component is the Command Alignment score in Sec. 4.2, Eq. (7): its expected parameter ranges use endpoints 'determined by the command intention labeled by human experts,' and the fine-tuning data in Sec. 3.6 were created using exactly that three-style expert annotation scheme. Consequently, a high Command Alignment score partly reflects successful reproduction of the training-label distribution rather than independent evidence of personalization. This inflates the paper's 'reasoning capabilities regarding personalization' statement, but it is an auxiliary metric inside the composite Driving Score and is not the basis of the headline takeover-rate result. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling chain was found. The personalization claim is weakly evidenced for other reasons—no per-user output comparison for identical instructions—but that is a missing-evidence concern, not circularity. Overall score 2.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on two externally provided components: a fine-tuned VLM and standard PID and MPC controllers. The main extra free parameters are in the evaluation scoring, not in the control law. No new physical entities are introduced. The most consequential assumptions are the sufficiency of three coarse driving styles and the absence of a safety envelope on generated parameters.

free parameters (3)
  • Driving score weights w_k = not reported, empirically tuned per scenario
    Eq. (6) combines ten metrics into S = sum w_k S_k; the text states weights are 'empirically tuned for each driving scenario,' so the evaluation metric itself is adjusted to the data.
  • Command alignment parameter ranges = not reported (Kp,min, Kp,max, Kp,lower, Kp,upper and equivalents for six parameters)
    Eq. (7) scores command alignment against ranges 'determined by the command intention labeled by human experts' from 'past experiments'; these ranges define what counts as correct personalization.
  • Time-to-collision safety threshold = 1.5 s
    Sec. 4.2 sets a critical TTC threshold to judge safety; chosen by the authors, not derived from an external standard.
assumptions (5)
  • domain assumption A fine-tuned VLM will output controller parameters that remain within safe, stable operating ranges in all driving situations.
    No safety envelope, constraint solver, or formal verification is described around the PID and MPC parameter outputs (Sec. 3.3-3.5); safety is assumed from the learned model.
  • ad hoc to paper Three discrete driving styles (aggressive, moderate, conservative) and expert-defined parameter ranges are sufficient to represent individual user preferences.
    Sec. 3.6 labels each image with three instruction types; Eq. (7) uses expert-set ranges. If real preferences are not captured by these bins, the personalization claim fails.
  • domain assumption The 1,200 or 10,000 NuScenes image-instruction pairs with synthetic action labels generalize to real vehicle scenes, weather, and user commands.
    Sec. 3.6 describes a dataset sourced from NuScenes images with human-provided instructions and action matrix labels; no distribution-shift analysis is provided.
  • domain assumption Instruction-similarity retrieval in the RAG memory retrieves historically relevant scenarios that improve current decisions.
    Sec. 3.4 uses the current instruction for similarity matching to retrieve top-k past entries; the paper does not evaluate retrieval quality directly.
  • domain assumption Human takeover rate on a test track is a valid and sensitive measure of personalized driving quality.
    Sec. 4.3 uses takeover rate as the primary personalization metric; takeovers may also reflect experimenter demand, novelty effects, or scenario difficulty unrelated to personalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation." pith.science (2026). https://pith.science/paper/WUKIWMCT

@misc{pith2026241111913,
  author       = {Pith},
  title        = {Pith review of: On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUKIWMCT}},
  note         = {Machine review of arXiv:2411.11913}
}
read the original abstract

Personalized driving refers to an autonomous vehicle's ability to adapt its driving behavior or control strategies to match individual users' preferences and driving styles while maintaining safety and comfort standards. However, existing works either fail to capture every individual preference precisely or become computationally inefficient as the user base expands. Vision-Language Models (VLMs) offer promising solutions to this front through their natural language understanding and scene reasoning capabilities. In this work, we propose a lightweight yet effective on-board VLM framework that provides low-latency personalized driving performance while maintaining strong reasoning capabilities. Our solution incorporates a Retrieval-Augmented Generation (RAG)-based memory module that enables continuous learning of individual driving preferences through human feedback. Through comprehensive real-world vehicle deployment and experiments, our system has demonstrated the ability to provide safe, comfortable, and personalized driving experiences across various scenarios and significantly reduce takeover rates by up to 76.9%. To the best of our knowledge, this work represents the first end-to-end VLM-based motion control system in real-world autonomous vehicles.

Figures

Figures reproduced from arXiv: 2411.11913 by the authors.

Figure 1
Figure 1. An overview of the proposed framework for personalized autonomous vehicle motion control. The system workflow begins with [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the experiment setup. the perception capabilities. The on-board computer has an Intel i9-9900 9th Gen 3.10/5.0GHz hexa-core 65W proces￾sor with eight cores and 16 threads, 64GB RAM, NVIDIA Quadro RTX-A4000 16GB GPU, and 512GB NVMe solid state drive. Test Track and Participants The field experiments1 aim at validating the real-world performance of our personalized motion control system. We include three t… view at source ↗
Figure 3
Figure 3. Sample vision inputs from different weather scenarios. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Takeover rate comparison between the baseline and our [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Median comparison between VLM-based model and the baseline under various driving scenarios. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Effectiveness of memory modules in takeover rates [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    ViLaD applies masked diffusion, instead of autoregressive decoding, to generate whole driving decision sequences in parallel within a vision-language model, reporting faster and more accurate planning on nuScenes.

  2. Multi-Objective Reinforcement Learning for Adaptable Personalized Autonomous Driving

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A single vision-based reinforcement learning policy can steer an autonomous vehicle in simulation with different driving styles by feeding it a user preference vector, without retraining.

  3. LangCoop: Collaborative Driving with Language

    cs.RO 2025-04 conditional novelty 5.0 of 10

    Natural-language messages under 2 KB replace image sharing between two simulated vehicles, cutting bandwidth by about 96% while achieving driving scores up to 48.8 and route completion up to 90.3% in closed-loop CARLA...

  4. A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving

    cs.RO 2025-06 conditional novelty 4.0 of 10

    A hierarchical real-world testing platform for VLM-integrated autonomous driving is demonstrated on a by-wire vehicle, with a cloud GPT-4 agent making high-level decisions inside an Autoware stack.

  5. Automated Vehicles Should be Connected with Natural Language

    cs.MA 2025-06 conditional novelty 3.0 of 10

    A vision paper recommending natural language as the universal communication medium for connected and automated vehicles.

Reference graph

Works this paper leans on

57 extracted references · 47 canonical work pages · cited by 5 Pith papers

  1. [1]

    autonomousstuff

    AutonomousStuff. “autonomousstuff”, 2023. 4

    work page 2023

  2. [2]

    Self-driving like a human driver in- stead of a robocar: Personalized comfortable driving experi- ence for autonomous vehicles, 2022

    Il Bae, Jaeyoung Moon, Junekyo Jhung, Ho Suk, Taewoo Kim, Hyungbin Park, Jaekwang Cha, Jinhyuk Kim, Dohyun Kim, and Shiho Kim. Self-driving like a human driver in- stead of a robocar: Personalized comfortable driving experi- ence for autonomous vehicles, 2022. 2

    work page 2022

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond, 2023. 2, 5

    work page 2023

  4. [4]

    The explicit solution of model predic- tive control via multiparametric quadratic programming

    Alberto Bemporad, Manfred Morari, Vivek Dua, and Efstra- tios N Pistikopoulos. The explicit solution of model predic- tive control via multiparametric quadratic programming. In Proceedings of the 2000 American control conference. ACC (IEEE Cat. No. 00CH36334), pages 872–876. IEEE, 2000. 5

    work page 2000

  5. [5]

    Per- sonalized driving styles in safety-critical scenarios for au- tonomous vehicles: An approach using driver-in-the-loop simulations

    Ioana-Diana Buzdugan, Silviu Butnariu, Ioana-Alexandra Ros, u, Andrei-Cristian Pridie, and Csaba Antonya. Per- sonalized driving styles in safety-critical scenarios for au- tonomous vehicles: An approach using driver-in-the-loop simulations. Vehicles, 5(3):1149–1166, 2023. 2

    work page 2023

  6. [6]

    A human centric framework for the analysis of automated driving systems based on meaningful human control

    Simeon C Calvert, Dani ¨el D Heikoop, Giulio Mecacci, and Bart Van Arem. A human centric framework for the analysis of automated driving systems based on meaningful human control. TheoreTical issues in ergonomics science , 21(4): 478–506, 2020. 1

    work page 2020

  7. [7]

    Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan H ¨unermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vec- tor modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093–14100. IEEE, 2024. 1

    work page 2024

  8. [8]

    The AI-native open-source embedding database,

    Chroma. The AI-native open-source embedding database,

    work page

Show all 57 references
  1. [9]

    Lmdeploy: A toolkit for com- pressing, deploying, and serving llm

    LMDeploy Contributors. Lmdeploy: A toolkit for com- pressing, deploying, and serving llm. https://github. com/InternLM/lmdeploy, 2023. 5

    2023

  2. [10]

    A survey on multimodal large language models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. A survey on multimodal larg...

    2024

  3. [11]

    Pan- chal, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, and Zi- ran Wang

    Can Cui, Yunsheng Ma, Zichong Yang, Yupeng Zhou, Peiran Liu, Juanwu Lu, Lingxi Li, Yaobin Chen, Jitesh H. Pan- chal, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, and Zi- ran Wang. Large language models for autonomous driving (llm4ad): Concept, benchmark, simulation, and real-veh...

    2024

  4. [12]

    Personalized autonomous driving with large lan- guage models: Field experiments, 2024

    Can Cui, Zichong Yang, Yupeng Zhou, Yunsheng Ma, Juanwu Lu, Lingxi Li, Yaobin Chen, Jitesh Panchal, and Zi- ran Wang. Personalized autonomous driving with large lan- guage models: Field experiments, 2024. 1

    2024

  5. [13]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  6. [14]

    Driver monitoring-based lane-change prediction: A personalized federated learning framework

    Runjia Du, Kyungtae Han, Rohit Gupta, Sikai Chen, Samuel Labi, and Ziran Wang. Driver monitoring-based lane-change prediction: A personalized federated learning framework. In 2023 IEEE Intelligent Vehicles Symposium (IV) , pages 1–7. IEEE, 2023. 1, 2

    2023

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 5

    2021

  8. [16]

    Personalized tra- jectory planning and control of lane-change maneuvers for autonomous driving

    Chao Huang, Hailong Huang, Peng Hang, Hongbo Gao, Jingda Wu, Zhiyu Huang, and Chen Lv. Personalized tra- jectory planning and control of lane-change maneuvers for autonomous driving. IEEE Transactions on Vehicular Tech- nology, 70(6):5511–5523, 2021. 2

    2021

  9. [17]

    An open approach to autonomous vehicles

    Shinpei Kato, Eijiro Takeuchi, Yoshio Ishiguro, Yoshiki Ni- nomiya, Kazuya Takeda, and Tsuyoshi Hamada. An open approach to autonomous vehicles. IEEE Micro, 35(6):60– 68, 2015. 5

    2015

  10. [18]

    Grounding human-to-vehicle advice for self-driving vehicles, 2019

    Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny. Grounding human-to-vehicle advice for self-driving vehicles, 2019. 2

    2019

  11. [19]

    Advisable learning for self-driving vehicles by internalizing observation-to-action rules

    Jinkyu Kim, Suhong Moon, Anna Rohrbach, Trevor Darrell, and John Canny. Advisable learning for self-driving vehicles by internalizing observation-to-action rules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9661–9670, 2020. 2

    2020

  12. [20]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Syst...

    2020

  13. [21]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 2

    2023

  14. [22]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. 2

    2024

  15. [23]

    Driver digital twin for online prediction of personal- ized lane change behavior

    Xishun Liao, Xuanpeng Zhao, Ziran Wang, Zhouqiao Zhao, Kyungtae Han, Rohit Gupta, Matthew J Barth, and Guoyuan Wu. Driver digital twin for online prediction of personal- ized lane change behavior. IEEE Internet of Things Journal,

  16. [24]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024. 5

    2024

  17. [25]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2

    2023

  18. [26]

    Deja vu: Con- textual sparsity for efficient LLMs at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuan- dong Tian, Christopher Re, and Beidi Chen. Deja vu: Con- textual sparsity for efficient LLMs at inference time. In Pro- ceedings of the 40th International Conference on...

    2023

  19. [27]

    Vlm- mpc: Vision language foundation model (vlm)-guided model predictive controller (mpc) for autonomous driving

    Keke Long, Haotian Shi, Jiaxi Liu, and Xiaopeng Li. Vlm- mpc: Vision language foundation model (vlm)-guided model predictive controller (mpc) for autonomous driving. arXiv preprint arXiv:2408.04821, 2024. 2

    2024 arXiv

  20. [28]

    Human-cyber-physical system for industry 5.0: A review from a human-centric perspective

    Shanhe Lou, Zhongxu Hu, Yiran Zhang, Yixiong Feng, Mengchu Zhou, and Chen Lv. Human-cyber-physical system for industry 5.0: A review from a human-centric perspective. IEEE Trans. Autom. Sci. Eng, pages 1–18, 2024. 1

    2024

  21. [29]

    Valley: Video assistant with large language model enhanced ability, 2023

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability, 2023. 2

    2023

  22. [30]

    Dolphins: Multimodal language model for driving, 2023

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving, 2023. 1, 2

    2023

  23. [31]

    Lampilot: An open benchmark dataset for autonomous driving with language model programs

    Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference on Computer Vi- si...

    2024

  24. [32]

    The three-dimensional normal- distributions transform: an efficient representation for reg- istration, surface analysis, and loop detection

    Martin Magnusson. The three-dimensional normal- distributions transform: an efficient representation for reg- istration, surface analysis, and loop detection . PhD thesis, ¨Orebro universitet, 2009. 5

    2009

  25. [33]

    Scan registration for autonomous mining vehicles using 3d- ndt

    Martin Magnusson, Achim Lilienthal, and Tom Duckett. Scan registration for autonomous mining vehicles using 3d- ndt. Journal of Field Robotics, 24(10):803–827, 2007. 5

    2007

  26. [34]

    Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving, 2024

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving, 2024. 2

    2024

  27. [35]

    Vlp: Vision language planning for autonomous driving,

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving,

  28. [36]

    Allievi, Senem Velipasalar, and Liu Ren

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G. Allievi, Senem Velipasalar, and Liu Ren. VLP: Vision Language Planning for Autonomous Driv- ing. In CVPR, 2024. 1

    2024

  29. [37]

    Vlaad: Vision and language assistant for autonomous driv- ing

    SungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driv- ing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages ...

    2024

  30. [38]

    Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario,

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual ques- tion answering benchmark for autonomous driving scenario,

  31. [39]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. 3

    2022

  32. [40]

    Gapformer: Fast autoregressive trans- formers meet rnns for personalized adaptive cruise control

    Noveen Sachdeva, Ziran Wang, Kyungtae Han, Rohit Gupta, and Julian McAuley. Gapformer: Fast autoregressive trans- formers meet rnns for personalized adaptive cruise control. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) , pages 2528–25...

    2022

  33. [41]

    Schrum, Emily Sumner, Matthew C

    Mariah L. Schrum, Emily Sumner, Matthew C. Gombolay, and Andrew Best. Maveric: A data-driven approach to per- sonalized autonomous driving, 2023. 2

    2023

  34. [42]

    Languagempc: Large language models as decision makers for autonomous driving, 2023

    Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempc: Large language models as decision makers for autonomous driving, 2023. 4

    2023

  35. [43]

    Drivelm: Driving with graph visual question answering, 2024

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering, 2024. 2

    2024

  36. [44]

    Automatic steering methods for au- tonomous automobile path tracking.Robotics Institute, Pitts- burgh, PA, Tech

    Jarrod M Snider et al. Automatic steering methods for au- tonomous automobile path tracking.Robotics Institute, Pitts- burgh, PA, Tech. Rep. CMU-RITR-09-08, 2009. 4

    2009

  37. [45]

    Exploring person- alised autonomous vehicles to influence user trust.Cognitive Computation, 12:1170–1186, 2020

    Xu Sun, Jingpeng Li, Pinyan Tang, Siyuan Zhou, Xiangjun Peng, Hao Nan Li, and Qingfeng Wang. Exploring person- alised autonomous vehicles to influence user trust.Cognitive Computation, 12:1170–1186, 2020. 2

    2020

  38. [46]

    DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models. arXiv, 2024. 1

    2024

  39. [47]

    Drivevlm: The convergence of autonomous driving and large vision-language models, 2024

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models, 2024. 2

    2024

  40. [48]

    Personalized adaptive cruise control via gaussian process regression

    Yanbing Wang, Ziran Wang, Kyungtae Han, Prashant Tiwari, and Daniel B Work. Personalized adaptive cruise control via gaussian process regression. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1496–1502. IEEE, 2021. 1, 2

    2021

  41. [49]

    Gaussian process-based personalized adaptive cruise control

    Yanbing Wang, Ziran Wang, Kyungtae Han, Prashant Tiwari, and Daniel B Work. Gaussian process-based personalized adaptive cruise control. IEEE Transactions on Intelligent Transportation Systems, 23(11):21178–21189, 2022. 1, 2

    2022

  42. [50]

    Driver behavior modeling using game engine and real vehicle: A learning-based approach

    Ziran Wang, Xishun Liao, Chao Wang, David Oswald, Guoyuan Wu, Kanok Boriboonsomsin, Matthew J Barth, Kyungtae Han, BaekGyu Kim, and Prashant Tiwari. Driver behavior modeling using game engine and real vehicle: A learning-based approach. IEEE Transactions on Intelligent Vehicle...

    2020

  43. [51]

    Mobility digital twin: Concept, architecture, case study, and future challenges

    Ziran Wang, Rohit Gupta, Kyungtae Han, Haoxin Wang, Ak- ila Ganlath, Nejib Ammar, and Prashant Tiwari. Mobility digital twin: Concept, architecture, case study, and future challenges. IEEE Internet of Things Journal , 9(18):17452– 17467, 2022. 2

    2022

  44. [52]

    Individual comparisons by ranking meth- ods

    Frank Wilcoxon. Individual comparisons by ranking meth- ods. In Breakthroughs in statistics: Methodology and distri- bution, pages 196–202. Springer, 1992. 8

    1992

  45. [53]

    Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assess- ment, 2023

    Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assess- ment, 2023. 5

    2023

  46. [54]

    The study of language

    George Yule. The study of language. Cambridge university press, 2022. 6

    2022

  47. [55]

    Proctor, and Jiansong Zhang

    Cong Zhang, Chi Tian, Tianfang Han, Hang Li, Yiheng Feng, Yunfeng Chen, Robert W. Proctor, and Jiansong Zhang. Evaluation of infrastructure-based warning system on driving behaviors-a roundabout study, 2023. 2

    2023

  48. [56]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding, 2023. 2

    2023

  49. [57]

    Personalized adaptive cruise control based on steady-state operation, 2023

    Zhouqiao Zhao, Ziran Wang, Rohit Gupta, Kyungtae Han, and Prashant Tiwari. Personalized adaptive cruise control based on steady-state operation, 2023. US Patent App. 17/578,330. 2

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.