Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight vision-language model, fed road-hazard alerts and bird's-eye-view frames, can correct a vehicle's planned path on an edge GPU in 0.57 seconds and cut simulated collisions by 77%.

desk verdict A sensible residual-fusion VLM system for V2X edge planning, undermined by an evaluation whose headline metrics are not actually computed from the stated outputs. read the letter →

arxiv 2508.01057 v2 pith:X3ZLND4Z submitted 2025-08-01 cs.AI cs.RO

Fengze Yang , Bo Yu , Yang Zhou , Xuewen Luo , Zhengzhong Tu , Chenxi Liu This is my paper
classification cs.AIcs.RO
keywords vision-languagemodelsautonomousdrivingV2Xcooperativeperceptiontrajectoryplanningcollisionavoidanceedgecomputingbird's-eyeviewresidualfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a lightweight ~500M-parameter vision-language model can act as a real-time trajectory co-pilot when roadside units warn it about hazards the ego vehicle cannot see. The proposed system, REACT, fuses two bird's-eye-view frames with structured numerical prompts describing hazard location, navigation waypoints, and ego state, then uses chain-of-thought supervised fine-tuning to output small residual corrections to the nominal path. On the DeepAccident benchmark it reports state-of-the-art motion-prediction scores ($\mathrm{mIOU}$ 59.1%, VPQ 48.2%), a 77% collision-rate reduction relative to BEVerse-tiny, and 0.57-second inference on a Jetson AGX device. If the claim is right, language-guided contextual reasoning becomes a practical, edge-deployable complement to onboard sensing rather than a costly offline tool.

What carries the argument

The load-bearing mechanism is Residual Trajectory Fusion (RTF): the VLM emits per-waypoint offsets $\Delta W=\{\Delta g_1,\ldots,\Delta g_M\}$ that are added element-wise to the nominal trajectory $W_{nom}$ to form the refined path $\hat W$. RTF reduces output token complexity and inference latency while keeping the refined plan kinematically close to the original, and the ablation shows removing it raises the collision rate to 45.5% and slows inference to 3.46 s. The supporting machinery is the prompt chain: two ego-anchored BEV frames (current and one $\Delta t$ earlier) with metric axis overlays supply visual motion cues; hazard, navigation, and ego-state data are translated into the ego frame and time-normalized; and chain-of-thought supervised fine-tuning teaches the model to reason before emitting residuals. Edge adaptation then compresses this pipeline through 16-bit quantization, scaled dot-product attention, reduced text-token length (1500 tokens), and a $64\times 64$ BEV.

What would settle it

Take REACT's final checkpoint and the DeepAccident validation set, define explicit conversion rules from its single refined trajectory to the occupancy maps used in Eqs. (25)–(27) and to the $N$ candidate trajectories used in Eqs. (28)–(30), and recompute Table 2; if the rewritten protocol moves REACT below AccidentGPT or shrinks the 77% collision-rate reduction, the central performance claim fails.

Watch

Extended reading notes

Core claim

REACT's central discovery is that residual trajectory fusion lets a small VLM outperform larger transformer fusion backbones on V2X motion prediction while remaining fast enough for edge use. Instead of generating a complete future path, the model predicts waypoint displacement residuals $\Delta W=\{\Delta g_j\}$, and the residual trajectory fusion module adds these to the nominal waypoints, $\hat W=\{g_j+\Delta g_j\}$. This keeps the output close to the original plan, shortens the decoding target, and makes the model's predictions directly usable for collision avoidance. The paper reports that this design, combined with two-frame BEV inputs, coordinate-overlaid spatial grounding, and chain-of-thought supervised fine-tuning, achieves the highest $\mathrm{mIOU}$ and VPQ among the compared V2X fusion methods and cuts the frame-wise collision rate by about 77% relative to the BEVerse-tiny baseline.

Load-bearing premise

The benchmark comparison assumes that REACT's single waypoint-residual output can be scored with VPQ/mIOU and minADE/minFDE exactly like prior motion-prediction models, even though the paper never specifies how that one trajectory is converted into occupancy motion maps and into the $N$ candidate trajectories these metrics require.

Editorial extensions

If this is right

  • The hazard-avoidance behavior generalizes across weather and lighting: collision-rate reduction stays between 76.95% and 77.07% over all 11 tested conditions.
  • The edge-adaptation package cuts inference time from about 1.7–2.0 s to 0.55–0.57 s on the Jetson AGX with VPQ dropping by less than 1 point relative to the full-resolution input.
  • Removing RTF raises the collision rate to 45.5% and minADE to 3.070 m, so predicting residuals rather than absolute trajectories is what keeps the system both safe and fast.
  • Camera frames and RSU hazard alerts are the two most safety-relevant inputs: ablating camera raises collision rate to 36.5%, and ablating RSU hazard raises it to 30.0%.
  • The planner produces qualitatively different proactive maneuvers—early stops, lane changes, and lateral deviations—in both vehicles directly behind a crash and vehicles farther upstream.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a fair re-benchmark would need to specify how REACT's single refined trajectory is converted into the occupancy motion maps used for VPQ/mIOU and into the $N$ candidate trajectories used for minADE/minFDE; the paper leaves that conversion undefined, so the published deltas are conditional on the conversion the authors used.
  • Beyond the paper, the residual-correction paradigm is portable: any planner with a nominal trajectory could attach a small VLM or even text-only LLM that emits waypoint offsets, making language-guided trajectory adjustment cheaper to adopt outside V2X accident scenarios.
  • Beyond the paper, since the 5-meter collision rule is applied to predicted positions, a closed-loop CARLA run with the ego vehicle actually executing REACT's outputs would reveal whether the 77% reduction survives control dynamics and perception noise.
  • Beyond the paper, a natural stress test is applying REACT to near-miss events and vulnerable-road-user intent—the cases listed as future work—where weaker visual cues would separate genuine contextual reasoning from simple braking at known hazards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents REACT, a V2X-integrated trajectory refinement framework based on a lightweight vision-language model. REACT takes onboard camera BEV maps, RSU hazard alerts, navigation waypoints, and ego state as input, and it is fine-tuned to output residual waypoint corrections. These residuals are added to a nominal trajectory (Eq. 24), and the framework is designed for deployment on edge hardware such as the Jetson AGX. The authors report state-of-the-art performance on the DeepAccident benchmark, including mIOU 59.1%, VPQ 48.2%, a 77% collision-rate reduction relative to a BEVerse-tiny baseline, and 0.57 s inference latency on Jetson AGX Orin. The paper also presents ablations over inputs, modules, and edge-adaptation strategies, together with robustness analyses across weather, time-of-day, and location types.

Significance. If the reported results were valid, REACT would be a useful demonstration that a 512M-parameter VLM can perform cooperative, safety-oriented trajectory refinement at edge-computing latencies. The paper has clear strengths: the system description is fairly complete, the edge-adaptation ablations (quantization, token reduction, BEV resolution) are a useful contribution, and the latency measurements on real hardware are concrete. However, the central performance claims rest on an evaluation protocol that is not aligned with the model's actual output. The VLM produces a single ego-centered waypoint residual, whereas the headline metrics (VPQ, mIOU, minADE, minFDE, CRR) require either occupancy maps over all agents, multiple candidate trajectories, or predicted positions of surrounding vehicles. The conversion from the model output to these metrics is never described, which makes the reported state-of-the-art comparison and the 77% collision-reduction figure unverifiable as stated.

major comments (5)
  1. [Section 5.2, Eqs. 25-27 vs. Section 3.7, Eq. 24] VPQ and mIOU are defined over predicted motion/occupancy maps M_t and A_s, but REACT's output is a single refined ego trajectory W_hat = {g_j + Δg_j}. No procedure is given for converting a waypoint sequence into the occupancy maps required by Eqs. 25-27, nor is it explained how the model predicts occupancy for non-ego vehicles. Table 2's mIOU=59.1% and VPQ=48.2% therefore lack a well-defined computation, and the comparison against V2XFormer/AccidentGPT is not meaningful without this conversion.
  2. [Section 5.2, Eqs. 28-30] minADE and minFDE require N candidate trajectories {W_n}, but REACT's forward pass produces one deterministic trajectory (Eq. 24). The number N is not stated anywhere, and no sampling or multi-modal decoding procedure is described. If N=1 is used, minADE/minFDE reduce to ordinary ADE/FDE and are not comparable to baselines that use N>1 candidates; if N>1, the candidate-generation mechanism is missing. Tables 5 and 6 report minADE/FDE values without resolving this ambiguity.
  3. [Section 5.2, Eqs. 31-37] The CRR and MCD metrics require predicted ego positions p_ego and predicted surrounding-vehicle positions p_j at each future time step, and the surrounding positions are defined as the set P_sur. The method section defines only residual corrections for the ego vehicle's waypoints (Eq. 23); it never specifies how the model predicts positions of surrounding vehicles. The paper must state whether those positions come from the VLM, from BEVerse-tiny detections, from ground-truth annotations, or from another module. Without this, the headline 77% collision-rate reduction is not reproducible and could be confounded by the source of the surrounding-vehicle predictions.
  4. [Section 3.6, Eq. 21] The ground-truth output Y is defined as waypoint-wise deltas ΔW_j^*, but the paper does not explain how these deltas are computed from the DeepAccident data. In particular, it is not specified which nominal trajectory is used to form the residual targets, what time horizon the deltas cover, or how the waypoints are matched to ground-truth future positions. This missing definition affects the reproducibility of the fine-tuning procedure and the interpretation of the RTF module's role.
  5. [Section 5.4.2-5.4.3, Eq. 35] The collision-rate reduction is measured against BEVerse-tiny, and BEVerse-tiny is also used to provide the nominal trajectory that REACT refines. Because the VLM is trained to predict residual corrections toward ground-truth trajectories, the reported 77% CRR partly reflects how well the model has fitted the training distribution relative to a fixed baseline rather than an independent safety improvement. A stronger evaluation would compare REACT against several different nominal planners or report absolute collision rates in addition to the relative reduction.
minor comments (5)
  1. [Tables 5 and 6] The column labeled 'Motion ↑' is never defined in Section 5.2 or elsewhere; please clarify what metric it denotes and how it is computed.
  2. [Figure 4 caption] The caption lists subfigures (a), (b), (c), (e), (f), and (g), but skips (d), and the text referring to '(d)' actually describes subfigure (e). Please renumber the panels or correct the references.
  3. [Abstract and Section 5.1] The Abstract reports inference on 'Jetson AGX Orin,' while Section 5.1 states that experiments were run on 'Jetson AGX Xavier.' Please reconcile the hardware name.
  4. [Eq. 15] The notation 'lengths H, Win pixels' is ambiguous; it should presumably read 'lengths H, W in pixels.'
  5. [Section 3.1, 3.6, and Table 5] The module is called 'Task Alignment' in the overview and 'Task Projection Enhancement' in the detailed description and ablation table. Use a single consistent name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain; the reported metric-conversion gap is a verification problem, not an equation-level circularity.

full rationale

The paper's central derivation is a supervised residual-trajectory predictor: the VLM is fine-tuned with a causal cross-entropy loss (Eq. 22) to emit waypoint residuals (Eq. 23), and the refined trajectory is the element-wise sum of the nominal trajectory and the residual (Eq. 24). The ground-truth residual labels are human- or dataset-derived target adjustments, not functions of the reported evaluation metrics, so the minADE, VPQ, mIOU, and CRR numbers are empirical measurements of the trained model rather than algebraic consequences of the definitions. The 77% collision-rate reduction compares a baseline trajectory (BEVerse-tiny) against that baseline plus the learned residual; this is the intended intervention effect, not a tautology. Self-citations (e.g., references [20], [23], and [46]) appear in the literature review and framing sections, and none is used as the load-bearing justification for the framework's architecture or for the claimed SOTA results. The main substantive concern is a missing specification: Eqs. 25-30 define VPQ, mIOU, minADE, and minFDE over occupancy motion maps and N candidate trajectories, while Eqs. 23-24 define only a single sequence of waypoint residuals, and no procedure is given for rasterizing these residuals into the required maps or for sampling N candidates; similarly, Eqs. 31-35 require predicted surrounding-vehicle positions that the stated method does not derive. That gap makes the numerical comparison difficult to reproduce, but it is a measurement-validity and reproducibility issue, not a circularity in which a claimed prediction reduces by construction to its own input.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims depend on several author-chosen thresholds and evaluation choices (5 m collision threshold, delta_t, K, M', edge configuration, checkpoint) and on assumptions about the dataset, baseline, RSU alerts, kinematics, and simulator fidelity. No new physical or conceptual entities are introduced.

free parameters (6)
  • Collision threshold (5 m) = 5 m
    Eq. 33 records a collision whenever min distance is below 5 m; the headline CRR depends on this hand-set threshold and the choice of baselines.
  • Alert recency threshold delta_t = not specified
    Eq. 9 accepts only alerts with |th - tnow| < delta_t, but delta_t is never given; it controls which hazards reach the planner.
  • Ego state history K = 2 s
    Eq. 12 keeps the past K seconds of ego states and states K=2 s is empirical; this choice shapes the symbolic prompt.
  • Effective waypoint horizon M' = not specified
    Eq. 11 retains a short horizon M' of future waypoints, but M' is never quantified; it determines navigation context.
  • Edge deployment configuration = 16-bit, 1500 tokens, 64x64 BEV
    Section 5.1 selects these values from the ablation results; the 0.57 s latency and VPQ 48.2% claims depend on them.
  • Deployed checkpoint = step 3,400
    Section 5.3 picks the checkpoint by monitoring validation minADE, so the reported results depend on this model selection choice.
assumptions (5)
  • domain assumption DeepAccident provides valid ground-truth trajectories and crash labels for training and evaluation.
    The fine-tuning pipeline and all metrics in Section 5 treat dataset annotations as ground truth without error analysis.
  • domain assumption BEVerse-tiny produces a reasonable nominal trajectory baseline.
    The 77% collision reduction in Eq. 35 is measured against BEVerse-tiny; an artificially weak baseline inflates the reduction.
  • domain assumption RSU hazard alerts are accurate after the simple validation filter.
    Eqs. 6-10 assume the broadcast hazard position and time are correct, with no communication or perception error model.
  • standard math Constant-velocity kinematics with zero vertical change describe vehicle motion over each interval.
    Eqs. 2-5 propagate states under constant velocity and constant height; this is standard for a 2 s horizon but is still an assumption.
  • domain assumption CARLA simulation performance transfers to real-world driving.
    All experiments use the CARLA-based DeepAccident benchmark; conclusions about traffic safety assume simulator fidelity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance." pith.science (2026). https://pith.science/paper/X3ZLND4Z

@misc{pith2026250801057,
  author       = {Pith},
  title        = {Pith review of: Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3ZLND4Z}},
  note         = {Machine review of arXiv:2508.01057}
}
read the original abstract

Autonomous driving (AD) systems relying solely on onboard sensors may fail to detect distant or obstacle hazards, potentially causing preventable collisions; however, existing transformer-based Vehicle-to-Everything (V2X) approaches, which mitigate AD sensing limitations, either lack effective multimodal fusion and reasoning or struggle to meet real-time performance requirements under complex, high-dimensional traffic conditions. This paper proposes the Real-time Edge-based Autonomous Co-pilot Trajectory planner (REACT), a V2X-integrated trajectory optimization framework for AD based on a fine-tuned lightweight Vision-Language Model (VLM). REACT integrates infrastructure-provided hazard alerts with onboard sensor data, capturing intricate surrounding traffic dynamics and vehicle intents through visual embeddings, interpreting precise numerical data from symbolic inputs, and employing contextual reasoning to generate optimized, safety-oriented trajectories. To ensure robust real-time deployment on edge devices, REACT innovatively employs Residual Trajectory Fusion (RTF) design and specialized edge-adaptation strategies to reduce model complexity and improve inference efficiency. Evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance, a 77% collision rate reduction, a 48.2% Video Panoptic Quality (VPQ), and a 0.57-second inference latency on the Jetson AGX Orin. Ablation studies validate the contribution of each input, module, and edge adaptation strategy. These results highlight the effectiveness of lightweight VLMs in enabling real-time cooperative planning on edge platforms and underscore the potential of language-guided contextual reasoning for improving traffic safety and responsiveness.

Figures

Figures reproduced from arXiv: 2508.01057 by the authors.

Figure 1
Figure 1. The Overview of the REACT are described in the following sections, after the formal configuration of the system. The detailed architecture of REACT is shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The Architecture of the REACT Each alert provides the predicted collision location (xh, yh, zh) in the RSU-centered coordinate frame FR, along with a timestamp th indicating the anticipated hazard time. Complementing this, the onboard route planner supplies a set of M target waypoints N . Additionally, real-time kinematic data, represented as a state vector se,k, is obtained through the Controller Area Network (CAN)… view at source ↗
Figure 3
Figure 3. Edge computing adaption process The final optimized trajectory Wˆ is computed by element-wise addition in Eq.24. Wˆ = {gj + ∆gj | gj ∈ Wnom} (24) This approach ensures that the updated plan Wˆ remains closely aligned with the original nominal trajectory while adaptively refining it to avoid predicted risks. The optimized trajectory is subsequently provided to the vehicle control module for execution. 4 Edge Computin… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative simple examples of REACT predictions in accident-prone scenes. (a) Left-turn avoidance at [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Qualitative complex examples of REACT predictions in accident-prone scenes. (a) Pull right to avoid accident [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Effect of location type on REACT performance [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Impact of BEV size and pruning on VPQ and inference time [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ATRACT: A Trustworthy Robotic Autonomous system to support Casualty Triage

    cs.HC 2026-05 unverdicted novelty 4.0 of 10

    ATRACT integrates drone video and wearable sensor data with conditional variational autoencoder augmentation to achieve 85.7% accuracy in casualty action classification for remote battlefield triage.

Reference graph

Works this paper leans on

73 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Road traffic injuries fact sheet,

    World Health Organization, “Road traffic injuries fact sheet,” 2023. Every year approximately 1.19 million deaths and 20–50 million non-fatal injuries globally

    work page 2023

  2. [2]

    Traffic safety facts 2023: A compilation of motor vehicle traffic crash data (annual report),

    National Highway Traffic Safety Administration, “Traffic safety facts 2023: A compilation of motor vehicle traffic crash data (annual report),” annual report, National Highway Traffic Safety Administration, National Center for Statistics and Analysis, Washington, DC, 2023

    work page 2023

  3. [3]

    Distracted driving: Cdc transportation safety,

    Centers for Disease Control and Prevention, “Distracted driving: Cdc transportation safety,” 2023. In U.S., about 3,275 deaths/year from distracted driving and texting increases crash risk by over 23×. 21 arXiv Template A PREPRINT

    work page 2023

  4. [4]

    Why anticipatory sensing matters in commercial acc systems under cut-in scenarios: A perspective from stochastic safety analysis,

    H. Zhang, S. Li, Z. Li, M. Anis, D. Lord, and Y . Zhou, “Why anticipatory sensing matters in commercial acc systems under cut-in scenarios: A perspective from stochastic safety analysis,” Accident Analysis & Prevention, vol. 218, p. 108064, 2025

    work page 2025

  5. [5]

    V2x cooperative perception for autonomous driving: Recent advances and challenges,

    T. Huang, J. Liu, X. Zhou, D. C. Nguyen, M. R. Azghadi, Y . Xia, Q.-L. Han, and S. Sun, “V2x cooperative perception for autonomous driving: Recent advances and challenges,” arXiv preprint arXiv:2310.03525, 2023

    arXiv 2023

  6. [6]

    Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happen

    Z. Li, X. Cao, X. Gao, K. Tian, K. Wu, M. Anis, H. Zhang, K. Long, J. Jiang, X. Li,et al., “Simulating the unseen: Crash prediction must learn from what did not happen,” arXiv preprint arXiv:2505.21743, 2025

    work page Pith review arXiv 2025

  7. [7]

    Advancing vulnerable road users safety: Interdisciplinary review on v2x communication and trajectory prediction,

    B. Abdi, S. Mirzaei, M. Adl, S. Hidajat, and A. Emadi, “Advancing vulnerable road users safety: Interdisciplinary review on v2x communication and trajectory prediction,”IEEE Transactions on Intelligent Transportation Systems, 2024

    work page 2024

  8. [8]

    Transformers in vision: A survey,

    S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,”ACM computing surveys (CSUR), vol. 54, no. 10s, pp. 1–41, 2022

    2022

Show all 73 references
  1. [9]

    Do vision transformers see like con- volutional neural networks?,

    M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like con- volutional neural networks?,” Advances in neural information processing systems , vol. 34, pp. 12116–12128, 2021

    2021

  2. [10]

    Cmoa: Contrastive mixture of adapters for generalized few-shot continual learning,

    Y . Cui, J. Zhao, Z. Yu, R. Cai, X. Wang, L. Jin, A. C. Kot, L. Liu, and X. Li, “Cmoa: Contrastive mixture of adapters for generalized few-shot continual learning,” IEEE Transactions on Multimedia, 2025

    2025

  3. [11]

    Decision-making driven by driver intelligence and environment reasoning for high-level autonomous vehicles: a survey,

    Y . Wang, J. Jiang, S. Li, R. Li, S. Xu, J. Wang, and K. Li, “Decision-making driven by driver intelligence and environment reasoning for high-level autonomous vehicles: a survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 10, pp. 10362–10381, 2023

    2023

  4. [12]

    Summary and reflections on pedestrian trajectory prediction in the field of autonomous driving,

    Z. Fu, K. Jiang, C. Xie, Y . Xu, J. Huang, and D. Yang, “Summary and reflections on pedestrian trajectory prediction in the field of autonomous driving,” IEEE Transactions on Intelligent Vehicles, 2024

    2024

  5. [13]

    A review on explainability in multimodal deep neural nets,

    G. Joshi, R. Walambe, and K. Kotecha, “A review on explainability in multimodal deep neural nets,”IEEE Access, vol. 9, pp. 59800–59821, 2021

    2021

  6. [14]

    Large language models and multimodal foundation models for precision oncology,

    D. Truhn, J.-N. Eckardt, D. Ferber, and J. N. Kather, “Large language models and multimodal foundation models for precision oncology,”NPJ Precision Oncology, vol. 8, no. 1, p. 72, 2024

    2024

  7. [15]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  8. [16]

    Generative ai for autonomous driving: Frontiers and opportunities,

    Y . Wang, S. Xing, C. Can, R. Li, H. Hua, K. Tian, Z. Mo, X. Gao, K. Wu, S. Zhou, et al., “Generative ai for autonomous driving: Frontiers and opportunities,” arXiv preprint arXiv:2505.08854, 2025

    2025 arXiv

  9. [17]

    Frontiers of emerging ai technologies best practices and workforce develop- ment in transportation: Nsf ai–transportation workshop phase ii,

    C. Pu, C. Liu, Y . Wang, and L. Du, “Frontiers of emerging ai technologies best practices and workforce develop- ment in transportation: Nsf ai–transportation workshop phase ii,” Journal of Transportation Engineering, Part A: Systems, vol. 150, no. 9, p. 02524002, 2024

    2024

  10. [18]

    Virtual roads, smarter safety: A digital twin framework for mixed autonomous traffic safety analysis,

    H. Zhang, X. Yue, K. Tian, S. Li, K. Wu, Z. Li, D. Lord, and Y . Zhou, “Virtual roads, smarter safety: A digital twin framework for mixed autonomous traffic safety analysis,”arXiv preprint arXiv:2504.17968, 2025

    2025 arXiv

  11. [19]

    Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning,

    Y . Wang, W. Chen, X. Han, X. Lin, H. Zhao, Y . Liu, B. Zhai, J. Yuan, Q. You, and H. Yang, “Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning,” arXiv preprint arXiv:2401.06805, 2024

    2024 arXiv

  12. [20]

    A self-supervised multi-agent large language model framework for customized traffic mobility analysis using machine learning models,

    F. Yang, X. C. Liu, L. Lu, B. Wang, and C. Liu, “A self-supervised multi-agent large language model framework for customized traffic mobility analysis using machine learning models,” Transportation Research Record, p. 03611981251322468, 2025

    2025

  13. [21]

    Vision-language models in remote sensing: Current progress and future trends,

    X. Li, C. Wen, Y . Hu, Z. Yuan, and X. X. Zhu, “Vision-language models in remote sensing: Current progress and future trends,” IEEE Geoscience and Remote Sensing Magazine, 2024

    2024

  14. [22]

    V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models,

    J. You, H. Shi, Z. Jiang, Z. Huang, R. Gan, K. Wu, X. Cheng, X. Li, and B. Ran, “V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models,”arXiv preprint arXiv:2408.09251, 2024

    2024 arXiv

  15. [23]

    V2x-unipool: Unifying multimodal perception and knowledge reasoning for autonomous driving,

    X. Luo, F. Yang, F. Ding, X. Gao, S. Xing, Y . Zhou, Z. Tu, and C. Liu, “V2x-unipool: Unifying multimodal perception and knowledge reasoning for autonomous driving,” arXiv preprint arXiv:2506.02580, 2025

    2025

  16. [24]

    Improved zero-shot classification by adapting vlms with text descriptions,

    O. Saha, G. Van Horn, and S. Maji, “Improved zero-shot classification by adapting vlms with text descriptions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17542–17552, 2024

    2024

  17. [25]

    Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection,

    Q. Lei, B. Wang, and R. Tan, “Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection,” Advances in Neural Information Processing Systems, vol. 37, pp. 55831–55857, 2024. 22 arXiv Template A PREPRINT

    2024

  18. [26]

    Interpreting black-box models: a review on explainable artificial intelligence,

    V . Hassija, V . Chamola, A. Mahapatra, A. Singal, D. Goel, K. Huang, S. Scardapane, I. Spinelli, M. Mahmud, and A. Hussain, “Interpreting black-box models: a review on explainable artificial intelligence,” Cognitive Computation, vol. 16, no. 1, pp. 45–74, 2024

    2024

  19. [27]

    Explainable ai: A brief survey on history, research areas, approaches and challenges,

    F. Xu, H. Uszkoreit, Y . Du, W. Fan, D. Zhao, and J. Zhu, “Explainable ai: A brief survey on history, research areas, approaches and challenges,” in Natural language processing and Chinese computing: 8th cCF international conference, NLPCC 2019, dunhuang, China, October 9–14, ...

    2019

  20. [28]

    Edge intelligence empowered vehicle detection and image segmentation for autonomous vehicles,

    C. Chen, C. Wang, B. Liu, C. He, L. Cong, and S. Wan, “Edge intelligence empowered vehicle detection and image segmentation for autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 11, pp. 13023–13034, 2023

    2023

  21. [29]

    Deepedgebench: Benchmarking deep neural networks on edge devices,

    S. P. Baller, A. Jindal, M. Chadha, and M. Gerndt, “Deepedgebench: Benchmarking deep neural networks on edge devices,” in 2021 IEEE International Conference on Cloud Engineering (IC2E), pp. 20–30, IEEE, 2021

    2021

  22. [30]

    Prompt engineering in large language models,

    G. Marvin, N. Hellen, D. Jjingo, and J. Nakatumba-Nabende, “Prompt engineering in large language models,” in International conference on data intelligence and cognitive informatics, pp. 387–402, Springer, 2023

    2023

  23. [31]

    An overview of domain-specific foundation model: key technologies, applications and challenges,

    H. Chen, H. Chen, Z. Zhao, K. Han, G. Zhu, Y . Zhao, Y . Du, W. Xu, and Q. Shi, “An overview of domain-specific foundation model: key technologies, applications and challenges,” arXiv preprint arXiv:2409.04267, 2024

    2024

  24. [32]

    Delving into multi-modal multi-task foundation models for road scene understanding: From learning paradigm perspectives,

    S. Luo, W. Chen, W. Tian, R. Liu, L. Hou, X. Zhang, H. Shen, R. Wu, S. Geng, Y . Zhou,et al., “Delving into multi-modal multi-task foundation models for road scene understanding: From learning paradigm perspectives,” IEEE Transactions on Intelligent Vehicles, 2024

    2024

  25. [33]

    Drivevlm: The convergence of autonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “Drivevlm: The convergence of autonomous driving and large vision-language models,” arXiv preprint arXiv:2402.12289, 2024

    2024 arXiv

  26. [34]

    Drivelm: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” in European Conference on Computer Vision , pp. 256–274, Springer, 2024

    2024

  27. [35]

    Autotrust: Benchmark- ing trustworthiness in large vision language models for autonomous driving,

    S. Xing, H. Hua, X. Gao, S. Zhu, R. Li, K. Tian, X. Li, H. Huang, T. Yang, Z. Wang,et al., “Autotrust: Benchmark- ing trustworthiness in large vision language models for autonomous driving,”arXiv preprint arXiv:2412.15206, 2024

    2024

  28. [36]

    Scvlm: Enhancing vision-language model for safety-critical event understanding,

    L. Shi, B. Jiang, T. Zeng, and F. Guo, “Scvlm: Enhancing vision-language model for safety-critical event understanding,” in Proceedings of the Winter Conference on Applications of Computer Vision, pp. 1061–1071, 2025

    2025

  29. [37]

    He-drive: Human-like end-to-end driving with vision language models,

    J. Wang, X. Zhang, Z. Xing, S. Gu, X. Guo, Y . Hu, Z. Song, Q. Zhang, X. Long, and W. Yin, “He-drive: Human-like end-to-end driving with vision language models,” arXiv preprint arXiv:2410.05051, 2024

    2024

  30. [38]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision,

    Y . Xu, Y . Hu, Z. Zhang, G. P. Meyer, S. K. Mustikovela, S. Srinivasa, E. M. Wolff, and X. Huang, “Vlm-ad: End-to-end autonomous driving through vision-language model supervision,” arXiv preprint arXiv:2412.14446, 2024

    2024 arXiv

  31. [39]

    Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,

    P. Liu, H. Liu, H. Liu, X. Liu, J. Ni, and J. Ma, “Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion,” arXiv preprint arXiv:2502.18042, 2025

    2025

  32. [40]

    Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driving,

    Z. Huang, Z. Sheng, Y . Qu, J. You, and S. Chen, “Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driving,”arXiv preprint arXiv:2412.15544, 2024

    2024 arXiv

  33. [41]

    Drivemm: All-in-one large multimodal model for autonomous driving,

    Z. Huang, C. Feng, F. Yan, B. Xiao, Z. Jie, Y . Zhong, X. Liang, and L. Ma, “Drivemm: All-in-one large multimodal model for autonomous driving,” arXiv preprint arXiv:2412.07689, 2024

    2024 arXiv

  34. [42]

    Emma: End-to-end multimodal model for autonomous driving,

    J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp,et al., “Emma: End-to-end multimodal model for autonomous driving,” arXiv preprint arXiv:2410.23262, 2024

    2024 arXiv

  35. [43]

    Edgeshard: Efficient llm inference via collaborative edge computing,

    M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang, “Edgeshard: Efficient llm inference via collaborative edge computing,” IEEE Internet of Things Journal, 2024

    2024

  36. [44]

    An advanced driving agent with the multimodal large language model for autonomous vehicles,

    J. Chen and S. Lu, “An advanced driving agent with the multimodal large language model for autonomous vehicles,” in 2024 IEEE International Conference on Mobility, Operations, Services and Technologies (MOST), pp. 1–11, IEEE, 2024

    2024

  37. [45]

    Autoreward: Closed-loop reward design with large language models for autonomous driving,

    X. Han, Q. Yang, X. Chen, Z. Cai, X. Chu, and M. Zhu, “Autoreward: Closed-loop reward design with large language models for autonomous driving,” IEEE Transactions on Intelligent Vehicles, 2024

    2024

  38. [46]

    Senserag: Constructing environmental knowledge bases with proactive querying for llm-based autonomous driving,

    X. Luo, C. Liu, F. Ding, F. Yang, Y . Zhou, J. Loo, and H. H. Tew, “Senserag: Constructing environmental knowledge bases with proactive querying for llm-based autonomous driving,” in Proceedings of the Winter Conference on Applications of Computer Vision, pp. 989–996, 2025. 23...

    2025

  39. [47]

    Agentscomerge: Large language model empowered collaborative decision making for ramp merging,

    S. Hu, Z. Fang, Z. Fang, Y . Deng, X. Chen, Y . Fang, and S. Kwong, “Agentscomerge: Large language model empowered collaborative decision making for ramp merging,”arXiv preprint arXiv:2408.03624, 2024

    2024 arXiv

  40. [48]

    V2x-llm: Enhancing v2x integration and understanding in connected vehicle corridors,

    K. Wu, P. Li, Y . Zhou, R. Gan, J. You, Y . Cheng, J. Zhu, S. T. Parker, B. Ran, D. A. Noyce,et al., “V2x-llm: Enhancing v2x integration and understanding in connected vehicle corridors,” arXiv preprint arXiv:2503.02239, 2025

    2025 arXiv

  41. [49]

    Deep learning with edge computing: A review,

    J. Chen and X. Ran, “Deep learning with edge computing: A review,”Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019

    2019

  42. [50]

    Edgevla: Efficient vision-language-action models,

    P. Budzianowski, W. Maa, M. Freed, J. Mo, W. Hsiao, A. Xie, T. Młoduchowski, V . Tipnis, and B. Bolte, “Edgevla: Efficient vision-language-action models,” 2025

    2025

  43. [51]

    Edgecloudai: Edge-cloud distributed video analytics,

    M. Ghasemi, Z. Kostic, J. Ghaderi, and G. Zussman, “Edgecloudai: Edge-cloud distributed video analytics,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp. 1778–1780, 2024

    2024

  44. [52]

    Generative diffusion-based contract design for efficient ai twin migration in vehicular embodied ai networks,

    Y . Zhong, J. Kang, J. Wen, D. Ye, J. Nie, D. Niyato, X. Gao, and S. Xie, “Generative diffusion-based contract design for efficient ai twin migration in vehicular embodied ai networks,”IEEE Transactions on Mobile Computing, 2025

    2025

  45. [53]

    Mobileaibench: Benchmarking llms and lmms for on-device use cases,

    R. Murthy, L. Yang, J. Tan, T. M. Awalgaonkar, Y . Zhou, S. Heinecke, S. Desai, J. Wu, R. Xu, S. Tan, et al., “Mobileaibench: Benchmarking llms and lmms for on-device use cases,” arXiv preprint arXiv:2406.10290, 2024

    2024 arXiv

  46. [54]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He,et al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  47. [55]

    Mobilevlm: A fast, strong and open vision language assistant for mobile devices,

    X. Chu, L. Qiao, X. Lin, S. Xu, Y . Yang, Y . Hu, F. Wei, X. Zhang, B. Zhang, X. Wei,et al., “Mobilevlm: A fast, strong and open vision language assistant for mobile devices,” arXiv preprint arXiv:2312.16886, 2023

    2023 arXiv

  48. [56]

    Mobilevlm v2: Faster and stronger baseline for vision language model,

    X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y . Yang, X. Sun, Y . Hu, X. Lin, B. Zhang,et al., “Mobilevlm v2: Faster and stronger baseline for vision language model,” arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  49. [57]

    Align-kd: Distilling cross-modal alignment knowledge for mobile vision- language model,

    Q. Feng, W. Li, T. Lin, and X. Chen, “Align-kd: Distilling cross-modal alignment knowledge for mobile vision- language model,” arXiv preprint arXiv:2412.01282, 2024

    2024 arXiv

  50. [58]

    Appvlm: A lightweight vision language model for online app control,

    G. Papoudakis, T. Coste, Z. Wu, J. Hao, J. Wang, and K. Shao, “Appvlm: A lightweight vision language model for online app control,” arXiv preprint arXiv:2502.06395, 2025

    2025 arXiv

  51. [59]

    Mobileexperts: A dynamic tool-enabled agent team in mobile devices,

    J. Zhang, C. Zhao, Y . Zhao, Z. Yu, M. He, and J. Fan, “Mobileexperts: A dynamic tool-enabled agent team in mobile devices,” arXiv preprint arXiv:2407.03913, 2024

    2024 arXiv

  52. [60]

    Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,

    D. Zhang, Z. Shen, R. Xie, S. Zhang, T. Xie, Z. Zhao, S. Chen, L. Chen, H. Xu, R. Cao, et al., “Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction,” arXiv preprint arXiv:2305.08144, 2023

    2023 arXiv

  53. [61]

    End-to-end autonomous driving: Challenges and frontiers,

    L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li, “End-to-end autonomous driving: Challenges and frontiers,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  54. [62]

    Privacy-aware anomaly detection and notification enhancement for vanet based on collaborative intrusion detection system,

    G. Zheng, Q. Ni, and Y . Lu, “Privacy-aware anomaly detection and notification enhancement for vanet based on collaborative intrusion detection system,” IEEE Transactions on Intelligent Transportation Systems, 2024

    2024

  55. [63]

    Towards c-v2x enabled collaborative autonomous driving,

    Y . He, B. Wu, Z. Dong, J. Wan, and W. Shi, “Towards c-v2x enabled collaborative autonomous driving,”IEEE Transactions on Vehicular Technology, vol. 72, no. 12, pp. 15450–15462, 2023

    2023

  56. [64]

    3d object detection for autonomous driving: A comprehensive survey,

    J. Mao, S. Shi, X. Wang, and H. Li, “3d object detection for autonomous driving: A comprehensive survey,” International Journal of Computer Vision, vol. 131, no. 8, pp. 1909–1963, 2023

    1909

  57. [65]

    Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving,

    T. Wang, S. Kim, J. Wenxuan, E. Xie, C. Ge, J. Chen, Z. Li, and P. Luo, “Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 5599–5606, 2024

    2024

  58. [66]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning, pp. 1–16, 2017

    2017

  59. [67]

    Fiery: Future instance prediction in bird’s-eye view from surround monocular cameras,

    A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V . Badrinarayanan, R. Cipolla, and A. Kendall, “Fiery: Future instance prediction in bird’s-eye view from surround monocular cameras,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15273–15282, 2021

    2021

  60. [68]

    Disconet: Shapes learning on disconnected manifolds for 3d editing,

    E. Mehr, A. Jourdan, N. Thome, M. Cord, and V . Guitteny, “Disconet: Shapes learning on disconnected manifolds for 3d editing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3474–3483, 2019

    2019

  61. [69]

    V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,

    R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma, “V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,” in European conference on computer vision, pp. 107–124, Springer, 2022. 24 arXiv Template A PREPRINT

    2022

  62. [70]

    Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers,

    R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma, “Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers,” arXiv preprint arXiv:2207.02202, 2022

    2022 arXiv

  63. [71]

    Accidentgpt: Accident analysis and prevention from v2x environmental perception with multi-modal large model,

    L. Wang, Y . Ren, H. Jiang, P. Cai, D. Fu, T. Wang, Z. Cui, H. Yu, X. Wang, H. Zhou, et al., “Accidentgpt: Accident analysis and prevention from v2x environmental perception with multi-modal large model,” arXiv preprint arXiv:2312.13156, 2023

    2023 arXiv

  64. [72]

    Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving,

    Y . Zhang, Z. Zhu, W. Zheng, J. Huang, G. Huang, J. Zhou, and J. Lu, “Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving,”arXiv preprint arXiv:2205.09743, 2022

    2022 arXiv

  65. [73]

    Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,

    J. Huang, G. Huang, Z. Zhu, Y . Yun, and D. Du, “Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,”arXiv preprint arXiv:2112.11790, 2021. 25

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.