Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DRAMA-X claims to be the first large-scale benchmark with nine-class directional intent labels for pedestrians and cyclists in high-risk scenes, and its scene-graph baseline improves intent and risk reasoning over direct VLM querying.

desk verdict Useful benchmark that fills a real gap, but the unvalidated automated intent labels and the overclaimed risk gain mean the headline results need scrutiny before the field builds on them. read the letter →

arxiv 2506.17590 v2 pith:FB7VZWK7 submitted 2025-06-21 cs.CV cs.AIcs.RO

Mihir Godbole , Xiangbo Gao , Zhengzhong Tu This is my paper · ORCID
classification cs.CVcs.AIcs.RO
keywords fine-grainedintentpredictionvulnerableroadusersautonomousdrivingbenchmarkscenegraphriskassessmentactionsuggestionvision-languagemodelsdirectional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DRAMA-X, a benchmark built from 5,686 accident-prone driving frames in which each pedestrian and cyclist receives one of nine directional intent labels, a binary risk score, and a recommended ego-vehicle action. The authors claim this is the first large-scale benchmark to evaluate fine-grained multi-class intent for vulnerable road users in high-risk scenes, and they generate the labels automatically from tracked object motion with an estimated camera-motion correction. They also propose SGG-Intent, a training-free pipeline that converts an image into a scene graph, predicts intents from that graph, then reasons to a risk verdict and an action suggestion. Their experiments report that current vision-language models localize objects poorly, and that the scene-graph pipeline improves intent and risk scores, with ground-truth boxes sharply boosting intent accuracy. A sympathetic reader would take the contribution as a reusable evaluation scaffold: if the labels are valid, four safety-relevant tasks become measurable on one benchmark.

What carries the argument

The load-bearing object is the automated annotation pipeline and its road-relative displacement computation. For each tracked vulnerable road user, the horizontal and vertical components of frame-to-frame displacement, corrected by an estimated camera motion obtained from the median Farneback optical flow in a region adjacent to the object, map onto a nine-class lateral-by-vertical intent taxonomy. This machinery turns raw video into ground-truth intent labels without human annotation, and those labels are what make the benchmark's four tasks and the SGG-Intent evaluation possible.

What would settle it

Take a random sample of DRAMA-X frames, have human annotators label each VRU's lateral and vertical direction from the video, and measure agreement with the pipeline's nine-class labels; if agreement is near chance (say, below 60%), the benchmark's ground truth is not trustworthy. A cheaper check is to compute the pipeline's lateral label for objects known to be stationary on the road, where a systematic nonzero lateral displacement would betray camera-motion bias.

Watch

Extended reading notes

Core claim

DRAMA-X extends DRAMA's high-risk scenes with automatically generated fine-grained intents: for each object, a lateral label (goes left, goes right, stationary) and a vertical label (toward ego, away from ego, stationary), together with relative position, risk, action recommendation, and a language motion description. The intent labels are derived from tracker trajectories made road-relative by subtracting the median optical flow in regions beside the object. On this benchmark, standalone VLMs score near zero on detection and intent; routing perception through SGG-Intent's scene graph improves risk F1 and action suggestion, and supplying ground-truth boxes raises Qwen2.5's combined intent accuracy from 13.47% to 41.73%. The authors conclude that precise localization is the current bottleneck and that structured sequential reasoning helps downstream tasks.

Load-bearing premise

The intent labels are only as good as the estimate of camera motion: the pipeline assumes the median optical flow in the region beside each tracked object reflects pure ego-motion, and if other moving objects, occlusions, or depth differences bias that estimate, the road-relative direction labels are systematically wrong.

Editorial extensions

If this is right

  • A benchmark now exists where detection, intent, risk, and action can be scored jointly, so progress in one stage can be traced to downstream gains.
  • Ground-truth localization lifts combined intent accuracy to 41.73% for Qwen2.5, implying that localization, not intent semantics, is the main bottleneck for current VLMs.
  • Sequential scene-graph reasoning (OD to IP to RA to AS) outperforms direct VLM querying, indicating that task order matters and structured context helps.
  • Risk assessment can remain strong even when object-level predictions are wrong, since SGG-Intent's scene-graph representation carries enough context for risk F1 above 90% for several models.
  • Future work has a clear target: specialized fine-grained intent models for VRUs, since even with perfect boxes current VLMs remain below 42% combined intent accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the annotation pipeline's camera-motion estimate is biased by other moving objects, occlusion, or depth variation, the nine-class labels inherit that bias, so a human-validation subset would materially strengthen trust in every downstream number.
  • The same automated recipe could be applied to other video datasets with tracked agents and known ego-motion, producing intent labels at scale without the optical-flow proxy.
  • One could test whether the nine-class taxonomy is the right granularity by measuring inter-rater agreement on the same clips with human annotators; if humans disagree on direction labels, the task itself is underdefined.
  • The finding that risk assessment survives poor object-level predictions suggests scene-level reasoning may be more robust than the pipeline's sequential logic assumes, possibly because global context, not per-object intent, drives risk.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. DRAMA-X introduces a large-scale benchmark derived from the DRAMA driving-risk dataset, adding automated fine-grained directional intent labels (a claimed nine-class taxonomy), inherited risk scores and action suggestions, and generated motion descriptions for vulnerable road users. The paper also proposes SGG-Intent, a training-free scene-graph-based reasoning baseline, and evaluates several VLMs on four tasks: object detection, intent prediction, risk assessment, and action suggestion. The central claims are that DRAMA-X is the first large-scale fine-grained intent benchmark for pedestrians and cyclists in high-risk scenarios, and that scene-graph-based reasoning improves intent prediction and risk assessment.

Significance. If the intent annotations were validated, DRAMA-X would fill a genuine gap: existing intent datasets use binary or ternary labels, rarely cover cyclists, and do not link intent to risk and action tasks. The benchmark's multi-task structure (detection, intent, risk, action) is well matched to the sequential perception-to-decision pipeline of autonomous driving, and the SGG-Intent baseline is a useful reference point for future work. The paper also provides a reproducible evaluation protocol with multiple VLMs and a public code/dataset link. However, the significance is conditional on the correctness of the automated intent labels and on the soundness of the claimed improvements, both of which are currently undermined by missing validation and inconsistent reporting.

major comments (4)
  1. [§3.2, Camera Motion Estimation] The road-relative intent labels are computed by subtracting an ego-motion estimate from tracked object motion, but the ego-motion estimate is the mean (or, per the text, median) optical flow in a rectangular region Rt 'around the object.' No details are given for how Rt excludes the target object, other moving VRUs, or depth discontinuities; if any of these contaminate Rt, the sign of the lateral and vertical displacement can flip. The paper also does not specify the thresholds used to convert displacement magnitude into 'stationary' versus movement labels, and it provides no human-annotated validation set, no inter-annotator agreement, and no comparison against an independent ego-motion estimator. The authors' own Limitations section admits the pipeline 'may introduce errors in complex scenes with multiple VRUs.' Because every downstream evaluation in Tables 2 and 3 inherits these labels, the benchmark's core novelty rests on an unvalidated proxy. I recommend adding a manually annotated validation subset with per-class confusion matrices and a sensitivity analysis of the Rt definition and displacement thresholds.
  2. [§5.3, Table 2, Risk Assessment paragraph] The claim that SGG-Intent 'enhances risk assessment' is not supported by the balanced accuracy column. For Qwen2.5, the F1 score rises from 66.70 to 95.41 with SGG-Intent, but balanced accuracy falls from 50.52 to 48.79; for GPT-4o, F1 rises from 28.54 to 91.64 while BA falls from 57.00 to 52.50. Since the risk labels are 97% positive in this benchmark (Section 3.3), a model that predicts 'Yes' almost always can achieve a very high positive-class F1 while being no better than chance on balanced accuracy. The paper acknowledges the BA drop but dismisses it by asserting that identifying hazardous scenarios is the primary objective; that assertion does not address the metric's vulnerability to the extreme class imbalance. The risk-assessment conclusion should be re-examined using BA as the primary metric (or with calibration/ROC analysis), and the text should not claim a uniform enhancement when the improvement appears only in F1.
  3. [Table 2 and §5.3, Evaluation] There are internal inconsistencies between the table's grouping labels and the prose that make the quantitative claims hard to verify. The text in 'Evaluating Object Localization Capability' attributes OD=48.53% to Qwen2.5 and OD=2.18% to GPT-4o, but these numbers appear in the '+ SSG-Intent' block of Table 2, not in the 'w/o SSG-Intent' block that the same section describes as 'standalone VLMs.' Also, the statement that standalone VLMs 'achieve significantly lower scores across all tasks' contradicts the table: GPT-4o's OD is 4.54 without SGG-Intent versus 2.18 with it, and its LIP/VIP/Combined intents are also higher in the w/o block. The current labeling makes it unclear which values correspond to which condition, and the narrative overstates the benefit of the framework. The table and text must be aligned and the claims restated to reflect the actual per-model comparisons.
  4. [Abstract, §2, §3.3, and Appendix B.1] The intent taxonomy is described inconsistently across the paper, which is problematic for a benchmark artifact. The abstract and §3.2 state a nine-class directional taxonomy (3 lateral x 3 vertical), but §2 says the dataset provides 'seven classes,' and the caption of Figure 2 lists 'five movement categories' with counts that do not obviously correspond to the nine-class decomposition. The intent generation prompt in Appendix B.1 only offers 'goes to the left' and 'goes to the right' as lateral options, omitting the 'stationary' lateral label that §3.2 defines; the vertical prompt also only lists 'moves away,' 'moves towards,' and 'stationary,' consistent with the vertical set. If the lateral 'stationary' class is never used in evaluation or prompting, the effective taxonomy differs from the stated one. The benchmark documentation must state the exact label set, the mapping from displacement to labels, and how the nine classes (or a reduced set) are derived, so that future users can reproduce the annotations.
minor comments (6)
  1. [Abstract and §3.2] The abstract has a typo: 'safety-critical situations, To address this gap' should be a period or semicolon. The phrase 'we utilize object motions from ground truth scene videos to ensure the validity of generated annotations' is not supported by the pipeline description, which relies on YOLOv8/Faster R-CNN detections and BoT-SORT tracking rather than ground-truth trajectories.
  2. [Table 2] The table headers read 'w/o SSG-Intent' and '+ SSG-Intent,' but the framework is consistently called SGG-Intent elsewhere; the acronym should be unified as SGG throughout.
  3. [Appendix B.1, Raw Output Generation Prompt] The raw output prompt asks the model to output 'Risk' and 'Suggested_action' alongside per-object intents, but the direct-evaluation protocol in Section 5.3 does not describe how these fields are parsed; also, the prompt's lateral intent options omit 'stationary,' contradicting the label space defined in Section 3.2.
  4. [§2, Related Work, second paragraph] The prose says DRAMA-X provides 'seven classes' while the abstract says 'nine-class directional intent taxonomy'; please reconcile the number and define whether the nine classes are the full Cartesian product and how 'stationary' is handled.
  5. [§3.3, Dataset Statistics] Figure 2's right panel caption lists five movement categories with counts; the relationship between these five categories and the nine-class combination of lateral and vertical intents should be explained, since the counts (3,447 leftward; 3,454 away; 3,000 rightward; 2,911 toward; 1,874 stationary) do not sum to 9,606 object instances and appear to be a projection of the label space.
  6. [Throughout] There are several typographical errors: 'LLavA' for LLaVA, 'Cylists' in Appendix B.1, 'V ora' in reference [37], 'stratgies' in Appendix B.1, and 'SSG' for 'SGG' in Table 2. These should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DRAMA-X labels are constructed from raw video via tracking and optical flow, and all evaluated models are zero-shot against those external labels.

full rationale

The paper's central derivation chain is self-contained. DRAMA-X intent labels are produced by an automated pipeline from raw DRAMA video: tracked object motion is compensated by a median-optical-flow estimate of camera displacement, and the nine-class directional intent is read off the sign of the resulting road-relative displacement (Section 3.2). Risk scores and suggested actions are inherited from the original DRAMA annotations, not generated by the models under evaluation. The SGG-Intent baseline is training-free: it prompts VLMs for scene graphs, then for intents, then uses GPT-4o for risk and action reasoning, and all of these outputs are compared against the externally constructed DRAMA-X labels. No parameter is fitted to the benchmark labels and no predicted quantity is defined in terms of the labels used to score it. The same-group citations (e.g., LangCoop, AutoTrust) appear only as contextual related work and are not load-bearing for the benchmark's validity or for the SGG-Intent conclusions. The acknowledged lack of human validation of the optical-flow-based annotation proxy is a correctness and validity risk, but it is not a circularity: the labels do not reduce to the models' outputs, and the models' scores are not inputs to the annotation pipeline.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central annotations depend on tracking and optical-flow assumptions that are not independently validated; the free parameters above are annotation thresholds that change the label set and hence every downstream score. No new physical entities are postulated; the only new artifacts are annotation labels and prompts.

free parameters (5)
  • Object salience filters (min box size and max instances) = min 8% image height, 1% width; max 3 instances per class
    Used in curation to decide which frames and objects enter DRAMA-X; no ablation or sensitivity analysis is reported (Appendix A.2).
  • Cyclist association thresholds = person-bicycle IoU > 0.3; vertical offset <= 160 px
    Combines person and bicycle detections into cyclist labels; changing thresholds changes the 361 cyclist instances and all cyclist intents.
  • Track linking threshold theta_link = 0.2 for gaps <= 3 frames, 0.3 otherwise
    Accepts or rejects links between fragmented tracks; directly affects trajectory length and hence the displacement used for intent labels (Appendix A.3.1).
  • Object matching IoU threshold theta_iou = 0.3
    Used in Hungarian assignment to align tracked objects with filtered detections; affects identity and motion context (Appendix A.3.1).
  • Stationary displacement threshold = unspecified
    Lateral and vertical intents are assigned by sign and magnitude of road-relative displacement, but no explicit threshold for classifying 'stationary' is given in Section 3.2; this hidden cutoff changes the label distribution.
assumptions (4)
  • domain assumption Optical flow in a region adjacent to the object estimates ego camera motion; subtracting it from tracked motion yields road-relative object motion.
    Section 3.2, Camera Motion Estimation. If the background-flow proxy is contaminated by other moving objects, parallax, or occlusion, the intent labels are systematically wrong.
  • domain assumption Automated tracking with YOLOv8, Faster R-CNN, and BoT-SORT, after linking, produces trajectories accurate enough to define ground-truth intents.
    Section 3.2 and Appendix A.3. No human verification or trajectory error analysis is provided; the Limitations section admits errors in complex scenes.
  • domain assumption Immediate short-term displacement direction in the image plane is a valid operationalization of intent for benchmarking.
    Section 3.2, Intent Inference: intent is defined as immediate short-term motion without temporal dependency. Under this definition the label is tautological relative to the trajectory, but as a human-intent construct it is unvalidated.
  • domain assumption DRAMA's existing risk scores and suggested actions are correct and can be reused as ground truth.
    Sections 3.1 and A.2 retain 'Risk' and 'suggested_action' fields from the source dataset without revalidation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving." pith.science (2026). https://pith.science/paper/FB7VZWK7

@misc{pith2026250617590,
  author       = {Pith},
  title        = {Pith review of: DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FB7VZWK7}},
  note         = {Machine review of arXiv:2506.17590}
}
read the original abstract

Understanding the short-term motion of vulnerable road users (VRUs) like pedestrians and cyclists is critical for safe autonomous driving, especially in urban scenarios with ambiguous or high-risk behaviors. While vision-language models (VLMs) have enabled open-vocabulary perception, their utility for fine-grained intent reasoning remains underexplored. Notably, no existing benchmark evaluates multi-class intent prediction in safety-critical situations, To address this gap, we introduce DRAMA-X, a fine-grained benchmark constructed from the DRAMA dataset via an automated annotation pipeline. DRAMA-X contains 5,686 accident-prone frames labeled with object bounding boxes, a nine-class directional intent taxonomy, binary risk scores, expert-generated action suggestions for the ego vehicle, and descriptive motion summaries. These annotations enable a structured evaluation of four interrelated tasks central to autonomous decision-making: object detection, intent prediction, risk assessment, and action suggestion. As a reference baseline, we propose SGG-Intent, a lightweight, training-free framework that mirrors the ego vehicle's reasoning pipeline. It sequentially generates a scene graph from visual input using VLM-backed detectors, infers intent, assesses risk, and recommends an action using a compositional reasoning stage powered by a large language model. We evaluate a range of recent VLMs, comparing performance across all four DRAMA-X tasks. Our experiments demonstrate that scene-graph-based reasoning enhances intent prediction and risk assessment, especially when contextual cues are explicitly modeled.

Figures

Figures reproduced from arXiv: 2506.17590 by the authors.

Figure 1
Figure 1. Filtering and data annotation pipeline used to identify relevant scenarios with key VRU [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution characteristics of the DRAMA-X dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Overview of the novel intent prediction SGG-Intent framework, for evaluation on DRAMA￾X. Given an input image, a vision-language model (VLM) generates a scene graph capturing objects, attributes, and relationships. A separate intent prompt predicts directional motion intents for each dynamic object. These structured representations are passed to a risk reasoning LLM that assesses scene-level risk and suggests ego-ve… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results of the track linking process. Left: fragmented tracks produced by [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Representative DRAMA-X annotations illustrating directional intent (lateral and vertical), [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automated Vehicles Should be Connected with Natural Language

    cs.MA 2025-06 conditional novelty 3.0 of 10

    A vision paper recommending natural language as the universal communication medium for connected and automated vehicles.

Reference graph

Works this paper leans on

78 extracted references · 57 canonical work pages · cited by 1 Pith paper

  1. [1]

    Unsupervised 3d perception with 2d vision-language distillation for autonomous driving

    Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Unsupervised 3d perception with 2d vision-language distillation for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8602–8612, 2023. 1

    work page 2023

  2. [2]

    Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment

    Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23497–23506, 2023

    work page 2023

  3. [3]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024

    arXiv 2024

  4. [4]

    Langcoop: Collaborative driving with language

    Xiangbo Gao, Yuheng Wu, Rujia Wang, Chenxi Liu, Yang Zhou, and Zhengzhong Tu. Langcoop: Collaborative driving with language. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4226–4237, 2025

    work page 2025

  5. [5]

    Generative ai for autonomous driving: Frontiers and opportunities

    Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, et al. Generative ai for autonomous driving: Frontiers and opportunities. arXiv preprint arXiv:2505.08854, 2025

    arXiv 2025

  6. [6]

    Autotrust: Benchmarking trustworthiness in large vision language models for autonomous driving

    Shuo Xing, Hongyuan Hua, Xiangbo Gao, Shenzhe Zhu, Renjie Li, Kexin Tian, Xiaopeng Li, Heng Huang, Tianbao Yang, Zhangyang Wang, et al. Autotrust: Benchmarking trustworthiness in large vision language models for autonomous driving. arXiv preprint arXiv:2412.15206, 2024

    arXiv 2024

  7. [7]

    V2x-unipool: Unifying multimodal perception and knowledge reasoning for autonomous driving

    Xuewen Luo, Fengze Yang, Fan Ding, Xiangbo Gao, Shuo Xing, Yang Zhou, Zhengzhong Tu, and Chenxi Liu. V2x-unipool: Unifying multimodal perception and knowledge reasoning for autonomous driving. arXiv preprint arXiv:2506.02580, 2025

    arXiv 2025

  8. [8]

    Automated vehicles should be connected with natural language

    Xiangbo Gao, Keshu Wu, Hao Zhang, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Automated vehicles should be connected with natural language. arXiv preprint arXiv:2507.01059, 2025. 1

    arXiv 2025

Show all 78 references
  1. [9]

    Simplellm4ad: An end-to-end vision-language model with graph visual question answering for autonomous driving

    Peiru Zheng, Yun Zhao, Zhan Gong, Hong Zhu, and Shaohua Wu. Simplellm4ad: An end-to-end vision-language model with graph visual question answering for autonomous driving. arXiv preprint arXiv:2407.21293, 2024. 1

    2024 arXiv

  2. [10]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024

    2024

  3. [11]

    V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models

    Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, and Bin Ran. V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models. arXiv preprint arXiv:2408.09251, 2024

    2024 arXiv

  4. [12]

    Senna: Bridging large vision-language models and end-to-end autonomous driving

    Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-to-end autonomous driving. arXiv preprint arXiv:2410.22313, 2024

    arXiv

  5. [13]

    Gpt-driver: Learning to drive with gpt

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.01415

    2024 arXiv

  6. [14]

    Ashqar, and Mohammed Elhenawy

    Mohammad Abu Tami, Huthaifa I. Ashqar, and Mohammed Elhenawy. Using multimodal large language models for automated detection of traffic safety-critical events. arXiv preprint arXiv:2406.13894, 2024

    2024 arXiv

  7. [15]

    Openemma: Open-source multimodal model for end-to-end autonomous driving

    Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1001–1009, 2025. 1

    2025

  8. [16]

    Mining patterns of autonomous vehicle crashes involving vulnerable road users to understand the associated factors

    Boniphace Kutela, Subasish Das, and Bahar Dadashova. Mining patterns of autonomous vehicle crashes involving vulnerable road users to understand the associated factors. Accident Analysis & Prevention, 165: 106473, 2022. 1

    2022

  9. [17]

    A matched case-control analysis of autonomous vs human- driven vehicle accidents

    Mohamed Abdel-Aty and Shengxuan Ding. A matched case-control analysis of autonomous vs human- driven vehicle accidents. Nature communications, 15(1):4931, 2024. 1

    2024

  10. [18]

    Simulating the unseen: Crash prediction must learn from what did not happen

    Zihao Li, Xinyuan Cao, Xiangbo Gao, Kexin Tian, Keshu Wu, Mohammad Anis, Hao Zhang, Keke Long, Jiwan Jiang, Xiaopeng Li, et al. Simulating the unseen: Crash prediction must learn from what did not happen. arXiv preprint arXiv:2505.21743, 2025. 1 10

    2025 arXiv

  11. [19]

    Amir Rasouli, Iuliia Kotseruba, and John K. Tsotsos. Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior. In Proc. IEEE Int. Conf. on Computer Vision Workshops (ICCVW), 2017. 2

    2017

  12. [20]

    Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K. Tsotsos. PIE: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In Proc. IEEE Int. Conf. on Computer Vision (ICCV), 2019. 2

    2019

  13. [21]

    Sophie: An attentive gan for predicting paths compliant to social and physical constraints

    Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognit...

    2019

  14. [22]

    Social lstm: Human trajectory prediction in crowded spaces

    Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961–971, 2016

    2016

  15. [23]

    Social graph convolutional lstm for pedestrian trajectory prediction

    Yutao Zhou, Huayi Wu, Hongquan Cheng, Kunlun Qi, Kai Hu, Chaogui Kang, and Jie Zheng. Social graph convolutional lstm for pedestrian trajectory prediction. IET Intelligent Transport Systems, 15(3): 396–405, 2021

    2021

  16. [24]

    Pedast-gcn: Fast pedestrian crossing intention prediction using spatial–temporal attention graph convolution networks.IEEE Transactions on Intelligent Transportation Systems, 2024

    Yancheng Ling, Zhenliang Ma, Qi Zhang, Bangquan Xie, and Xiaoxiong Weng. Pedast-gcn: Fast pedestrian crossing intention prediction using spatial–temporal attention graph convolution networks.IEEE Transactions on Intelligent Transportation Systems, 2024. 2

    2024

  17. [25]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1043–1052, 2023. 2, 3

    2023

  18. [26]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 6

    2025 arXiv

  19. [27]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023. 6

    2023 arXiv

  20. [28]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 6

    2024 arXiv

  21. [29]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,

    arXiv

  22. [30]

    Julien F. P. Kooij, N. Schneider, F. Flohr, and Dariu M. Gavrila. Context-based pedestrian path prediction. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI), 36(6):1242–1257, 2014. 2

    2014

  23. [31]

    Context-based detection of pedestrian crossing intention for autonomous driving in urban environments

    Friederike Schneemann and Patrick Heinemann. Context-based detection of pedestrian crossing intention for autonomous driving in urban environments. In Proc. IEEE Int. Conf. on Intelligent Transportation Systems (ITSC), pages 2793–2798, 2016. 2

    2016

  24. [32]

    Zhijie Fang and Antonio M. Lopez. Is the pedestrian going to cross? answering by 2d pose estimation. In IEEE Intelligent Vehicles Symposium (IV), 2018. 2

    2018

  25. [33]

    Mambast: A plug-and-play cross-spectral spatial-temporal fuser for efficient pedestrian detection

    Xiangbo Gao, Asiegbu Miracle Kanu-Asiegbu, and Xiaoxiao Du. Mambast: A plug-and-play cross-spectral spatial-temporal fuser for efficient pedestrian detection. In 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), pages 2027–2034. IEEE, 2024. 2

    2024

  26. [34]

    Iuliia Kotseruba, Amir Rasouli, and John K. Tsotsos. Pedestrian intention prediction in autonomous driving: A review of data, methods, and evaluations. arXiv:2105.04149, 2021. 2

    2021 arXiv

  27. [35]

    Airv2x: Unified air-ground vehicle-to-everything collaboration

    Xiangbo Gao, Yuheng Wu, Xuewen Luo, Keshu Wu, Xinghao Chen, Yuping Wang, Chenxi Liu, Yang Zhou, and Zhengzhong Tu. Airv2x: Unified air-ground vehicle-to-everything collaboration. arXiv preprint arXiv:2506.19283, 2025. 3

    2025 arXiv

  28. [36]

    Vision meets robotics: The KITTI dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. Journal of Robotics Research, 32(11):1231–1237, 2013. 3

    2013

  29. [37]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3 11

    2020

  30. [38]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, and etc. Scalability in perception for autonomous driving: Waymo open dataset. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    2020

  31. [39]

    Argoverse: 3d tracking and forecasting with rich maps

    Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 3

    2019

  32. [40]

    BDD100K: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    2020

  33. [41]

    Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset

    Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Internat...

    2021

  34. [42]

    DriveLM: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beiswenger, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with graph visual question answering. In Proc. European Conf. on Computer Vision (ECCV), 2024. 3

    2024

  35. [43]

    WOMD-Reasoning: A large-scale language dataset for interaction and driving intentions reasoning

    Yiheng Li, Chongjian Ge, Chenran Li, Chenfeng Xu, Masayoshi Tomizuka, Chen Tang, Mingyu Ding, and Wei Zhan. WOMD-Reasoning: A large-scale language dataset for interaction and driving intentions reasoning. arXiv:2407.04281, 2024. 3

    2024 arXiv

  36. [44]

    Textual explanations for self-driving vehicles

    Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny. Textual explanations for self-driving vehicles. In Proc. European Conf. on Computer Vision (ECCV), 2018. 3

    2018

  37. [45]

    Explainable object-induced action decision for autonomous vehicles

    Yiran Xu and et al. Explainable object-induced action decision for autonomous vehicles. In Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  38. [46]

    Rea- son2Drive: Towards interpretable and chain-based reasoning for autonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Rea- son2Drive: Towards interpretable and chain-based reasoning for autonomous driving. In Proc. European Conf. on Computer Vision (ECCV), 2024. 3

    2024

  39. [47]

    Shamma, Michael Bernstein, and Li Fei-Fei

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Jack Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Interna...

    2017

  40. [48]

    Choy, and Li Fei-Fei

    Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  41. [49]

    Neural motif networks for visual scene graph generation

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motif networks for visual scene graph generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3

    2018

  42. [50]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. arXiv preprint arXiv:2311.17076, 2023. 3

    2023 arXiv

  43. [51]

    Prism-0: A predicate-rich scene graph generation framework for zero-shot open-vocabulary tasks

    Abdelrahman Elskhawy, Mengze Li, Nassir Navab, and Benjamin Busam. Prism-0: A predicate-rich scene graph generation framework for zero-shot open-vocabulary tasks. arXiv preprint arXiv:2504.00844, 2025. 3

    2025

  44. [52]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. In NeurIPS 2024, 2024. 3

    2024

  45. [53]

    From pixels to graphs: Open- vocabulary scene graph generation with vision-language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open- vocabulary scene graph generation with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  46. [54]

    Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention (ovsgtr)

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention (ovsgtr). In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 3

    2024

  47. [55]

    Egtr: Extracting graph from transformer for scene graph generation

    Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3 12

    2024

  48. [56]

    Relation-aware hierarchical prompt for open- vocabulary scene graph generation

    Tao Liu, Rongjie Li, Chongyu Wang, and Xuming He. Relation-aware hierarchical prompt for open- vocabulary scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 5576–5584, 2025. 3

    2025

  49. [57]

    End-to-end open-vocabulary video visual relationship detection using multi-modal prompting

    Yongqi Wang, Xinxiao Wu, Shuo Yang, and Jiebo Luo. End-to-end open-vocabulary video visual relationship detection using multi-modal prompting. arXiv preprint arXiv:2409.12499, 2024. 3

    2024 arXiv

  50. [58]

    Dada-2000: Can driving acci- dent be predicted by driver attentionƒ analyzed by a benchmark

    Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, He Wang, and Sen Li. Dada-2000: Can driving acci- dent be predicted by driver attentionƒ analyzed by a benchmark. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 4303–4309. IEEE, 2019. 3

    2000

  51. [59]

    Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving

    Tianqi Wang, Sukmin Kim, Ji Wenxuan, Enze Xie, Chongjian Ge, Junsong Chen, Zhenguo Li, and Ping Luo. Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5599–560...

    2024

  52. [60]

    Uncertainty-based traffic accident anticipation with spatio-temporal relational learning

    Wentao Bao, Qi Yu, and Yu Kong. Uncertainty-based traffic accident anticipation with spatio-temporal relational learning. In Proceedings of the 28th ACM International Conference on Multimedia , pages 2682–2690, 2020. 3

    2020

  53. [61]

    Yolov8: Open-source object detection model

    Glenn Jocher et al. Yolov8: Open-source object detection model. https://github.com/ultralytics/ ultralytics, 2023. Accessed: 2025-05-10. 3, 4, 14

    2023

  54. [62]

    Bot-sort: Robust associations multi-pedestrian tracking

    Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651, 2022. 4, 14

    2022 arXiv

  55. [63]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): 1137–1149, 2016. 4, 14

    2016

  56. [64]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024. 5, 17

    2024

  57. [65]

    Naturalbench: Evaluating vision-language models on natural adversarial samples

    Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models on natural adversarial samples. arXiv preprint arXiv:2410.14669, 2024. 6

    arXiv

  58. [66]

    sample_n

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 6 13 A Dataset details A.1 Original DRAMA dataset The DRAMA (Driving Risk Assessment Mechanism with A captioning module) ...

    2023

  59. [67]

    The special requirement is that you must include every pedestrian and cyclist separately and not group them as people or cyclists

    Objects in the frame. The special requirement is that you must include every pedestrian and cyclist separately and not group them as people or cyclists

  60. [68]

    Object attribute should include the state of the object eg

    Object attributes inside object dictionary that are relevant to answering the question. Object attribute should include the state of the object eg. moving or static, description of the object such as color, orientation, etc

  61. [69]

    These should be with respect to the original image dimensions

    Object bounding boxes. These should be with respect to the original image dimensions

  62. [70]

    Objects": {

    Object relationships between objects. This should be detailed of upto 4 words. Limit your response to only at most 5 most relevant objects in the scene. An example structure would look like this: { "Objects": { "name_of_object": { "attributes": [], "bounding_box": [] } }, "Rel...

  63. [71]

    All objects from the scene graph

  64. [72]

    goes to the left

    Predicted intent for every object. Intent should be one of these values: 2.1 Lateral (Sideways) Intent Options (choose one): - “goes to the left” - “goes to the right” 2.2 Vertical Intent Options (choose one): - “moves away from ego vehicle” - “moves towards ego vehicle” - “st...

  65. [73]

    Reason for this prediction

  66. [74]

    name_of_object

    Bounding box of the object, these should be with respect to orginal image dimensions. An example structure (dictionary, not a list): { "name_of_object": { "Intent": ["predicted lateral intent", "predicted vertical intent"], "Reason": "reason for this prediction", "Bounding_box...

  67. [75]

    goes to the left

    Predicted intent for every object. Intent should be one of these values: 2.1 Lateral (Sideways) Intent Options (has to be from these two options): - “goes to the left” - “goes to the right” 2.2 Vertical Intent Options: -“moves away from ego vehicle” - “moves towards ego vehicl...

  68. [76]

    Risk is defined as a hazardous scenario that poses danger to the ego vehicle

    Risk score for this prediction (Yes or No). Risk is defined as a hazardous scenario that poses danger to the ego vehicle

  69. [77]

    these should be with respect to orginal image dimensions

    Bounding box of each object. these should be with respect to orginal image dimensions

  70. [78]

    Risk": "Yes/No

    Suggested action given the scene and risk score. An example structure would look like this: { "Risk": "Yes/No", "Suggested_action": "suggested action for ego vehicle", "pedestrian": { "Intent": ["predicted lateral intent", "predicted vertical intent"], "Reason": "reason for th...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.