Pith. sign in

REVIEW 5 major objections 6 minor 74 references

A dedicated traffic-surveillance multimodal dataset lifts fine-tuned vision-language models from roughly 0.5 to above 0.9 average score.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:28 UTC pith:6NUQX72X

load-bearing objection MITS is a genuinely useful dataset resource, but the reported gains are inflated by an insufficiently split test set and in-distribution evaluation; it deserves a careful referee, not acceptance as-is. the 5 major comments →

arxiv 2509.09730 v1 pith:6NUQX72X submitted 2025-09-10 cs.CV cs.AI

MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance

Kaikai Zhao , Zhaoxiang Liu , Peng Wang , Xin Wang , Zhicheng Ma , Yajun Xu , Wenjing Zhang , Yibing Nan
show 2 more authors
Kai Wang Shiguo Lian
This is my paper
classification cs.CV cs.AI
keywords intelligent traffic surveillancelarge multimodal modelsvisual question answeringinstruction tuningtraffic benchmark datasetobject countingobject localizationdomain adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that general large multimodal models underperform on intelligent traffic surveillance because no dedicated multimodal dataset exists, and that this gap is fixable with data rather than new architectures. It introduces MITS, a dataset of 170,400 real surveillance images labeled for eight categories and 24 subcategories, paired with 5 million visual question-answer items spanning recognition, counting, localization, background analysis, and event reasoning. Fine-tuning LLaVA and Qwen models on MITS raises their average test scores from roughly 0.49-0.73 to about 0.90-0.93. The authors present this as evidence that the main missing ingredient for ITS applications is domain-specific instruction data, not model design or scale. If correct, MITS is a practical resource for building queryable traffic-camera systems and a template for other vertical domains.

Core claim

MITS is a large-scale multimodal benchmark built from four years of desensitized real-world traffic-camera footage. Around each image the authors construct a hierarchy of annotations: detection and classification labels for eight main categories and 24 subcategories of objects and events; human-generated question templates whose answers are derived automatically from those verified labels; GPT-4o-generated background-analysis and event-reasoning QA pairs that pass expert review; and an optimized caption that merges the base caption, human QAs, and verified LLM QAs. Fine-tuning four mainstream LMM families—LLaVA-1.5, LLaVA-1.6, Qwen2-VL, and Qwen2.5-VL—on this data improves average test score

What carries the argument

The carrying mechanism is a human-in-the-loop multimodal data-generation pipeline. Object and event labels are produced by an iterative YOLOv8-assisted annotation loop with human validation; question templates then generate recognition, counting, and localization QA pairs whose answers are exact functions of the verified labels, and GPT-4o generates background and reasoning QAs that experts review. An optimized-caption step fuses all verified information into one dense caption, and the models are fine-tuned with LoRA on the captions and QA pairs. Balanced positive and negative sampling per category is what makes the counting and localization answers trustworthy enough to serve as training si

Load-bearing premise

The benchmark's answers—both the object labels and the question-answer pairs—are assumed to be correct and complete because they were produced by a machine-assisted loop and expert review rather than measured for error rate; if those labels contain systematic mistakes, the reported scores overstate how well the models actually understand traffic scenes.

What would settle it

Take a random sample of MITS test images, have independent experts annotate the same counting and localization questions without seeing MITS labels, and compare their answers to the benchmark's ground truth; if agreement is materially below the fine-tuned models' scores, the ground truth is too lenient and the reported improvements are inflated. A second check: re-evaluate the fine-tuned models on a held-out set of traffic-camera images from a different region or camera system to test whether the gains generalize beyond the dataset's collection sites.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Fine-tuned LMMs can act as queryable traffic-surveillance assistants, answering natural-language questions about camera feeds for monitoring and incident management.
  • Because counting and localization are the most improved tasks, downstream applications such as vehicle counts and incident location can plausibly rely on LMM output rather than separate small detection models.
  • Optimized captions that fuse verified foreground labels with reviewed background descriptions improve downstream performance more than base captions alone, pointing to data quality as a key lever.
  • The consistent gains across five model families suggest the approach transfers across architectures, so the dataset itself is the reusable asset.
  • Releasing the dataset, code, and fine-tuned models lets other groups reproduce the benchmark and extend it to new cameras or regions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported gains are measured against MITS's own ground truth; a test with independently annotated or human-expert baselines would clarify how much of the remaining gap is label noise versus model error.
  • Because all images are single-frame and mostly from fixed third-person cameras, the benchmark does not test temporal reasoning; extending MITS to multi-frame sequences would likely be needed for accident prediction or intent analysis.
  • The category taxonomy was chosen from the authors' ITS project experience; other deployments with different camera placements or legal regimes may need new categories, and the pipeline rather than the exact label set is the reusable part.
  • The 98% human-agreement check on the scoring model suggests automatic evaluation is workable, but that agreement was measured on 5,000 QA pairs and should be re-examined per task because background and reasoning answers are open-ended.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MITS, a large-scale multimodal benchmark for Intelligent Traffic Surveillance, comprising 170,400 real-world surveillance images from roughly 1,100 cameras, annotated with 8 main categories and 24 subcategories, together with image captions and more than 5 million VQA pairs spanning recognition, counting, localization, background analysis, and event reasoning. The dataset is constructed through a hybrid pipeline: YOLOv8-assisted human-validated object/event annotations, template-based human-generated QAs derived from those annotations, and GPT-4o-generated background/reasoning QAs refined by expert review. To validate MITS, the authors fine-tune several general LMMs (LLaVA-1.5/1.6, Qwen2-VL, Qwen2.5-VL) on the training set and report large test-set gains (27%–83% relative). The central claim of the paper is that MITS significantly improves LMM performance for ITS applications.

Significance. If the reported improvements are real and reflect generalization to unseen ITS scenes, MITS would be a valuable contribution: it is large-scale, sourced from real surveillance cameras, includes privacy desensitization, and is released openly with code and models. The five-task organization and the human-in-the-loop annotation process are sensible. However, the evaluation currently has load-bearing weaknesses: the train/test split is not demonstrated to be independent at the source-video/camera level, there is no control condition using non-ITS instruction data, and the test questions are generated by the same pipeline as the training questions. These issues undermine the strength of the central claim until addressed.

major comments (5)
  1. [§3.3.1, §3.1, Table 4]
  2. [§4.1.1, Table 4]
  3. [§3.2.3, §3.4, §4.2.1]
  4. [§3.2.1, Table 4]
  5. [§4.1.2, Table 4]
minor comments (6)
  1. [Table 4] The table layout is confusing: average scores are concatenated with the next cell, e.g., "0.8840.905(+83.2%)" and "0.8860.921(+35.8%)". Please add separators and clarify which column is the average.
  2. [Table 1, §3.2.3 prompt] There are typos: "suverillance" should be "surveillance" in Table 1 and in the GPT-4o prompt text.
  3. [References [67]] Reference [67] cites the ultralytics/yolov5 repository, but the text refers to YOLOv8. Please cite the correct YOLOv8 source.
  4. [§3.3.1] The test-set description should report the exact number of test images per category, whether balancing was done independently per category, and whether any source-camera/corridor constraints were used. Currently only "uniformly sampled 10%" is stated.
  5. [§4.2.3, Figures 7–8] The figure captions should state which model variant is shown (e.g., Qwen2.5-VL-7B) and clearly distinguish ground-truth boxes from model predictions.
  6. [§1, Abstract] The percentage gains in the abstract are relative improvements; consider reporting absolute deltas as well for clarity (e.g., 0.494→0.905 rather than only +83.2%).

Circularity Check

0 steps flagged

No significant circularity: the central fine-tuning claim is an empirical evaluation on a held-out test split; self-citations are methodological and not load-bearing.

full rationale

The paper's main claim is that fine-tuning general LMMs on MITS improves their performance on ITS tasks. This is supported by a standard supervised protocol: models are trained on a training subset and evaluated on a test subset sampled from the same dataset (Sec. 3.3.1). The QA pairs used for evaluation are generated by the same templating/GPT-4o pipeline as the training data, so the evaluation is in-distribution, but that is a normal property of benchmark construction and does not make the reported improvement equivalent to the training objective by construction. The recognition/counting/localization ground truths derive from manual plus YOLOv8-assisted annotations, while background/reasoning answers are expert-reviewed; the DeepSeek-R1 scoring is itself human-validated on 5,000 samples. The paper's self-citations ([5], [6], [61], [69]) are to related datasets, a training methodology, and an evaluation methodology; none of them supplies a premise that definitionally forces the central result. A potential issue is that the test split is image-level rather than video- or camera-level, given the images come from ~8,000 recordings at ~21 frames per recording, which could in principle leak near-duplicate frames into training and inflate gains. That is an experimental validity concern, not a circularity: no equation or definition in the paper reduces the test scores to the training fit. Overall, the derivation chain is not circular in the sense required by the review criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claim depends on the reliability of annotations and generated QA data, on the representativeness of the category taxonomy, and on the validity of the LLM-based scoring metric. These are domain assumptions rather than fitted parameters; the paper provides qualitative descriptions but no quantitative validation of label quality.

axioms (4)
  • domain assumption The eight main categories and 24 subcategories cover the most critical ITS objects and events.
    Stated as based on years of project experience (Section 3.2.1) but no formal completeness analysis is provided.
  • domain assumption The YOLOv8-assisted annotation loop with human review produces accurate and complete labels for all images.
    No label error rates or inter-annotator agreement are reported (Section 3.2.1).
  • domain assumption GPT-4o-generated captions and QA pairs, after expert review, are factually reliable.
    Expert review is described but only 1,000 captions and 5,000 QA scores were human-verified (Sections 3.2.3 and 3.4).
  • domain assumption DeepSeek-R1 scores for background and reasoning answers align with human judgment.
    98% agreement on a 5,000-pair sample, but the full test set is scored automatically (Section 3.4).

pith-pipeline@v1.3.0-alltime-deepseek · 18187 in / 11563 out tokens · 110248 ms · 2026-08-04T20:28:45.374343+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance." pith.science (2026). https://pith.science/paper/6NUQX72X

@misc{pith2026250909730,
  author       = {Pith},
  title        = {Pith review of: MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NUQX72X}},
  note         = {Machine review of arXiv:2509.09730}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

General-domain large multimodal models (LMMs) have achieved significant advances in various image-text tasks. However, their performance in the Intelligent Traffic Surveillance (ITS) domain remains limited due to the absence of dedicated multimodal datasets. To address this gap, we introduce MITS (Multimodal Intelligent Traffic Surveillance), the first large-scale multimodal benchmark dataset specifically designed for ITS. MITS includes 170,400 independently collected real-world ITS images sourced from traffic surveillance cameras, annotated with eight main categories and 24 subcategories of ITS-specific objects and events under diverse environmental conditions. Additionally, through a systematic data generation pipeline, we generate high-quality image captions and 5 million instruction-following visual question-answer pairs, addressing five critical ITS tasks: object and event recognition, object counting, object localization, background analysis, and event reasoning. To demonstrate MITS's effectiveness, we fine-tune mainstream LMMs on this dataset, enabling the development of ITS-specific applications. Experimental results show that MITS significantly improves LMM performance in ITS applications, increasing LLaVA-1.5's performance from 0.494 to 0.905 (+83.2%), LLaVA-1.6's from 0.678 to 0.921 (+35.8%), Qwen2-VL's from 0.584 to 0.926 (+58.6%), and Qwen2.5-VL's from 0.732 to 0.930 (+27.0%). We release the dataset, code, and models as open-source, providing high-value resources to advance both ITS and LMM research.

Figures

Figures reproduced from arXiv: 2509.09730 by Kaikai Zhao, Kai Wang, Peng Wang, Shiguo Lian, Wenjing Zhang, Xin Wang, Yajun Xu, Yibing Nan, Zhaoxiang Liu, Zhicheng Ma.

Figure 1
Figure 1. Figure 1: Performance comparison between original and fine-tuned models in ITS. LLaVA [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Base Statistics of the source images. The charts illustrate the distribution of [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An example from MITS dataset, consisting of an image and three groups of QA [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The MITS data generation pipeline. each category Ci , we manually design a set of question templates covering three tasks: Recognition, Counting, and Localization. These templates incor￾porate various query formats, including Yes/No, What, Where, How many, and Which, ensuring diverse and comprehensive question formulations. The generated question set Qi focuses on foreground information specific to Ci and … view at source ↗
Figure 5
Figure 5. Figure 5: (a) Distribution and quantity of images across the 8 main categories. (b) Number [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Statistics of QA pairs in MITS. (a) Number of QAs for each task. (b) Average [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Representative success cases across five tasks. Text in red indicates incorrect [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Representative failure cases across five tasks. Red text highlights erroneous [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 3 canonical work pages

  1. [1]

    Veres, M

    M. Veres, M. Moussa, Deep learning for intelligent transportation sys- tems: A survey of emerging trends, IEEE Transactions on Intelligent Transportation Systems 21 (8) (2020) 3152–3168.doi:10.1109/TITS. 2019.2929020. 2

    arXiv 2020

  2. [2]

    S. Wan, X. Xu, T. Wang, Z. Gu, An intelligent video analysis method for abnormal event detection in intelligent transportation systems, IEEE 23 Transactions on Intelligent Transportation Systems 22 (7) (2021) 4487– 4495.doi:10.1109/TITS.2020.3017505. 2, 5

    arXiv 2021

  3. [3]

    Ashraf, A

    J. Ashraf, A. D. Bakhshi, N. Moustafa, H. Khurshid, A. Javed, A. Be- heshti, Novel deep learning-enabled lstm autoencoder architecture for discovering anomalous events from intelligent transportation systems, IEEE Transactions on Intelligent Transportation Systems 22 (7) (2021) 4507–4518.doi:10.1109/TITS.2020.3017882. 2

    arXiv 2021

  4. [4]

    Y. Tang, C. Zhang, R. Gu, P. Li, B. Yang, Vehicle detection and recog- nition for intelligent traffic surveillance system, Multimedia tools and applications 76 (2017) 5817–5832. 2

    2017

  5. [5]

    Y. Xu, C. Huang, Y. Nan, S. Lian, Tad: A large-scale benchmark for traffic accidents detection from video surveillance, arXiv preprint arXiv:2209.12386 (2022). 2, 4, 5, 6

    Pith/arXiv arXiv 2022

  6. [6]

    Y. Xu, H. Hu, X. Zhu, Y. Nan, K. Wang, Z. Liu, S. Lian, Raod: A benchmark for road abandoned object detection from video surveillance, IEEE Access (2024). 2, 5, 6

    2024

  7. [7]

    Y. Xia, N. Qian, L. Guo, Z. Cai, Cf-solt: Real-time and ac- curate traffic accident detection using correlation filter-based tracking, Image and Vision Computing 152 (2024) 105336. doi:https://doi.org/10.1016/j.imavis.2024.105336. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856240044142, 4

    arXiv 2024

  8. [8]

    H. Pan, S. Guan, X. Zhao, Lvd-yolo: An efficient lightweight vehicle detection model for intelligent transportation systems, Image and Vision Computing 151 (2024) 105276.doi:https: //doi.org/10.1016/j.imavis.2024.105276. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856240038102

    arXiv 2024

  9. [9]

    Z. Yang, L. S. Pun-Cheng, Vehicle detection in intelligent trans- portation systems and its applications under varying environ- ments: A review, Image and Vision Computing 69 (2018) 143–154. doi:https://doi.org/10.1016/j.imavis.2017.09.008. 24 URLhttps://www.sciencedirect.com/science/article/pii/ S02628856173015922

    work page doi:10.1016/j.imavis.2017.09.008 2018

  10. [10]

    E. Lee, D. Kim, Accurate traffic light detection using deep neural net- work with focal regression loss, Image and Vision Computing 87 (2019) 24–36.doi:https://doi.org/10.1016/j.imavis.2019.04.003. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856193005382

    work page doi:10.1016/j.imavis.2019.04.003 2019

  11. [11]

    Faghri, H

    F. Faghri, H. Pouransari, S. Mehta, M. Farajtabar, A. Farhadi, M. Rastegari, O. Tuzel, Reinforce data, multiply impact: Improved model accuracy and robustness with dataset reinforcement, in: Proceed- ings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17032–17043. 2

    2023

  12. [12]

    Y. Wang, H. Jiang, G. Chen, T. Zhang, J. Zhou, Z. Qing, C. Wang, W. Zhao, Efficient and robust multi-camera 3d object detection in bird-eye-view, Image and Vision Computing 154 (2025) 105428. doi:https://doi.org/10.1016/j.imavis.2025.105428. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250001622

    arXiv 2025

  13. [13]

    D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304.10592 (2023). 2, 7

    Pith/arXiv arXiv 2023

  14. [14]

    H. Liu, C. Li, Q. Wu, Y. J. Lee, Visual instruction tuning, Advances in neural information processing systems 36 (2024). 2, 7, 16

    2024

  15. [15]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou, Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, arXiv preprint arXiv:2308.12966 (2023). 2

    Pith/arXiv arXiv 2023

  16. [16]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, J. Lin, Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, arXiv preprint arXiv:2409.12191 (2024). 2, 7 25

    Pith/arXiv arXiv 2024

  17. [17]

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, J. Gao, Llava-med: Training a large language-and-vision as- sistant for biomedicine in one day, Advances in Neural Information Pro- cessing Systems 36 (2024). 2, 3, 7, 17

    2024

  18. [18]

    F. Bai, Y. Du, T. Huang, M. Q.-H. Meng, B. Zhao, M3d: Advancing 3d medical image analysis with multi-modal large language models, arXiv preprint arXiv:2404.00578 (2024). 2, 7

    Pith/arXiv arXiv 2024

  19. [19]

    A.-M.Marcu, L.Chen, J.Hünermann, A.Karnsund, B.Hanotte, P.Chi- dananda, S.Nair, V.Badrinarayanan, A.Kendall, J.Shotton, etal., Lin- goqa: Video question answering for autonomous driving, arXiv preprint arXiv:2312.14115 (2023). 2, 5, 6

    Pith/arXiv arXiv 2023

  20. [20]

    X. Cao, T. Zhou, Y. Ma, W. Ye, C. Cui, K. Tang, Z. Cao, K. Liang, Z. Wang, J. M. Rehg, et al., Maplm: A real-world large-scale vision- language benchmark for map and traffic scene understanding, in: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 21819–21830. 2, 3, 5

    2024

  21. [21]

    L. Zha, C. Gong, K. Lv, Real-time localization and navigation method for autonomous vehicles based on multi-modal data fusion by integrating memory transformer and ddqn, Image and Vision Computing 156 (2025) 105484.doi:https://doi.org/10.1016/j.imavis.2025.105484. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250007212

    arXiv 2025

  22. [22]

    14615–14624

    K.Zhu, Y.Wang, Y.Sun, Q.Chen, J.Liu, G.Zhang, J.Wang, Continual sft matches multimodal rlhf with negative supervision, in: Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 14615–14624. 2

    2025

  23. [23]

    Y. Sun, J. Chen, S. Zhang, X. Zhang, Q. Chen, G. Zhang, E. Ding, J. Wang, Z. Li, Vrp-sam: Sam with visual reference prompt, in: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 23565–23574. 2

    2024

  24. [24]

    Y. Sun, Q. Chen, X. He, J. Wang, H. Feng, J. Han, E. Ding, J. Cheng, Z. Li, J. Wang, Singular value fine-tuning: Few-shot segmentation re- quires few-parameters fine-tuning, Advances in neural information pro- cessing systems 35 (2022) 37484–37496. 2 26

    2022

  25. [25]

    W. Tang, Y. Sun, Q. Gu, Z. Li, Visual position prompt for mllm based visual grounding, arXiv preprint arXiv:2503.15426 (2025). 2

    Pith/arXiv arXiv 2025

  26. [26]

    W. Tang, L. Li, X. Liu, L. Jin, J. Tang, Z. Li, Context disentangling and prototype inheriting for robust visual grounding, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (5) (2023) 3213–3229. 2

    2023

  27. [27]

    Y. Sun, Q. Chen, J. Wang, J. Wang, Z. Li, Exploring effective factors for improving visual in-context learning, IEEE Transactions on Image Processing (2025). 2

    2025

  28. [28]

    X. Li, X. Liu, L. Qi, J. Dong, Weakly supervised camou- flaged object detection based on the sam model and mask guidance, Image and Vision Computing 159 (2025) 105571. doi:https://doi.org/10.1016/j.imavis.2025.105571. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250015932

    arXiv 2025

  29. [29]

    Paulraj, S

    S. Paulraj, S. Vairavasundaram, M2vad: Multiview multi- modality transformer-based weakly supervised video anomaly detection, Image and Vision Computing 149 (2024) 105139. doi:https://doi.org/10.1016/j.imavis.2024.105139. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856240024392

    arXiv 2024

  30. [30]

    X. Hong, T. Adam, M. Ghazali, Uhdnet: Unified multimodal fusion harmonization and hierarchical dependency learning for visible-infrared person re-identification, Image and Vision Computing 161 (2025) 105628.doi:https://doi.org/10.1016/j.imavis.2025.105628. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250021612

    arXiv 2025

  31. [31]

    Zhang, Y

    J. Zhang, Y. Yu, Y. Mao, Y. Ren, Event-level multimodal feature fusion for audio–visual event localization, Image and Vision Computing 161 (2025) 105610.doi:https://doi.org/10.1016/j.imavis.2025. 105610. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250019822 27

    work page doi:10.1016/j.imavis.2025 2025

  32. [32]

    M. Li, Z. Zhu, K. Li, L. Zhou, Z. Zhao, H. Pei, Joint train- ing strategy of unimodal and multimodal for multimodal senti- ment analysis, Image and Vision Computing 149 (2024) 105172. doi:https://doi.org/10.1016/j.imavis.2024.105172. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856240027742

    arXiv 2024

  33. [33]

    Z. Wang, W. Wang, N. Li, S. Zhang, Q. Chen, Z. Jiang, Multimodal parallel attention network for medical image seg- mentation, Image and Vision Computing 147 (2024) 105069. doi:https://doi.org/10.1016/j.imavis.2024.105069. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856240017322

    arXiv 2024

  34. [34]

    Guarrasi, F

    V. Guarrasi, F. Aksu, C. M. Caruso, F. Di Feola, A. Ro- fena, F. Ruffini, P. Soda, A systematic review of interme- diate fusion in multimodal deep learning for biomedical ap- plications, Image and Vision Computing 158 (2025) 105509. doi:https://doi.org/10.1016/j.imavis.2025.105509. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250009762

    arXiv 2025

  35. [35]

    Jiang, M

    B. Jiang, M. Liao, Y. Zhao, G. Li, S. Cheng, X. Wang, Q. Xia, Deep learning for brain tumor segmentation in multimodal mri images: A re- view of methods and advances, Image and Vision Computing 156 (2025) 105463.doi:https://doi.org/10.1016/j.imavis.2025.105463. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250005142

    arXiv 2025

  36. [36]

    S. R. Pulari, M. Umadevi, S. K. Vasudevan, Optimizing multimodal personalized disease prediction accuracy using generated prompts and large language models, Image and Vision Computing 161 (2025) 105649. doi:https://doi.org/10.1016/j.imavis.2025.105649. URLhttps://www.sciencedirect.com/science/article/pii/ S02628856250023792

    arXiv 2025

  37. [37]

    L. Xu, H. Huang, J. Liu, Sutd-trafficqa: A question answering bench- mark and an efficient network for video reasoning over traffic events, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 9878–9888. 5, 6, 11 28

    2021

  38. [38]

    Q. Kong, Y. Kawana, R. Saini, A. Kumar, J. Pan, T. Gu, Y. Ozao, B. Opra, D. C. Anastasiu, Y. Sato, N. Kobori, Wts: A pedestrian- centric traffic video dataset for fine-grained spatial-temporal under- standing (2024).arXiv:2407.15350. URLhttps://arxiv.org/abs/2407.153505, 6

    Pith/arXiv arXiv 2024

  39. [39]

    Parikh, D

    C. Parikh, D. Rawat, R. R. T., T. Ghosh, R. K. Sarvadevabhatla, Road- social: A diverse videoqa dataset and benchmark for road event under- standing from social video narratives, in: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025, pp. 19002– 19011. 5, 6

    2025

  40. [40]

    Fang, L.-l

    J. Fang, L.-l. Li, J. Zhou, J. Xiao, H. Yu, C. Lv, J. Xue, T.-S. Chua, Abductive ego-view accident video understanding for safe driving per- ception, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 22030–22040. 5, 6, 11

    2024

  41. [41]

    A. P. Shah, J.-B. Lamare, T. Nguyen-Anh, A. Hauptmann, Cadp: A novel dataset for cctv traffic camera based accident analysis, in: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE, 2018, pp. 1–9. 4

    2018

  42. [42]

    J. Fang, D. Yan, J. Qiao, J. Xue, H. Wang, S. Li, Dada-2000: Can driv- ing accident be predicted by driver attentionƒanalyzed by a benchmark, in: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), IEEE, 2019, pp. 4303–4309. 4

    2000

  43. [43]

    2110–2118

    Z.Zhu, D.Liang, S.Zhang, X.Huang, B.Li, S.Hu, Traffic-signdetection and classification in the wild, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2110–2118. 5

    2016

  44. [44]

    Møgelmose, D

    A. Møgelmose, D. Liu, M. M. Trivedi, Detection of us traffic signs, IEEE Transactions on Intelligent Transportation Systems 16 (6) (2015) 3116–

    2015

  45. [45]

    L. Wen, D. Du, Z. Cai, Z. Lei, M.-C. Chang, H. Qi, J. Lim, M.-H. Yang, S. Lyu, Ua-detrac: A new benchmark and protocol for multi-object detection and tracking, Computer Vision and Image Understanding 193 (2020) 102907. 5 29

    2020

  46. [46]

    Naphade, Z

    M. Naphade, Z. Tang, M.-C. Chang, D. C. Anastasiu, A. Sharma, R. Chellappa, S. Wang, P. Chakraborty, T. Huang, J.-N. Hwang, et al., The 2019 ai city challenge., in: CVPR workshops, Vol. 8, 2019, p. 2. 5

    2019

  47. [47]

    Snyder, M

    C. Snyder, M. Do, Data for streets: A novel camera network dataset for traffic flow, University of Illinois at Urbana-Champaign: Urbana/Cham- paign, IL, USA (2019). 5

    2019

  48. [48]

    Krajewski, J

    R. Krajewski, J. Bock, L. Kloeker, L. Eckstein, The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems, in: 2018 21st interna- tional conference on intelligent transportation systems (ITSC), IEEE, 2018, pp. 2118–2125. 5

    2018

  49. [49]

    F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, T. Darrell, Bdd100k: A diverse driving dataset for heterogeneous multi- tasklearning, in: Proceedings of theIEEE/CVF conferenceon computer vision and pattern recognition, 2020, pp. 2636–2645. 5

    2020

  50. [50]

    G. Tom, M. Mathew, S. Garcia-Bordils, D. Karatzas, C. Jawahar, Read- ing between the lanes: Text videoqa on the road, in: International Conference on Document Analysis and Recognition, Springer, 2023, pp. 137–154. 5

    2023

  51. [51]

    S. Jain, S. Thapa, K.-T. Chen, A. L. Abbott, A. Sarkar, Semantic un- derstanding of traffic scenes with large vision language models, in: 2024 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2024, pp. 1580–1587. 5

    2024

  52. [52]

    Deruyttere, D

    T. Deruyttere, D. Grujicic, M. B. Blaschko, M.-F. Moens, Talk2car: Predicting physical trajectories for natural language commands, Ieee Access 10 (2022) 123809–123834. 5

    2022

  53. [53]

    Zhang, F

    J. Zhang, F. Ilievski, K. Ma, A. Kollaa, J. Francis, A. Oltramari, A study of situational reasoning for traffic understanding, arXiv preprint arXiv:2306.02520 (2023). 5

    Pith/arXiv arXiv 2023

  54. [54]

    T. Qian, J. Chen, L. Zhuo, Y. Jiao, Y.-G. Jiang, Nuscenes-qa: A multi- modal visual question answering benchmark for autonomous driving sce- nario, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, 2024, pp. 4542–4550. 5 30

    2024

  55. [55]

    Malla, C

    S. Malla, C. Choi, I. Dwivedi, J. H. Choi, J. Li, Drama: Joint risk localizationandcaptioningindriving, in: ProceedingsoftheIEEE/CVF winter conference on applications of computer vision, 2023, pp. 1043–

    2023

  56. [56]

    D. Wu, W. Han, T. Wang, Y. Liu, X. Zhang, J. Shen, Language prompt for autonomous driving, arXiv preprint arXiv:2309.04379 (2023). 5

    Pith/arXiv arXiv 2023

  57. [57]

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, H. Li, Drivelm: Driving with graph visual question answering, arXiv preprint arXiv:2312.14150 (2023). 5

    Pith/arXiv arXiv 2023

  58. [58]

    Haotian, L

    L. Haotian, L. Chunyuan, W. Qingyang, Y. jae Lee, Llava: Large lan- guage and vision assistant visual instruction tuning (2023). URLhttps://huggingface.co/llava-hf/llava-1.5-7b-hf7, 18, 19, 20

    2023

  59. [59]

    Qwen Team, Qwen2.5-vl-7b-instruct hugging face model. (2025). URLhttps://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct7, 18, 19, 20

    2025

  60. [60]

    Haotian, L

    L. Haotian, L. Chunyuan, L. Yuheng, L. Bo, Z. Yuanhan, S. Sheng, Y. J. Lee, Llava-next: Improved reasoning, ocr, and world knowledge (2023). URLhttps://huggingface.co/llava-hf/llava-v1. 6-mistral-7b-hf7, 18, 19, 20

    2023

  61. [61]

    X. Liu, Z. Liu, H. Hu, Z. Chen, K. Wang, K. Wang, S. Lian, A mul- timodal benchmark dataset and model for crop disease diagnosis, in: European Conference on Computer Vision, Springer, 2025, pp. 157–170. 7, 17

    2025

  62. [62]

    T. Wei, Z. Chen, Z. Huang, X. Yu, Benchmarking in-the-wild mul- timodal disease recognition and a versatile baseline, arXiv preprint arXiv:2408.03120 (2024). 7

    Pith/arXiv arXiv 2024

  63. [63]

    J. Yan, J. Yu, Hyperlpr3 - high performance license plate recognition framework. (2023). URLhttps://github.com/szad670401/HyperLPR8 31

    2023

  64. [64]

    J. Deng, J. Guo, E. Ververas, I. Kotsia, S. Zafeiriou, Retinaface: Single- shot multi-level face localisation in the wild, in: CVPR, 2020. 8

    2020

  65. [65]

    J. Deng, J. Guo, Insightface: 2d and 3d face analysis project. (2024). URLhttps://github.com/deepinsight/insightface8

    2024

  66. [66]

    Kittinaradorn, Easyocr

    R. Kittinaradorn, Easyocr. (2024). URLhttps://github.com/JaidedAI/EasyOCR8

    2024

  67. [67]

    Michael, TaoXie, J

    G.Jocher, A.Chaurasia, A.Stoken, J.Borovec, NanoCode012, Y.Kwon, K. Michael, TaoXie, J. Fang, imyhxy, Lorna, Z. Yifu, C. Wong, A. V, D. Montes, Z. Wang, C. Fati, J. Nadar, Laughing, UnglvKitDe, V. Sonck, tkianai, yxNONG, P. Skalski, A. Hogan, D. Nair, M. Strobel, M. Jain, ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation (Nov. 2022).do...

    doi:10.5281/zenodo.7347926 2022

  68. [68]

    URLhttps://openai.com/index/hello-gpt-4o/11

    OpenAI, Hello gpt-4o (2024). URLhttps://openai.com/index/hello-gpt-4o/11

    2024

  69. [69]

    S. Lian, K. Zhao, X. Liu, X. Lei, B. Yang, W. Zhang, K. Wang, Z. Liu, Whatisthebestmodel? application-drivenevaluationforlargelanguage models, arXiv preprint arXiv:2406.10307 (2024). 16

    Pith/arXiv arXiv 2024

  70. [70]

    URLhttps://arxiv.org/abs/2501.1294816

    DeepSeek-AI, Deepseek-r1: Incentivizing reasoning capabilityin llms via reinforcement learning (2025).arXiv:2501.12948. URLhttps://arxiv.org/abs/2501.1294816

    Pith/arXiv arXiv 2025

  71. [71]

    Qwen Team, Qwen2-vl-7b-instruct hugging face model. (2024). URLhttps://huggingface.co/Qwen/Qwen2-VL-7B-Instruct18, 19, 20

    2024

  72. [72]

    Qwen Team, Qwen2.5-vl-3b-instruct hugging face model. (2025). URLhttps://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct18, 19, 20

    2025

  73. [73]

    Qwen Team, Qwen2.5-vl-32b-instruct hugging face model. (2025). URLhttps://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct18, 19, 20

    2025

  74. [74]

    AI, Yi vision language model

    01. AI, Yi vision language model. (2025). URLhttps://huggingface.co/01-ai/Yi-VL-34B18, 19, 20 32

    2025