REVIEW 3 major objections 4 minor 16 references
A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Large vision-language models can match or beat a standard YOLO detector in recall on degraded traffic scenes, while the conventional detector keeps an edge in geometric box precision.
desk verdict Useful zero-shot LVLM benchmark on PeSOTIF, but the YOLOv5 baseline protocol is under-specified and the arXiv abstract overclaims RT-DETRv4—addressable, but currently conditional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is visual grounding through prompted coordinate reading: each image is resized with aspect ratio preserved, framed with a 2-pixel border, and overlaid with ruler ticks every 10% of width and height; a four-part chain-of-thought prompt asks the LVLM to output JSON bounding boxes in normalized [0,1] coordinates. The ruler gives the model a direct spatial reference, and the JSON schema makes outputs machine-parseable, so purely semantic models become measurable detectors on the PeSOTIF benchmark.
What would settle it
Take the PeSOTIF natural subset, give the YOLOv5 baseline the identical ruler-augmented and border-framed images used for the LVLMs, and retrain or re-evaluate it on the same splits with matched hyperparameters. If YOLOv5's recall then reaches or exceeds the LVLMs' recall, the claimed over-25% semantic-recall advantage would be shown to be an artifact of the comparison protocol rather than a property of LVLMs.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that zero-shot visual grounding by modern LVLMs is a viable quantitative perception path under SOTIF conditions: top models convert semantic understanding into normalized bounding-box coordinates well enough to exceed a production YOLOv5 detector on recall in real-world degradation (over 25% relative margin) and to approach a transformer-based detector, while remaining inferior in exact localization for hand-crafted perturbations. The paper interprets this as semantic reasoning being more robust to degradation of local visual features, and geometric regression being more precise when the image itself is clean and synthetic. The stated practical cons
Load-bearing premise
The comparison assumes YOLOv5 60e was trained and evaluated under the same protocol as the LVLMs — including whether it saw the same resized, border-annotated, ruler-augmented images and which data split was used for training — but the paper leaves these details unspecified, so the measured recall gap could be inflated by a mismatch in preprocessing or training data.
Editorial extensions
If this is right
- If the recall advantage holds, LVLMs can act as an independent high-recall perception branch that flags objects (rare vehicles, anomalies) that a geometric detector like YOLOv5 misses in rain, snow, fog, and glare.
- A hybrid perception stack — LVLM proposals for coverage, YOLO for precise boxes — would combine the two strengths measured here, which the authors explicitly suggest.
- Because LVLMs were used zero-shot without fine-tuning, the measured semantic recall is a lower bound of what task-specific adaptation could achieve; the paper's prompting approach provides a testable recipe.
- The large latency gap (5 s to over 50 s per image) means LVLMs cannot meet real-time automated-driving constraints today, so deployment must be as a validation layer or offline analysis tool rather than the primary detector.
- Ranking by parameter count is not predictive: a 5B model (Gemini 2.5 Flash) beats trillion-parameter models on mAP50, suggesting architecture and training matter more than scale for spatial grounding.
Reading between the lines
- A fairer head-to-head would feed the YOLOv5 baseline the same ruler-annotated, border-framed images used for the LVLMs; the paper does not specify baseline preprocessing, so part of the 25% recall margin could be an artifact of input mismatch rather than model reasoning. This is my inference, not a claim in the paper.
- The ruler-tick prompt essentially turns an object-detection problem into a coordinate-reading task; the same trick could be tested on other visual grounding benchmarks to see if the recall boost generalizes beyond traffic scenes.
- If the trade-off is structural, then fine-tuning a small LVLM with geometric supervision (box regression heads) might close the precision gap while keeping semantic recall — a testable architecture the paper does not evaluate.
- The paper's safety-validator framing implies a system-level design where LVLM detections with low confidence but high recall trigger a conservative response (e.g., slow down); a concrete next step is to measure how often such triggers are true positives versus false alarms in real driving logs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical benchmark of ten large vision-language models (LVLMs) for 2D object detection on the PeSOTIF dataset, which targets long-tail traffic scenes and adverse weather/perturbation conditions. The authors design a visual-prompting pipeline that adds a border and ruler ticks to images and asks LVLMs to output JSON-formatted bounding boxes. They compare the LVLMs against a YOLOv5 60e baseline using COCO-style mAP50, mAR50, and mmAP50:95. The central claim is a trade-off: top LVLMs such as Gemini 3 and Doubao achieve substantially higher recall than YOLOv5 in natural degraded scenes, while YOLOv5 retains better geometric precision on handcrafted perturbations. The paper also reports inference-time differences among LVLMs and suggests using LVLMs as high-level safety validators.
Significance. If the comparison is valid, the paper would provide a useful quantitative reference for deploying LVLMs as complementary semantic detectors in SOTIF-oriented automated driving. The work has concrete strengths: it uses a dedicated SOTIF benchmark (PeSOTIF), evaluates a broad set of ten current LVLMs, adopts standard COCO metrics, includes qualitative examples, and releases code. The proposed trade-off between semantic recall and geometric precision is potentially actionable for hybrid perception architectures. However, the central comparison is currently under-specified: the YOLOv5 baseline training and evaluation protocol is not described, and the abstract mentions RT-DETRv4 while the full text never reports RT-DETRv4 results. These gaps affect the credibility of the headline quantitative claims.
major comments (3)
- [Section II.E and Figure 2] The YOLOv5 60e baseline protocol is not specified. The paper does not state whether YOLOv5 was trained on PeSOTIF, which split was used, what hyperparameters or training schedule define '60e', or whether the baseline received the same preprocessing as the LVLMs. Section II.B describes adding a border and ruler ticks to LVLM inputs; if YOLOv5 was evaluated on raw images while LVLMs received ruler-augmented inputs, the reported recall advantage is confounded because the ruler provides an explicit spatial reference designed to improve coordinate estimation. Since the headline claim that LVLMs exceed YOLOv5 recall by over 25% in natural subsets depends entirely on this comparison, the baseline details must be provided or the comparison must be re-run under matched preprocessing.
- [Abstract vs. full text] The abstract states that performance is compared against 'the anchor-based YOLOv5 and the transformer-based RT-DETRv4,' and claims top LVLMs 'closely match RT-DETRv4 under natural visual degradation.' However, the full text never reports any RT-DETRv4 results: Section II.E introduces only YOLOv5, and all result figures compare only YOLOv5 60e. This is a load-bearing inconsistency because the paper's framing as a comparison against 'specialized detectors' is part of its central contribution. Either add the RT-DETRv4 experiments and results, or remove all RT-DETRv4 mentions from the abstract and introduction.
- [Section III, Figs. 6–10] No uncertainty quantification is reported. The LVLM outputs come from API-based models with stochastic decoding, yet every figure appears to show a single run with no error bars, confidence intervals, or repeated-sampling statistics. Many between-model differences are small (e.g., Gemini 2.5 Flash versus larger models in Fig. 6), and without repeated runs the reader cannot tell whether the reported ordering is stable. The central quantitative claims should be accompanied by at least three repeated runs per model and a variance estimate, or a clear statement that outputs are deterministic and a justification of that assumption.
minor comments (4)
- [Section II.E / Fig. 11] Inference time is reported only for LVLMs; there is no YOLOv5 latency baseline, so the 'real-time gap' discussed in Section IV is not quantified. Including a YOLOv5 inference-time number would make the trade-off discussion concrete.
- [Fig. 2 and Figs. 6–10] The figure legends use circled numbers (①②…⑩) to denote models and (❶❷❸) to denote metrics. The captions are understandable, but the figures themselves do not show numeric values on the axes, making it hard to read exact mAP/mAR figures from the plots. Consider adding value labels or a table of the exact numbers.
- [Prompt output field] The prompt asks LVLMs to output a 'safety critical' indicator, but the paper never analyzes or evaluates this field. Either report on its accuracy/utility or remove it from the prompt and stated contribution, since it is currently an unused output.
- [References] Reference [2] contains a formatting inconsistency ('ISO 21 448:2022' with a space). Also, the reference list uses mixed capitalization for some titles; a consistent style would improve readability.
Circularity Check
No circularity found: the paper is an empirical benchmark with standard external metrics and no fitted-parameter predictions.
full rationale
The paper makes no derivation claim that reduces to its inputs. It reports measured COCO-style mAP/mAR/mmAP metrics computed against the externally labeled PeSOTIF ground truth (Eqs. 2–4), and the LVLM outputs are parsed from model responses rather than fitted to the evaluation labels. No parameter is fitted to a subset and then renamed as a prediction, and no equation is defined in terms of the quantity it claims to establish. The preprocessing and ruler prompting are protocol choices that may affect measured performance, but they do not make the result true by construction. Citations to PeSOTIF and YOLO-related work are not self-citations by the present authors and are not used to forbid alternatives. The skeptic's concern about the unspecified YOLOv5 training/evaluation protocol and the abstract's mention of RT-DETRv4 absent from the full text are questions of experimental fairness and completeness, not circularity. These issues belong under correctness risk, not under the circularity score, which remains 0.
Assumptions & free parameters
assumptions (5)
- domain assumption PeSOTIF ground-truth annotations are correct and complete.
- domain assumption Visual border and ruler-tick preprocessing does not systematically bias the LVLM comparison.
- domain assumption YOLOv5 60e is a meaningful, comparably trained baseline.
- domain assumption Single API inference per image is representative of LVLM performance.
- domain assumption The 11-class prompt taxonomy aligns with PeSOTIF's label set.
Cite this review
Pith. "Pith review of A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions." pith.science (2026). https://pith.science/paper/BMLSSUEJ
@misc{pith2026260122830,
author = {Pith},
title = {Pith review of: A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions},
year = {2026},
howpublished = {\url{https://pith.science/paper/BMLSSUEJ}},
note = {Machine review of arXiv:2601.22830}
}
read the original abstract
Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against two specialized detectors: the anchor-based YOLOv5 and the transformer-based RT-DETRv4. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3) surpass the YOLOv5 in recall by over 25% and closely match RT-DETRv4 under natural visual degradation, while specialized detectors retain an advantage in geometric precision for handcrafted perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
A review of sensor technologies for perception in automated driving,
E. Marti, M. A. de Miguel, F. Garcia, and J. Perez, “A review of sensor technologies for perception in automated driving,”IEEE Intelligent Transportation Systems Magazine, vol. 11, no. 4, pp. 94–108, 2019
2019
-
[2]
ISO 21 448:2022, Jun
International Organization for Standardization,Road vehicles — Safety of the intended functionality, Std. ISO 21 448:2022, Jun. 2022, published 2022-06; 181 pages
2022
-
[3]
PeSOTIF: A challenging visual dataset for perception SOTIF problems in long-tail traffic scenarios,
L. Peng, J. Li, W. Shao, and H. Wang, “PeSOTIF: A challenging visual dataset for perception SOTIF problems in long-tail traffic scenarios,” in 2023 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2023, pp. 1–8
2023
-
[4]
Are we ready for autonomous driving? the KITTI vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the KITTI vision benchmark suite,” in2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354–3361
2012
-
[5]
nuScenes: A multi- modal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuScenes: A multi- modal dataset for autonomous driving,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
2020
-
[6]
Bdd100k: A diverse driving dataset for heterogeneous multitask learning,
F. Yu, H. Chen, X. Wang, W. Xian, Y . Chen, F. Liu, V . Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
2020
-
[7]
Uncertainty evaluation of object detection algorithms for autonomous vehicles,
L. Peng, H. Wang, and J. Li, “Uncertainty evaluation of object detection algorithms for autonomous vehicles,”Automotive Innovation, vol. 4, no. 3, pp. 241–252, 2021
2021
-
[8]
Ensuring SOTIF: Enhanced object detection techniques for autonomous driving,
S. Wang, Z. Wang, S. Hong, P. Wang, and S. Zhang, “Ensuring SOTIF: Enhanced object detection techniques for autonomous driving,”Accident Analysis & Prevention, vol. 218, p. 108094, 2025
2025
Show all 16 references
-
[9]
You only look once: Unified, real-time object detection,
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
2016
-
[10]
Semantic understanding of traffic scenes with large vision language models,
S. Jain, S. Thapa, K.-T. Chen, A. L. Abbott, and A. Sarkar, “Semantic understanding of traffic scenes with large vision language models,” in 2024 IEEE Intelligent Vehicles Symposium (IV), 2024, pp. 1580–1587
2024
-
[11]
DriveSOTIF: Advancing SOTIF through multimodal large language models,
S. Huang, F. Shi, C. Sun, J. Zhong, M. Ning, Y . Yang, Y . Lu, H. Wang, and A. Khajepour, “DriveSOTIF: Advancing SOTIF through multimodal large language models,”IEEE Transactions on Vehicular Technology, 2025
2025
-
[12]
LLM-DiffAug: Enhancing few-shot object detection via LLM-guided diffusion augmentation,
Y . Jiang, S. Qiang, W. Li, and Y . Liang, “LLM-DiffAug: Enhancing few-shot object detection via LLM-guided diffusion augmentation,” Knowledge-Based Systems, vol. 326, p. 114066, 2025
2025
-
[13]
Filters for common resampling tasks,
K. Turkowski, “Filters for common resampling tasks,” inGraphics gems, 1990, pp. 147–165
1990
-
[14]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, an...
2022
-
[15]
Yolo evolution: A comprehensive benchmark and architectural review of yolov12, yolo11, and their previous versions,
N. Jegham, C. Y . Koh, M. Abdelatti, and A. Hendawi, “Yolo evolution: A comprehensive benchmark and architectural review of yolov12, yolo11, and their previous versions,” 2025. [Online]. Available: https://arxiv.org/abs/2411.00201
2025 arXiv
-
[16]
Microsoft COCO: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” inComputer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014...
2014
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.