Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Domain Generalization in Autonomous Driving: Evaluating YOLOv8s, RT-DETR, and YOLO-NAS with the ROAD-Almaty Dataset

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that RT-DETR, evaluated without retraining on Kazakhstan's ROAD-Almaty dataset, outperforms YOLOv8s and YOLO-NAS, while all three degrade under stricter IoU and harsh weather.

desk verdict A genuinely new dataset from an underrepresented region, but the central performance table duplicates sequence rows, so RT-DETR's reported advantage is not established. read the letter →

arxiv 2412.12349 v1 pith:CYWP2F5T submitted 2024-12-16 cs.CV

Madiyar Alimov , Temirlan Meiramkhanov This is my paper · ORCID
classification cs.CV
keywords domaingeneralizationobjectdetectionautonomousdrivingROAD-AlmatydatasetYOLOv8sRT-DETRYOLO-NASshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish how well three popular pre-trained object detectors transfer to a driving environment they were not trained on: the streets of Almaty, Kazakhstan. Using a newly built dataset of 1,844 annotated dashcam frames spanning clear, rainy, foggy, and cloudy conditions plus day and night, the authors test YOLOv8s, RT-DETR, and YOLO-NAS with no retraining. They report that RT-DETR reaches an average F1-score of 0.672 at IoU=0.5, ahead of YOLOv8s (0.458) and YOLO-NAS (0.526), and that all models drop roughly 20% when the IoU threshold rises to 0.75. If correct, this makes RT-DETR the safer default among these three for deployment in Central Asian traffic and reinforces the need for geographic diversity in training data.

What carries the argument

The central object is the ROAD-Almaty test set: 1,844 annotated frames drawn from one-minute dashcam recordings in Almaty at 30 fps, sampled to 10 fps, with about 20% of frames reserved for testing and selected to maximize weather and lighting variability. The comparison mechanism is a zero-retraining protocol in which each pre-trained model is scored on the same five test sequences with COCO-style IoU/F1 metrics at thresholds 0.5 and 0.75, so any performance difference is attributed to the architecture's capacity to generalize rather than to adaptation.

What would settle it

Recompute the average F1 scores after removing the duplicated sequences (keep only one of Seq1/Seq4 and one of Seq2/Seq5) and verify whether RT-DETR still outperforms YOLOv8s and YOLO-NAS by approximately 46% and 27%; a rerun on the original non-duplicated frames, with per-frame scores reported, would settle the ranking.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that RT-DETR, a transformer-based detector, generalizes better to the ROAD-Almaty domain than two CNN-based YOLO variants when all three are evaluated without retraining. At IoU=0.5, RT-DETR's average F1 is 0.672 versus 0.458 for YOLOv8s and 0.526 for YOLO-NAS, and the same ordering holds at IoU=0.75 (0.525 versus 0.325 and 0.400). The authors interpret this as evidence that transformer-based feature extraction is more robust to domain shift, while noting that every model degrades under stricter localization and in difficult conditions such as heavy snowfall and low light, which they take as a case for domain adaptation rather than reliance on any single architecture.

Load-bearing premise

The reported averages treat the five test sequences as five independent samples of the same target domain, but Table I lists identical scores for Seq1 and Seq4 and for Seq2 and Seq5, so the headline numbers rest on duplicate data.

Editorial extensions

If this is right

  • If RT-DETR is indeed the most robust of the three without retraining, teams deploying vehicle detection in Central Asia should prefer it over YOLOv8s and YOLO-NAS until local adaptation data are available.
  • The roughly 20% F1 drop when IoU moves from 0.5 to 0.75 implies that strict localization requirements will need domain adaptation even for the best-performing model.
  • The large performance gap in heavy snow and low-light sequences shows that geographic and climatic diversity in training data is a concrete reliability lever for autonomous driving systems.
  • The ROAD-Almaty dataset can serve as a reusable benchmark for measuring how other pre-trained detectors behave in an underrepresented driving domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Table I lists identical scores for Seq1 and Seq4 and for Seq2 and Seq5, the effective test size may be three rather than five, and the reported margins may shrink or change on a deduplicated evaluation.
  • A direct test of the paper's architectural explanation would be to run the same three models on a second, geographically distinct dataset, since the current evidence covers only Almaty.
  • The paper itself notes that its three model references are official documentation rather than peer-reviewed publications, so reproducing the comparison requires exact checkpoint versions and weight sources.
  • The paper's own recommendation implies a testable extension: fine-tune each model on a small sample of local frames and measure whether RT-DETR's lead persists after adaptation or whether the YOLO variants catch up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript evaluates the out-of-the-box object detection performance of YOLOv8s, RT-DETR, and YOLO-NAS on a newly collected driving dataset from Almaty, Kazakhstan (ROAD-Almaty). The authors report IoU and F1 scores at thresholds 0.5 and 0.75 for five test sequences and conclude that RT-DETR generalizes best, with an average F1 of 0.672 at IoU=0.5, outperforming YOLOv8s and YOLO-NAS by roughly 46% and 27%. No retraining or domain adaptation is performed; the paper frames the results as evidence about domain generalization in an underrepresented geographic region.

Significance. If the reported results were reliable, the study would provide a useful empirical data point on off-the-shelf detector robustness in Central Asia and a modest extension of domain-generalization evidence to an underrepresented region. The dataset construction and annotation protocol are described in reasonable detail, and the use of standard COCO-style metrics supports comparability. However, the paper makes no methodological contribution, releases no data or code, and its central quantitative claim is compromised by apparent duplicate rows in the main results table. The significance of the contribution is therefore contingent on a corrected and independently verifiable evaluation.

major comments (4)
  1. [Sec. IV-A, Table I] The entries for Seq1 and Seq4 are identical for every model and every IoU threshold (e.g., RT-DETR at IoU=0.5: 0.890/0.900 for both), and the entries for Seq2 and Seq5 are likewise identical (e.g., 0.692/0.517 for RT-DETR at IoU=0.5). If Seq4 and Seq5 are truly distinct sequences, exact equality across all six model-threshold combinations is implausible; if they are the same sequence listed twice, the reported averages and the headline 46%/27% improvement percentages double-count two of the five sequences. The authors must clarify the sequence definitions, correct or remove the duplicate rows, and recompute all averages and percentages.
  2. [Sec. IV-B] The comparison rests on aggregate F1/IoU values with no measure of uncertainty, no per-sequence frame counts, and no statistical test. With only five sequences (or effectively three unique ones after the duplication issue), the claim that RT-DETR consistently outperforms YOLOv8s and YOLO-NAS is not supported by any variance estimate or significance test. Error bars, per-sequence sample sizes, and a paired comparison (e.g., per-sequence differences) are needed to sustain the ranking.
  3. [Sec. III-B / Sec. V-A] The study is framed as an evaluation of domain generalization, but it measures performance only on the target dataset. Without a matched evaluation of the same models on their source-domain data or on a familiar benchmark such as COCO or BDD100K under the same protocol, the observed scores cannot be attributed to domain shift; they are absolute performance levels. A source-domain baseline is required to quantify the generalization gap and to support the conclusion that geographic diversity is the cause of the performance degradation.
  4. [Sec. III] The manuscript provides no availability statement for the ROAD-Almaty dataset, no sequence definitions, and no evaluation code. Because the paper's entire contribution is empirical and the central table contains an apparent duplication, the results cannot be independently checked or reproduced. The authors should release the per-sequence evaluation results and sequence metadata at minimum, and ideally the dataset and scripts.
minor comments (4)
  1. [References [11], [20]] The reference list is inconsistent with the text: [11] is listed as TrafficPredict but cited as ApolloScape in Section II-C, and [20] is cited as a general UDA reference but is a specific residual transfer network paper. Please reconcile all numeric citations with the bibliography.
  2. [Sec. III-B1] The description of dividing the dataset into training, validation, and testing sets is confusing because the models are not trained on ROAD-Almaty; clarify that the split is used only for selecting evaluation frames or for future adaptation work.
  3. [Data availability] Add a Data Availability statement indicating whether the ROAD-Almaty dataset and the evaluation scripts will be made publicly available, and if so, under what license.
  4. [Sec. II-C, Sec. V-C] There are typographical issues such as 'W AYMO' in Section II-C and 'A Vs' in Section V-C; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is a pure off-the-shelf evaluation with no fitted parameters, no derived equations, and no load-bearing self-citation chain.

full rationale

The paper reports a straightforward empirical benchmark: three externally published object detection models are downloaded pre-trained and evaluated on a newly collected dataset using standard COCO-style IoU and F1 metrics. No parameter is fitted to the ROAD-Almaty data, no quantity is predicted from a model whose constants were set using the target result, and no uniqueness theorem or prior derivation is invoked to force the conclusion that RT-DETR outperforms the YOLO variants. The model architecture citations point to official documentation, but those citations are not used as mathematical evidence for the ranking; the ranking rests on the measured Table I values. The reported averages are simple arithmetic means of the per-sequence scores, so the central claim is not equivalent to any input by construction. The concern that Seq1 equals Seq4 and Seq2 equals Seq5 in Table I is a data-integrity and experimental-design issue about whether the five sequences are independent samples, not a circularity of the derivation chain. Since this paper is self-contained against external benchmarks and contains no fitted-input-as-prediction or self-citation-as-proof step, the circularity score is zero.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entity or fitted parameter; its main load-bearing inputs are the dataset construction choices and the unverified assumptions about model provenance, annotation quality, and sequence independence. The duplicated sequence rows make the independence assumption especially fragile.

free parameters (2)
  • test_split_fraction = 0.20
    The authors held out approximately 20% of frames for testing. All reported scores depend on which frames land in this split, and the split is chosen by hand rather than determined by data.
  • frame_sampling_rate = 10 fps
    Frames were sampled at 10 fps from 30 fps recordings. This choice affects frame redundancy and the effective diversity of the test set, and it is made by the authors.
assumptions (4)
  • domain assumption The pretrained models had no exposure to ROAD-Almaty frames during their original training.
    The method section states models received no retraining on the dataset, but no leakage test or provenance check is provided. If any test frames appeared in pretraining data, the domain generalization claim would be invalid.
  • domain assumption The five test sequences are mutually independent and representative of Almaty driving conditions.
    Table I shows exact duplicates between Seq1 and Seq4, and between Seq2 and Seq5, which contradicts the assumption of five distinct independent sequences.
  • standard math The COCO-style F1 and IoU evaluation scripts were implemented correctly.
    The authors state they used standard evaluation scripts aligned with COCO benchmarking, but no code is provided to verify the implementation.
  • domain assumption The ground-truth annotations are accurate, especially in low-visibility frames.
    The annotation protocol includes a two-stage review, but no inter-annotator agreement or quality metric is reported. Annotation errors in snow and low-light frames would directly bias the measured F1-scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Domain Generalization in Autonomous Driving: Evaluating YOLOv8s, RT-DETR, and YOLO-NAS with the ROAD-Almaty Dataset." pith.science (2026). https://pith.science/paper/CYWP2F5T

@misc{pith2026241212349,
  author       = {Pith},
  title        = {Pith review of: Domain Generalization in Autonomous Driving: Evaluating YOLOv8s, RT-DETR, and YOLO-NAS with the ROAD-Almaty Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYWP2F5T}},
  note         = {Machine review of arXiv:2412.12349}
}
read the original abstract

This study investigates the domain generalization capabilities of three state-of-the-art object detection models - YOLOv8s, RT-DETR, and YOLO-NAS - within the unique driving environment of Kazakhstan. Utilizing the newly constructed ROAD-Almaty dataset, which encompasses diverse weather, lighting, and traffic conditions, we evaluated the models' performance without any retraining. Quantitative analysis revealed that RT-DETR achieved an average F1-score of 0.672 at IoU=0.5, outperforming YOLOv8s (0.458) and YOLO-NAS (0.526) by approximately 46% and 27%, respectively. Additionally, all models exhibited significant performance declines at higher IoU thresholds (e.g., a drop of approximately 20% when increasing IoU from 0.5 to 0.75) and under challenging environmental conditions, such as heavy snowfall and low-light scenarios. These findings underscore the necessity for geographically diverse training datasets and the implementation of specialized domain adaptation techniques to enhance the reliability of autonomous vehicle detection systems globally. This research contributes to the understanding of domain generalization challenges in autonomous driving, particularly in underrepresented regions.

Figures

Figures reproduced from arXiv: 2412.12349 by the authors.

Figure 1
Figure 1. Collage of sample frames from the ROAD-Almaty dataset under varying weather and lighting conditions. TABLE I: F1 and IoU Scores Across Sequences for Different Models (Average Values) Model IoU Thresh Seq1 (IoU/F1) Seq2 (IoU/F1) Seq3 (IoU/F1) Seq4 (IoU/F1) Seq5 (IoU/F1) Average (IoU/F1) NAS 0.5 0.878/0.832 0.584/0.324 0.366/0.315 0.878/0.832 0.584/0.324 0.658/0.526 NAS 0.75 0.895/0.689 0.454/0.225 0.226/0.174 0.895/0… view at source ↗
Figure 2
Figure 2. Comparative F1-scores at IoU=0.5 across five test sequences (Seq1–Seq5) for YOLOv8s, RT-DETR, and YOLO-NAS. Each group of three bars represents a single sequence, illustrating how performance varies with changing environmental conditions and how RT-DETR consistently outperforms the YOLO-based models. distinct conditions in Kazakhstan. Additionally, our dataset, although diverse within the region, is smaller than lar… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Object Detection for Autonomous Driving in Chinese Rural Scenes: An Experimental Study on Real-Synthetic Data Mixing and Model Evaluation

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A 1:0.5 real-to-synthetic mix on a new 14-class Chinese rural dataset lifts YOLO11m to 0.758 mAP@0.5, while a 1:1 mix degrades performance via domain shift.

Reference graph

Works this paper leans on

25 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    arXiv 2014

  2. [2]

    Fast r-cnn,

    R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE International Conference on Computer Vision , pp. 1440–1448, 2015

    work page 2015

  3. [3]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 779–788, 2016

    work page 2016

  4. [4]

    Yolov4: Op- timal speed and accuracy of object detection,

    A. Bochkovskiy, C.-Y . Wang, and H.-Y . M. Liao, “Yolov4: Op- timal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020

    arXiv 2004

  5. [5]

    Yolov3: An incremental improvement,

    J. Redmon, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018

    arXiv 2018

  6. [6]

    Unbiased look at dataset bias,

    A. Torralba and A. A. Efros, “Unbiased look at dataset bias,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1521–1528, June 2011

    work page 2011

  7. [7]

    Recognition in terra incognita,

    S. Beery, G. Van Horn, and P. Perona, “Recognition in terra incognita,” in Proceedings of the European Conference on Computer Vision (ECCV) , pp. 456–473, 2018

    work page 2018

  8. [8]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1137–1149, 2016

    2016

Show all 25 references
  1. [9]

    Vision meets robotics: The kitti dataset,

    A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” International Journal of Robotics Research (IJRR) , 2013

    2013

  2. [10]

    KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,

    Y . Liao, J. Xie, and A. Geiger, “KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” Pattern Analysis and Machine Intelligence (PAMI) , 2022

    2022

  3. [11]

    Trafficpredict: Trajectory prediction for heterogeneous traffic-agents,

    Y . Ma, X. Zhu, S. Zhang, R. Yang, W. Wang, and D. Manocha, “Trafficpredict: Trajectory prediction for heterogeneous traffic-agents,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2019

    2019

  4. [12]

    Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,

    A. Rasouli, I. Kotseruba, T. Kunic, and J. Tsotsos, “Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6261–6270, 2019

    2019

  5. [13]

    nuscenes: A multimodal dataset for autonomous driving,

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Kr- ishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,”arXiv preprint arXiv:1903.11027, 2019

    1903 arXiv

  6. [14]

    Large scale interactive motion forecasting for autonomous driving : The waymo open motion dataset,

    S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y . Chai, B. Sapp, C. Qi, Y . Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V . Vasudevan, A. McCauley, J. Shlens, and D. Anguelov, “Large scale interactive motion forecasting for autonomous driving : The waymo open ...

    2021

  7. [15]

    Argoverse: 3d tracking and forecasting with rich maps,

    M.-F. Chang, J. W. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3d tracking and forecasting with rich maps,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2019

    2019

  8. [16]

    One thousand and one hours: Self- driving motion prediction dataset,

    J. L. Houston, G. C. A. Zuidhof, L. Bergamini, Y . Ye, A. Jain, S. Omari, V . I. Iglovikov, and P. Ondruska, “One thousand and one hours: Self- driving motion prediction dataset,” in Conference on Robot Learning , 2020

    2020

  9. [17]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning,

    F. Yu, H. Chen, X. Wang, W. Xian, Y . Chen, F. Liu, V . Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 2636–2645, 2020

    2020

  10. [18]

    Shift: A synthetic driving dataset for continuous multi-task domain adaptation,

    T. Sun, M. Segu, J. Postels, Y . Wang, L. V . Gool, B. Schiele, F. Tombari, and F. Yu, “Shift: A synthetic driving dataset for continuous multi-task domain adaptation,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (Los Alamitos, CA, USA), pp. ...

    2022

  11. [19]

    Virtual kitti 2,

    Y . Cabon, N. Murray, and M. Humenberger, “Virtual kitti 2,” 2020

    2020

  12. [20]

    Unsupervised domain adaptation with residual transfer networks,

    M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Unsupervised domain adaptation with residual transfer networks,” Advances in neural infor- mation processing systems , vol. 29, 2016

    2016

  13. [21]

    Adversarial discrim- inative domain adaptation,

    E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discrim- inative domain adaptation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 7167–7176, 2017

    2017

  14. [22]

    Domain generalization for object recognition with multi-task autoencoders,

    M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi, “Domain generalization for object recognition with multi-task autoencoders,” in Proceedings of the IEEE international conference on computer vision , pp. 2551–2559, 2015

    2015

  15. [23]

    Yolov8 documentation,

    Ultralytics, “Yolov8 documentation,” 2023

    2023

  16. [24]

    Rt-detr: Real-time detection transformer,

    Ultralytics, “Rt-detr: Real-time detection transformer,” 2023

    2023

  17. [25]

    Yolo-nas documentation,

    Ultralytics, “Yolo-nas documentation,” 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.