COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving

Ali K. AlShami; Ananya Kalita; Jugal Kalita; Khang Lam; Rishabh Bezbarua; Ryan Rabinowitz; Terrance Boult

REVIEW 4 major objections 5 minor 2 cited by

COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces COOOL, a benchmark of over 200 dashcam videos annotated by humans to mark road hazards, designed to evaluate whether autonomous driving systems can detect, name, and react to out-of-label hazards that were never seen…

desk verdict COOOL is a sensible evaluation-benchmark idea for out-of-label road hazards, but the dataset is unreleased, the label reliability is unmeasured, and the baseline provides no numbers, so it cannot be used as-is. read the letter →

arxiv 2412.05462 v1 pith:GQXZFFTA submitted 2024-12-06 cs.CV

Ali K. AlShami , Ananya Kalita , Ryan Rabinowitz , Khang Lam , Rishabh Bezbarua , Terrance Boult , Jugal Kalita This is my paper · ORCID

classification cs.CV

keywords out-of-labeldetectionautonomousdrivingdashcamvideoshazardbenchmarkopen-setrecognitionanomalylow-resolutionhazards

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces COOOL, a new evaluation benchmark for autonomous driving built from over 200 dashcam videos in which human annotators mark objects of interest and potential driving hazards. The benchmark is designed to test a system's ability to handle out-of-label hazards, meaning dangers that were not among the categories it was trained to recognize. The authors argue that existing autonomous-driving datasets evaluate recognition within a fixed set of classes and leave the novelty problem largely unaddressed. If COOOL works as intended, it gives researchers in anomaly detection, open-set recognition, open vocabulary, and domain adaptation a shared resource to measure how well a self-driving vision system detects, names, and reacts to the unexpected.

What carries the argument

The core object is the COOOL dataset itself: over 200 dashcam videos with per-frame human annotations of hazards and potential hazards, a Tag system for contextual metadata, and the associated evaluation metric. The metric computes accuracy for each of the three challenge tasks and takes the macro average, penalizing over-prediction by dividing by the maximum of ground-truth hazards or predicted hazards. This mechanism forces a system to localize the hazardous object, describe it, and detect the driver's reaction, which together define out-of-label hazard detection operationally.

What would settle it

Count the actual videos and annotations in a public release of COOOL and compare with the claimed 200 collections, or run an inter-annotator agreement study on a sample of frames; if the number is far below 200, or if annotators disagree at near-chance levels about which object is hazardous, the benchmark's evaluation metric cannot be trusted as a measure of out-of-label hazard detection.

Watch

Extended reading notes

Core claim

The central claim is that COOOL fills the gap of out-of-label hazard detection by providing a benchmark that is evaluation-only: 200 high-resolution dashcam videos, annotated frame-by-frame with bounding boxes for hazards and nuisance objects, plus a Tag system capturing vehicle movement and driver decisions. The benchmark defines three tasks—determining when the driver begins reacting to a hazard, identifying which object is hazardous, and naming the hazard—and scores systems by macro accuracy over these three measures. The paper argues that this combination makes it possible to evaluate not just closed-set object detection but the genuinely novel cases that matter for road safety.

Load-bearing premise

The benchmark's validity depends on the assumption that the human annotations are correct and reliable ground truth for which objects are hazardous and when the driver reacts, yet the paper reports no inter-annotator agreement or quality audit and the dataset is not released for independent verification.

Editorial extensions

If this is right

Researchers in anomaly detection, open-set recognition, open vocabulary, and domain adaptation gain a common benchmark for measuring performance on unseen road hazards.
The three-task metric encourages systems to localize and name the hazardous object, not just flag that something is wrong, which is closer to what a driver needs.
The inclusion of more than 10,000 low-resolution hazards under 50 by 50 pixels lets the community test early detection of distant dangers.
Because COOOL is evaluation-only, it can serve as a held-out test set without risk of training contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark's operational definition of out-of-label hazard detection—reaction timing, object localization, and naming—could serve as a template for evaluating novelty in other safety-critical vision domains such as robotics or medical imaging.
If COOOL reveals that current open-vocabulary or anomaly-detection models are easily confused by nuisance objects such as plastic bags, it would suggest that hazard detection needs better reasoning about object dynamics, not just better recognition.
The dataset could be used to measure how much performance degrades as hazard size shrinks, disentangling small-object difficulty from novelty difficulty.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper proposes COOOL, an evaluation-only benchmark of 200 dashcam videos with human annotations for out-of-label hazards and driver reactions. The core idea fills a real gap: most driving datasets only cover fixed object classes, so evaluating novelty detection on the road is genuinely underserved. The three-task decomposition—when the driver reacts, which object is hazardous, and what it is called—is practical and gives multiple entry points for anomaly detection, open-vocabulary, and open-set researchers. The reported scale (100k+ vehicle annotations, 40k+ animals, 10k+ low-res hazards) suggests a nontrivial effort, and the focus on hazards under 50 pixels is a nice challenge that most benchmarks ignore. The evaluation metric's built-in penalty for over-prediction is also reasonable.

The soft spots are real and, proportionate to the paper's claims, serious. The biggest is verifiability: the dataset is not actually released (the paper gives a generic GitHub link with no access instructions), no annotation protocol is described, and no inter-annotator agreement is reported. For an evaluation-only benchmark, the entire metric depends on trusting that the human labels are correct, and the driver-reaction labels in particular are inherently ambiguous from dashcam video alone. Without kappa, IoU, or adjudication rules, there is no way to tell whether the ground truth is stable. The baseline section is essentially a placeholder—logistic regression for reaction timing, closest bounding box for object selection, a clip interrogator for naming—but no results are given. That means the paper does not demonstrate that any of the three tasks is solvable even in a simple form. The related work also skips the established anomaly and out-of-distribution driving benchmarks (BDD-Anomaly, Fishyscales, and the like), so the novelty claim is under-contextualized.

These are correctable problems rather than fatal ones. The paper seems honest and the dataset likely exists. But as a scientific artifact, it is currently unverifiable, and the stress-test concern about label reliability holds up on reading. The reader's CONDITIONAL verdict is appropriate.

The paper is aimed at researchers working on novelty-adjacent driving perception who might want a new evaluation resource. It deserves peer review—a serious referee could push the authors to release the data, report annotation agreement, and provide baseline results. Without those, it belongs in a workshop, not as a citable benchmark. I would not cite it until the dataset is accessible and validated.

Referee Report

4 major / 5 minor

Summary. The paper introduces COOOL, a proposed evaluation-only benchmark of over 200 dashcam videos annotated by human labelers to mark objects of interest and potential driving hazards. The stated goal is to support research on out-of-label hazard detection in autonomous driving. The manuscript describes three challenge tasks (driver reaction timing, hazardous object identification, and hazard naming), a macro-accuracy evaluation metric, a qualitative baseline approach, and aggregate annotation statistics. It positions COOOL as filling a gap left by existing autonomous-driving datasets, which mostly address predefined object categories.

Significance. If the dataset were publicly released and its labels validated, COOOL would address a real and currently underserved need: dedicated evaluation of out-of-label hazard detection for autonomous driving. The idea of bridging anomaly detection, open-set recognition, open vocabulary, and domain adaptation through a common video benchmark with three well-separated tasks is attractive, and the paper is appropriately modest in framing COOOL as evaluation-only. However, in its current form the contribution is not yet substantiated. The dataset is not accessible, no inter-annotator agreement or annotation-protocol information is provided, and no baseline results or metric-calibration numbers are reported. Because the benchmark's usefulness depends entirely on the reliability of the human labels and on whether other researchers can use the data, the significance of the contribution cannot be fairly assessed until these gaps are addressed.

major comments (4)

[Sections 3 and 5; GitHub link in title page] The central artifact of the paper, the COOOL dataset, is not accessible. The only link provided is a bare GitHub URL with no description of its contents, no download instructions, and no sample annotation images or metadata. Since COOOL is explicitly an evaluation-only benchmark, readers cannot verify the claimed statistics (over 100,000 vehicle annotations, 40,000 animals, 10,000 low-resolution hazards) and cannot run or validate any evaluation. A dataset paper of this type must provide either a release mechanism or a detailed supplementary description that allows independent audit of the annotation format and content.
[Sections 3 and 5; Section 4.3] No evidence of label reliability is reported. The evaluation metric directly measures agreement with human ground truth on three inherently subjective judgments: which object is hazardous, the hazard name, and the frame in which the driver begins reacting. The paper states that annotations were made by undergraduate and high-school students under supervision and that every annotation was reviewed, but it provides no annotation protocol, no definition of a hazard, no adjudication rules, and no inter-annotator agreement statistics such as Cohen's kappa or bounding-box IoU. This is especially problematic for the driver-reaction label, which is highly ambiguous when inferred from dashcam video alone. Without such evidence, the benchmark scores cannot be interpreted as measuring hazard-detection ability rather than label noise.
[Section 4.2] The baseline is described only in qualitative terms, and no experimental results are reported. The text states that a baseline is provided via a GitHub link, but it does not give any performance numbers on the three tasks, no evaluation on the proposed metric, and no comparison point that would allow readers to calibrate the difficulty of the benchmark. For a benchmark paper, at least sanity-check results from the baseline are necessary to show that the evaluation protocol is implementable and that the metric is not degenerate.
[Section 4.3] The evaluation metric is not precisely defined. The description that accuracy is computed "over the maximum of ground truth hazards present or number of hazards predicted" and that over-prediction is penalized "by increasing the total number of hazards by the total number of guesses" is ambiguous without a mathematical formulation. It is also unclear how the three heterogeneous measures (binary reaction per frame, hazardous-object selection, and description matching) are combined into the stated "macro accuracy". An explicit formula and per-task scoring rules are needed for the benchmark to be reproducible.

minor comments (5)

[Abstract vs. Section 3] The abstract states "over 200 collections of dashcam-oriented videos" while Section 3 says "a collection of 200 high-resolution dashcam videos". These should be made consistent, and the exact number of videos and the definition of a "collection" should be clarified.
[Figures 1 and 2] The figure numbering is inconsistent: Figure 1 is captioned as COOOL benchmark examples, but Section 3 says examples are shown in Figure 2, and the later figure is a statistics chart ("Analysis the COOOL Benchmark"). The figures should be renumbered and each figure explicitly referenced where it is discussed.
[Section 4.1] The text contains a typo: "avoidance manuvers" should be "avoidance maneuvers".
[Section 5.1] The phrase "smaller than 50x50 pixels" should use a multiplication sign (50×50) and ideally state whether this refers to the width and height of the bounding box or the image area.
[Section 4.2] The phrase "Github Link" is not a usable reference; the full URL of the baseline repository should be included in the text or footnotes, not only in the title-page header.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: COOOL is a dataset/benchmark paper whose evaluation metric measures agreement with human ground truth; no prediction is derived from its own fitted inputs.

full rationale

The paper's contribution is the COOOL dataset and evaluation benchmark. There is no derivation chain in which an output quantity is defined in terms of the quantity it is supposed to predict. Section 4.3 defines the evaluation metric as accuracy of predicted hazardous objects, hazard descriptions, and driver-reaction frames against human-provided ground truth; this is a standard benchmark evaluation protocol, not a circular reduction. The baseline in Section 4.2 uses logistic regression, closest-bounding-box selection, and a clip interrogator, but these are presented as a simple reference implementation, not as a result derived from the dataset's own definitions. No parameter is fitted to a subset of data and then renamed as a prediction. The paper cites prior work for context, but none of the citations is from the present authors and none carries a load-bearing uniqueness or ansatz claim. The concern that human annotation quality is not quantitatively verified with inter-annotator agreement or a released dataset is a verification and robustness limitation, not circularity: the benchmark's ground truth is an external human judgment rather than a re-encoding of the metric or of system outputs. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper adds a dataset and benchmark, so no free parameters or invented entities are involved. The central assumptions are about label quality, dataset representativeness, and the meaningfulness of the proposed evaluation metric.

assumptions (2)

domain assumption Human annotations in COOOL are treated as ground truth for hazard objects and driver reactions, despite no measured inter-annotator agreement.
The evaluation metric and all benchmark conclusions depend on label quality; Section 3 and Section 5 assert careful reviewing but provide no quantitative agreement or audit.
domain assumption The dashcam video collection is representative of real-world out-of-label driving hazards.
The paper does not describe how videos were sourced, selected, or balanced across weather, location, and hazard rarity; Section 5 gives counts but no sampling design. If selection is biased, benchmark conclusions may not generalize.

how reviews work

0 comments

Cite this review

Pith. "Pith review of COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving." pith.science (2026). https://pith.science/paper/GQXZFFTA

@misc{pith2026241205462,
  author       = {Pith},
  title        = {Pith review of: COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQXZFFTA}},
  note         = {Machine review of arXiv:2412.05462}
}

read the original abstract

As the Computer Vision community rapidly develops and advances algorithms for autonomous driving systems, the goal of safer and more efficient autonomous transportation is becoming increasingly achievable. However, it is 2024, and we still do not have fully self-driving cars. One of the remaining core challenges lies in addressing the novelty problem, where self-driving systems still struggle to handle previously unseen situations on the open road. With our Challenge of Out-Of-Label (COOOL) benchmark, we introduce a novel dataset for hazard detection, offering versatile evaluation metrics applicable across various tasks, including novelty-adjacent domains such as Anomaly Detection, Open-Set Recognition, Open Vocabulary, and Domain Adaptation. COOOL comprises over 200 collections of dashcam-oriented videos, annotated by human labelers to identify objects of interest and potential driving hazards. It includes a diverse range of hazards and nuisance objects. Due to the dataset's size and data complexity, COOOL serves exclusively as an evaluation benchmark.

Figures

Figures reproduced from arXiv: 2412.05462 by the authors.

**Figure 1.** COOOL benchmark examples. driving research. Each object is meticulously labeled in every frame, and a unique ’Tag’ system provides insights into the vehicle’s movements and the driver’s decisions. The dataset was annotated by undergraduate students from the University of Colorado Colorado Springs (UCCS) and local high school students under the supervision of expert graduate students and professors, using a professio… view at source ↗

**Figure 2.** Analysis the COOOL Benchmark Four visualizations of COOOL benchmark characteristics describing the distribution of hazard types, low-resolution hazards, hazards at driver reaction moments, and potential hazards per frame. All videos are captured in High-Definition, providing rich visual information and enabling annotation of distant hazards. Further, every annotation has been proposed and reviewed by skilled human a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety
cs.CV 2025-04 conditional novelty 4.0 of 10

A zero-shot multi-agent pipeline of ViLA, OmniVLM, GPT-4o-mini, and CLIP detects and describes novel road hazards, scoring 0.3922 on the authors' new COOOLER benchmark.
Zero-shot Hazard Identification in Autonomous Driving: A Case Study on the COOOL Benchmark
cs.CV 2024-12 conditional novelty 4.0 of 10

A competition pipeline combining change-point detection, optical flow, ViT classification, and the MOLMO vision-language model to identify and caption out-of-label road hazards, scoring 2nd on the COOOL leaderboard.

Reference graph

Works this paper leans on

11 extracted references · 7 canonical work pages · cited by 2 Pith papers

[1]

Uno: Unsupervised occupancy fields for perception and forecasting

Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14487–14496, 2024

work page 2024
[2]

Point cloud forecasting as a proxy for 4d occupancy forecasting

Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1116–1124, 2023

2023
[3]

S2net: Stochastic sequential pointcloud forecasting

Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference on Computer Vision, pages 549–564. Springer, 2022

work page 2022
[4]

Are we ready for autonomous driving? the kitti vision benchmark suite

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354–3361. IEEE, 2012

2012
[5]

nuscenes: A multimodal dataset for autonomous driving

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621–11631, 2020

2020
[6]

Towards long-tailed 3d detection

Neehar Peri, Achal Dave, Deva Ramanan, and Shu Kong. Towards long-tailed 3d detection. In Conference on Robot Learning, pages 1904–1915. PMLR, 2023

work page 1904
[7]

Predicting the intention of distracted pedestrians at road crossings

Amy L O’Dell, Ashleigh J Filtness, and Andrew P Morris. Predicting the intention of distracted pedestrians at road crossings. Accident Analysis & Prevention, 173:106707, 2022

work page 2022
[8]

A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook

Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C Knoll. A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook. IEEE Transactions on Intelligent V ehicles, 2024

work page 2024

Show all 11 references

[9]

Variational autoencoder for end-to-end control of autonomous driving with novelty detection and training de-biasing

Alexander Amini, Wilko Schwarting, Guy Rosman, Brandon Araki, Sertac Karaman, and Daniela Rus. Variational autoencoder for end-to-end control of autonomous driving with novelty detection and training de-biasing. In 2018 IEEE/RSJ International Conference on Intelligent Robots a...

2018
[10]

Towards explainable, safe autonomous driving with language embeddings for novelty identification and active learning: Framework and experimental analysis with real-world data sets

Ross Greer and Mohan Trivedi. Towards explainable, safe autonomous driving with language embeddings for novelty identification and active learning: Framework and experimental analysis with real-world data sets. arXiv preprint arXiv:2402.07320, 2024

2024 arXiv
[11]

Lisa amazonmlsl vehicle attributes (lava) dataset

Ninad Kulkarni, Akshay Rangesh, Jonathan Buck, Jeremy Feltracco, Mohan M Trivedi, Nachiket Deo, Ross Greer, Saman Sarraf, and Suchitra Sathyanarayana. Lisa amazonmlsl vehicle attributes (lava) dataset. 2021. 5

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Uno: Unsupervised occupancy fields for perception and forecasting

Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14487–14496, 2024

work page 2024

[2] [2]

Point cloud forecasting as a proxy for 4d occupancy forecasting

Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1116–1124, 2023

2023

[3] [3]

S2net: Stochastic sequential pointcloud forecasting

Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference on Computer Vision, pages 549–564. Springer, 2022

work page 2022

[4] [4]

Are we ready for autonomous driving? the kitti vision benchmark suite

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354–3361. IEEE, 2012

2012

[5] [5]

nuscenes: A multimodal dataset for autonomous driving

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621–11631, 2020

2020

[6] [6]

Towards long-tailed 3d detection

Neehar Peri, Achal Dave, Deva Ramanan, and Shu Kong. Towards long-tailed 3d detection. In Conference on Robot Learning, pages 1904–1915. PMLR, 2023

work page 1904

[7] [7]

Predicting the intention of distracted pedestrians at road crossings

Amy L O’Dell, Ashleigh J Filtness, and Andrew P Morris. Predicting the intention of distracted pedestrians at road crossings. Accident Analysis & Prevention, 173:106707, 2022

work page 2022

[8] [8]

A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook

Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C Knoll. A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook. IEEE Transactions on Intelligent V ehicles, 2024

work page 2024