{"id":"6361fd84-7e9f-45c1-adc5-7b381a846556","arxiv_id":"2412.01983","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A post-processing pixel-wise ROI mask applied to pretrained YOLO detections improved parking occupancy counting, with YOLOv9e reaching 99.68% balanced accuracy on a custom dataset, though the evaluation metrics are not clearly tied to the labels.","lead":"Four recent YOLO models are tested for counting cars in a university parking lot, using a new pixel-level mask that discards detections outside the lot after inference. The best configuration, YOLOv9e with the post-processing mask, reports 99.68% balanced accuracy on a private 3,484-image dataset, but the result depends on a metric definition that may not match the image-level labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-space confusion matrix metrics cannot be computed from the count-only labels described in §4.1; §6 confirms bounding-box labeling is future work. The 99.68% balanced accuracy is therefore not computable from the described data.","rationale":"The reader's weakest assumption identifies exactly the same internal inconsistency I find most load-bearing. The central claim of the paper is the 99.68% balanced accuracy, but the evaluated quantities (TP/TN/FP/FN per space) are not computable from the described labels. The paper's own text in §4.1 and §6 provides strong evidence: labels are image-level counts, and bounding-box labeling is left as future work. This is not a matter of external scrutiny; it is a logical gap in the evaluation protocol. The qualitative conclusion that post-processing ROI beats pre-processing ROI is plausible and supported by the trend in Table 3, and the paper honestly notes limitations such as the lack of IoU. However, the quantitative headline number cannot be audited or reproduced without per-space ground truth. Given the paper's primary contribution is that specific accuracy figure, the rejection is warranted. I see no reason to soften or strengthen the reader's verdict.","tokens_in":15972,"tokens_out":4556,"duration_ms":42452,"concrete_test":"Analytically demonstrate the non-identifiability: pick a single image from the dataset with known true vehicle count V and predicted count P (e.g., V=5, P=3). Enumerate all possible confusion matrices over the 16 spaces consistent with these counts. If more than one balanced accuracy value results, the metrics in Table 3 cannot be derived from count-only labels. This check requires no new data; it only needs the count labels and the detector's per-image counts, which the paper already summarizes. If the authors can supply per-space annotations or a deterministic mapping, the concern would be resolved; otherwise the headline accuracy is undefined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—99.68% balanced accuracy for YOLOv9e with post-processing ROI (Table 3)—requires per-space occupancy ground truth. Section 4.4 defines TP as 'correctly predicted empty space' and TN as 'correctly predicted vehicle', i.e., a binary classification per parking space. However, Section 4.1 states the labeling process only 'counting the number of vehicles in each image', with no spatial annotation. Section 6 explicitly lists 'improving the labeling of the dataset by following bounding box coordinates' as future work, confirming that no per-space or bounding-box ground truth exists. Without per-space labels, the confusion matrices and all derived metrics (accuracy, F1, balanced accuracy) in Table 3 are not uniquely determined by the data; multiple confusion matrices are consistent with any given (true count, predicted count) pair. For example, with 16 spaces, true count 5, predicted count 3, the number of true positives (correctly predicted empty spaces) can range from 9 to 11, producing different accuracies and balanced accuracies. Thus the reported numbers are not computable from the described evaluation protocol. This is an internal inconsistency, not merely a disagreement with external best practices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a smart-parking system that uses off-the-shelf pretrained YOLOv8, YOLOv9, YOLOv10, and YOLOv11 models to count vehicles in a fixed 16-space parking lot. A hand-drawn pixel-wise ROI mask is applied either as a pre-processing step (graying out pixels outside the ROI) or as a post-processing step (keeping only detections whose bounding-box center falls inside the ROI). The paper reports per-space accuracy, balanced accuracy, and F1-score for 16 model/mask configurations on a custom 3,484-image dataset, with a headline balanced accuracy of 99.68% for YOLOv9e with post-processing. It also reports inference-time measurements on six hardware platforms and a cost comparison between camera-based and sensor-based solutions.","tokens_in":16130,"tokens_out":7192,"duration_ms":66226,"significance":"If the reported evaluation were valid, the paper would provide a useful engineering data point for fixed-camera parking occupancy counting with recent YOLO versions. The post-processing ROI filter is simple and practical, and the latency benchmark across GPUs, CPUs, and Raspberry Pi devices together with the cost analysis are concrete, reproducible contributions. The paper correctly gives credit to the use of off-the-shelf models and does not claim to train a new detector. However, the central quantitative claim depends on per-space confusion-matrix metrics that cannot be computed from the count-only labels described in the manuscript, so the headline accuracy and the model-comparison table are not supported by the described evaluation protocol.","major_comments":[{"comment":"The evaluation protocol is internally inconsistent. Section 4.1 describes labeling only as \"counting the number of vehicles in each image,\" with no spatial annotation, while Section 4.4 defines TP, TN, FP, and FN at the level of individual parking spaces (e.g., TP = \"correctly predicted empty space\"). Section 6 explicitly lists \"improving the labeling of the dataset by following bounding box coordinates\" as future work, confirming that no per-space or bounding-box ground truth currently exists. Under count-only labels, the per-space confusion matrices in Figures 7–8 and all metrics in Table 3 are not uniquely determined. For example, for one image with 16 spaces, a true count of 5 vehicles and a predicted count of 3 vehicles, the number of correctly predicted empty spaces can range from 8 to 11, yielding per-image balanced accuracies between 30.77% and 92.31% depending on which specific spaces are misclassified. The reported 99.68% balanced accuracy for YOLOv9e is therefore not computable from the described dataset and labeling procedure, and this is a load-bearing flaw because that value is the paper's headline result.","section":"§4.1, §4.4, §6"},{"comment":"The headline 99.68% balanced accuracy is the maximum over 16 model/mask configurations evaluated on the same 3,484 images, with no confidence intervals, repeated runs, or a separate selection set. Section 4.2 states that \"100% of the data was used as validation,\" so the reported best value is a selected maximum rather than a statistically grounded estimate. The qualitative conclusion that post-processing beats pre-processing is robust across all eight model rows in Table 3, but the specific numerical claim needs an uncertainty estimate or a held-out evaluation protocol to support it.","section":"§4.2, §5.1, Table 3"}],"minor_comments":[{"comment":"The definitions of TP and TN are reversed relative to the standard convention: TP is defined as \"correctly predicted empty space\" and TN as \"correctly predicted vehicle.\" This is confusing and should be clarified by explicitly naming the positive class, or by using labels such as \"empty-space detection\" rather than the standard TP/TN terminology.","section":"§4.4"},{"comment":"The statement that filtering out empty images \"helps to balance the number of backgrounds and vehicles\" is unclear; removing all images with zero vehicles removes the empty-parking-lot class entirely, which seems to unbalance rather than balance the dataset.","section":"§4.1"},{"comment":"Please specify whether the same 80 random images and the same random seed were used for all hardware/model timing measurements, and state how many repeated runs produced the reported standard deviations.","section":"§5.2, Table 4"},{"comment":"The mask was manually drawn in GIMP on one image; the paper should describe how the 16 parking spaces are represented in the mask and whether the mask requires fixed camera alignment or per-image registration.","section":"§4.3"},{"comment":"The future-work sentence about \"improving the labeling of the dataset by following bounding box coordinates\" is in direct tension with the space-level confusion-matrix metrics used in Section 4.4; this inconsistency should be resolved explicitly.","section":"§6"},{"comment":"Reference [32] contains a typo (\"supremum to\" should be \"superior to\"), and reference [51] lacks journal, volume, and page information.","section":"References"}],"recommendation":"reject","confidential_remarks":"The primary issue is an internal inconsistency between the count-only labels described in Section 4.1 and the per-space confusion-matrix metrics used in Section 4.4 and Table 3. Because the dataset is not available and the paper itself states that bounding-box labeling is future work, this is not a simple presentation fix. A resubmission with spatial ground-truth annotations, recomputed metrics, and uncertainty quantification for the selected best model would be needed to make the central claim verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: the 99.68% balanced accuracy is not supported by the data as described. Section 4.4 defines TP/TN/FP/FN per parking space, but Section 4.1 says labeling was done by counting vehicles per image, with no spatial annotations. Section 6 confirms that bounding-box labeling is future work. Without per-space ground truth, the confusion matrices and all derived metrics are not uniquely determined by the data; the reported numbers are simply not computable under the described protocol. This is a load-bearing flaw, not a style issue.\n\nWhat is genuinely useful: the pixel-wise post-processing ROI mask is a simple, transferable idea - check the box center against a hand-drawn mask after inference - and the paper honestly notes it generalizes the polygonal counting filters already in Ultralytics. The comparison of YOLOv8 through YOLOv11 on a parking dataset with off-the-shelf weights is new to this domain. The latency measurements across six devices are concrete and reproducible in spirit, and the cost analysis is sensible. The qualitative finding that post-processing beats pre-processing is plausible and consistent with the table, since graying out image context hurts YOLO performance.\n\nSoft spots beyond the central one: the best result is selected from 16 model/mask combinations measured on the same 100% validation set, no confidence intervals are given, and the dataset and code are not public. That said, there are no fitted parameters, so the comparison is an honest empirical measurement rather than a circular fit.\n\nWho this is for: applied people building low-cost camera-based parking counters. They will get the engineering story even if the headline number is suspect. The paper deserves a serious referee, but the evaluation must be redone, ideally on a public dataset with per-space labels, before the accuracy claim can be taken seriously. I would not cite the 99.68% number in my own work, though the post-processing mask trick and the latency table could be worth a footnote.\n\nRecommendation: send to peer review rather than desk reject, with the expectation of major revision. The flaw is correctable and the empirical comparison has value.","headline":"The 99.68% balanced accuracy claim is not computable from the count-only labels the paper describes; the useful core is the post-processing mask trick and the hardware latency comparison.","tokens_in":16778,"tokens_out":3142,"would_cite":false,"duration_ms":30264,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pixel-wise post-processing mask lets off-the-shelf YOLO models count parked cars with 99.68% balanced accuracy.","keywords":["smart parking","YOLOv8","YOLOv9","YOLOv10","YOLOv11","region of interest","vehicle detection","edge computing"],"falsifier":"Inspect the dataset's annotation files: if each image has only an integer vehicle count and no per-space occupied or empty ground truth, then the per-space TP/TN/FP/FN values and the 99.68% balanced accuracy cannot be reproduced from the described data, and recomputing at the image-count level would yield different error rates.","tokens_in":15714,"feed_emoji":"🅿️","tokens_out":7041,"duration_ms":68108,"temperature":0.7,"pith_summary":"This paper tries to establish that a cheap fixed camera plus off-the-shelf YOLO models can count parked cars almost perfectly if the region of interest is selected after detection, pixel by pixel, rather than by masking the image before detection. The proposed post-processing mask, applied to the centers of detected bounding boxes, beats the pre-processing mask for all eight YOLO variants tested, and with YOLOv9e it reaches 99.68% balanced accuracy and an F1-score of 0.9975 on 3,484 images of a 16-space parking lot. The paper also shows that inference can run at the edge, from 1 to 92 seconds per image on Raspberry Pi devices, while sending only the final count to the cloud, and estimates that a camera system becomes cheaper than per-space sensors beyond roughly four spaces. A reader should care because, if correct, this is a simple, low-cost path to privacy-preserving parking occupancy monitoring without fine-tuning or specialized hardware.","feed_headline":"A pixel-wise ROI mask pushes parking-lot counts to 99.68%","feed_subtitle":"Post-processing masks beat pre-processing for counting cars with off-the-shelf YOLO models, on edge hardware.","key_machinery":"The central mechanism is the pixel-wise post-processing mask: a binary mask, drawn once in an image editor, that labels each pixel as inside or outside the parking ROI. After YOLO inference, the mask is not applied to the image; the detector's bounding boxes are filtered by looking up the pixel value at each box's center and keeping the box only if that pixel is black. This preserves full image context for the network while restricting the count to the area of interest, and it is the component that produces the reported accuracy gap.","core_discovery":"The paper's central claim is that applying a pixel-wise region-of-interest mask after object detection, rather than before, materially improves vehicle counting in parking-lot images. The same hand-drawn mask is used either to gray out non-parking areas in the input image or to filter detected bounding boxes by checking whether each box's center falls inside the black ROI. Across all eight YOLO variants tested, the post-processing approach wins; with YOLOv9e it reaches 99.68% balanced accuracy and an F1-score of 0.9975 on a 3,484-image dataset of a 16-space lot, without any fine-tuning of the pretrained models.","pith_inferences":["The center-point-in-mask filter is object-class agnostic, so the same post-processing trick should transfer to other fixed-camera counting tasks such as people in queues or items on shelves; the paper does not test those cases.","The 99.68% figure is tied to one camera angle and a 16-space layout; multi-camera lots would need per-camera masks and a fusion step, which the paper flags only as future work.","Mask geometry is a free parameter that the paper does not sweep; redrawing the same ROI slightly larger or smaller would show how much of the reported accuracy depends on hand-drawn boundaries."],"forward_implications":["Post-processing ROI selection improves balanced accuracy over pre-processing for every YOLO model tested, with gains exceeding 28 percentage points for the smallest model.","A deployment needs no training: pretrained COCO weights, one hand-drawn mask, and a fixed camera are enough for near-perfect occupancy counts in a small lot.","The method can run at the edge on a Raspberry Pi, sending only the number of vehicles to the cloud and keeping images on site.","The camera solution is estimated to become cheaper than per-space sensors once a parking lot has more than about four spaces.","The lightest models run in about one to two seconds per image on Raspberry Pi hardware with only a modest accuracy penalty, making always-on occupancy monitoring feasible on very cheap devices."],"supporting_citations":[{"why":"Supplies the pretrained YOLOv8 through YOLOv11 models and the inference library through which all detections are obtained.","marker":"[40]"},{"why":"Earlier YOLO-based parking detection using a pre-processing ROI mask; this is the pre-processing baseline the proposed method improves on.","marker":"[18]"},{"why":"Earlier YOLOv5 parking-management framework using the same pre-processing ROI approach and custom/PKLot evaluations.","marker":"[23]"},{"why":"Camera-network smart parking work that defines the per-space TP/TN occupancy metrics and edge-inference design adopted here.","marker":"[16]"},{"why":"Another camera-based smart parking system whose metric definitions for empty and occupied spaces are referenced for the evaluation.","marker":"[49]"},{"why":"The COCO benchmark that supplies the pretrained model accuracy and latency numbers used to choose the YOLO variants.","marker":"[45]"}],"fun_headline_variants":["Post-processing ROI mask boosts parking count to 99.68%","Pixel-wise ROI after detection hits 99.68% accuracy","YOLOv9e with post-mask tops parking lot counting","Off-the-shelf YOLO models achieve 99.68% with pixel ROI","ROI after detection, not before, wins parking counting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dataset contains per-space occupancy labels rather than only a total vehicle count per image, since the reported confusion matrices and balanced accuracy are per-space metrics; the paper's labeling description mentions only counting vehicles per image.","fun_headline_variants_meta":{"raw":{"variants":["Post-processing ROI mask boosts parking count to 99.68%","Pixel-wise ROI after detection hits 99.68% accuracy","YOLOv9e with post-mask tops parking lot counting","Off-the-shelf YOLO models achieve 99.68% with pixel ROI","ROI after detection, not before, wins parking counting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2733,"prompt_tokens":932,"completion_tokens":1801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1709}},"tokens_in":548,"tokens_out":1801,"duration_ms":11764,"temperature":1.0,"reasoning_tokens":1709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:57:54.826204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the dataset's annotation files: if each image has only an integer vehicle count and no per-space occupied or empty ground truth, then the per-space TP/TN/FP/FN values and the 99.68% balanced accuracy cannot be reproduced from the described data, and recomputing at the image-count level would yield different error rates.","supporting_citations":[{"cited_title":"Ultralytics YOLO (Version 8.0.0)","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained YOLOv8 through YOLOv11 models and the inference library through which all detections are obtained."},{"cited_title":"T-yolo: Tiny vehicle detection based on yolo and multi-scale convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Earlier YOLO-based parking detection using a pre-processing ROI mask; this is the pre-processing baseline the proposed method improves on."},{"cited_title":"Optimized real-time parking management framework using deep learning","cited_arxiv_id":null,"evidence_quote":"Earlier YOLOv5 parking-management framework using the same pre-processing ROI approach and custom/PKLot evaluations."},{"cited_title":"An edge based smart parking solution using camera networks and deep learning","cited_arxiv_id":null,"evidence_quote":"Camera-network smart parking work that defines the per-space TP/TN occupancy metrics and edge-inference design adopted here."},{"cited_title":"A camera- based smart parking system employing low-complexity deep learning for outdoor environments","cited_arxiv_id":null,"evidence_quote":"Another camera-based smart parking system whose metric definitions for empty and occupied spaces are referenced for the evaluation."}],"review_version":1}