{"id":"65838ac5-d85a-4978-8e64-6cf5233125f8","arxiv_id":"2506.02359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Off-the-shelf vision-language models can auto-label training data well enough that lightweight object detectors trained on those labels are competitive with human-labeled training on VOC and COCO.","lead":"Researchers used off-the-shelf vision-language models to generate object-detection labels automatically, then trained lightweight detectors on those labels without human annotation. On common datasets like VOC and COCO, the auto-labeled models came within a few points of human-labeled baselines at a tiny fraction of the labeling cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sec. 3.1 asserts 'no leakage' between foundation-model pre-training and evaluation sets without any deduplication evidence; image overlap from shared Flickr sources could inflate the claimed VOC/COCO/LVIS results.","rationale":"The reader's weighted concern is the selection of α on the same validation splits used for reporting. That is legitimate but likely second-order: the mAP50-versus-α curves in Fig. 3 are flat from 0.1 to 0.2, and Tab. 6 shows α=0.2 is optimal in 9 of 11 rows, so the chosen threshold has cross-dataset stability rather than a single sharp peak. The leakage question is more fundamental because it is a yes/no condition on the zero-shot premise. The paper's Sec. 3.1 states 'there is no leakage' without any deduplication or image-identity check, and the dataset provenance makes overlap plausible: VOC/COCO/LVIS images are from Flickr, and so are the pre-training sets. If the foundation models saw any evaluation images, the reported VOC/COCO/LVIS numbers no longer measure zero-shot auto-labeling; they measure some degree of memorisation, and the central 'viability' claim loses its support for novel user data. I am not asserting that leakage exists, only that the paper's unsupported assertion is a load-bearing condition that can and should be tested. Since the conditional verdict already requires additional evidence, my concern does not move the verdict; it sharpens the required condition.","tokens_in":19023,"tokens_out":12259,"duration_ms":111638,"concrete_test":"Compute perceptual hashes (e.g., pHash or PDQ) for the pre-training images of YOLOW/YOLOE (Objects365, GQA, Flickr30k) and GDINO (GoldG, Cap4M), and compare them against the train/val images of VOC, COCO, LVIS, and BDD to find exact or near-duplicates. If duplicates are found, exclude them from the auto-labeled train set and retrain YOLO11n on YOLOW-0.2 labels for VOC and COCO; if validation mAP50 drops by more than about 0.02, the reported competitive-performance claim is contaminated. If no duplicates are found, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim (Sec. 4, Tab. 11): YOLO11n trained on YOLOW-0.2 auto-labels reaches mAP50 0.715/0.460/0.059/0.271 on VOC/COCO/LVIS/BDD versus 0.756/0.496/0.087/0.434 with human labels. That the method is a viable alternative to standard labeling in many scenarios depends on the auto-labels being zero-shot, i.e., the foundation models have not seen the evaluation images during pre-training. Sec. 3.1 asserts 'there is no leakage between these datasets and the AL foundation models' but provides no image-dedup analysis. VOC, COCO, and LVIS images are drawn from Flickr, as are the pre-training corpora of YOLOW and YOLOE (Objects365, GQA, Flickr30k) and GDINO (GoldG, Cap4M); exact or near-duplicate images are therefore plausible, and LVIS uses COCO's image pool. If such duplicates exist, the reported mAP scores are inflated and the conclusion would not generalize to a user's own unlabeled images. BDD, the least likely contaminated set, shows the largest AL-vs-human gap (0.271 vs 0.434), which is consistent with contamination though not proof. Because a single leaked image set would directly invalidate the zero-shot premise, this unverified assertion is the most load-bearing point in the paper.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an auto-labeling pipeline for object detection: off-the-shelf vision-language foundation models (YOLO-World, YOLOE, Grounding DINO) are configured with a text-prompt set and a confidence threshold to generate pseudo ground-truth boxes for unlabeled images, and lightweight detectors (YOLO11 variants, RT-DETR) are then trained on these auto-labels. The authors evaluate the approach on VOC, COCO, LVIS, and BDD, comparing auto-labels to human labels via precision/recall/F1 and comparing downstream detection performance via mAP50. They report 445 training experiments across 169 unique label sets and conclude that a YOLO-World model with threshold 0.2 is the most reliable configuration, achieving mAP50 values of 0.715, 0.460, 0.059, and 0.271 on VOC, COCO, LVIS, and BDD, versus 0.756, 0.496, 0.087, and 0.434 with human labels, at a fraction of the estimated labeling cost.","tokens_in":19332,"tokens_out":4655,"duration_ms":44259,"significance":"If the findings hold, this is a practically valuable contribution: it provides a large-scale empirical benchmark for auto-labeling in object detection, compares three modern foundation models and multiple downstream architectures on four standard datasets, and gives concrete best-practice guidance that is actionable for practitioners. The paper's main strengths are the breadth of the experiments, the direct F1 comparison of auto-labels against human labels, the inclusion of challenging large-vocabulary and driving-domain datasets, and the honest acknowledgment that auto-labeling is not competitive on LVIS and BDD. However, the headline 'competitive performance' claim rests on evaluation-protocol choices and on an unverified zero-shot/leakage assumption, so the quantitative conclusions should be treated with caution until those points are addressed.","major_comments":[{"comment":"The sentence 'there is no leakage between these datasets and the AL foundation models' is load-bearing for the zero-shot premise, but no supporting evidence is provided. VOC, COCO, and LVIS draw images from Flickr-like web sources, and YOLOW/YOLOE are pretrained on Objects365, GQA, and Flickr30k while GDINO uses Objects365, GoldG, and Cap4M; LVIS in particular reuses the COCO image pool. Since exact or near-duplicate images could directly inflate the reported mAP differences in Tab. 11 and invalidate the generalization claim to a user's own unlabeled images, the authors should add an image-level deduplication analysis (e.g., exact hash plus feature-based near-duplicate checks) and report how many evaluation images are similar to pretraining images.","section":"3.1 (Dataset Selection)"},{"comment":"The recommended configuration YOLOW-0.2 is selected as the best-performing alpha based on validation mAP on the same splits that are later reported as final results. This makes the headline numbers (VOC 0.715, COCO 0.460, LVIS 0.059, BDD 0.271 in Tab. 11) optimistic, because the validation set has also been used for model selection. The authors should either withhold a held-out test set for final evaluation or use nested cross-validation, and they should explicitly describe the selection procedure so readers can assess the degree of overfitting.","section":"3.4-3.5, Tab. 6 and Tab. 11"},{"comment":"All headline comparisons are based on a single training run per configuration; no seeds or error bars are reported. For example, Tab. 5 lists YOLO11n VOC mAP50 of 0.756 (human labels) versus 0.715 (YOLOW auto-labels), but training from scratch for 100 epochs is stochastic, so this gap may be within run-to-run variation. Reporting mean and standard deviation over at least three seeds for the central Tab. 5 and Tab. 11 configurations is necessary to support the 'competitive performance' claim.","section":"3.4 (Downstream Inference Model Training)"},{"comment":"The cost and time savings in Tab. 4 depend on external estimates (7 seconds per box, $0.036 per box, $0.93 per hour for an L40S GPU) and on the assumption that human labeling time scales linearly with object count. These are point estimates, and the paper should include a sensitivity analysis (e.g., a range of annotation rates and GPU prices) so that the '1/100,000th cost' and '1/5,000th time' claims are not overstated.","section":"3.2 (Auto-Labeling Costs)"}],"minor_comments":[{"comment":"There are several typos in the introduction, including 'superivsed' and 'langauge', and the author affiliation lists 'V oxel51' with a stray space.","section":"1 (Introduction)"},{"comment":"In Tab. 11, the header 'Total Label' appears truncated, and the human-label total cost is printed as '$1,240,92.54' instead of '$124,092.54'.","section":"4 (Conclusions), Tab. 11"},{"comment":"Figure 6 uses a log scale for label counts but the caption does not explain how the three confidence thresholds are visually encoded; adding a short caption note or legend would improve readability.","section":"3.6, Fig. 6"},{"comment":"Equation (1) is written as an optimization over SA, but in practice the authors do not optimize SA directly; they select fA and alpha. Clarifying that Eq. (1) is a formal goal rather than a procedure that is actually optimized would avoid confusion.","section":"2.1, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The main obstacles to acceptance are the unverified no-leakage assertion and the selection-on-validation protocol, both of which directly affect the headline numbers. I do not see a fundamental flaw that would require rejection, but the current evidence is not sufficient for the strength of the claims; a deduplication analysis, a held-out validation procedure, and seed variance reporting would materially increase confidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a large, mostly honest empirical study of using VLM-generated pseudo-labels to train lightweight detectors. The core recipe isn't new — Nagase et al. already used VLM ensembles for VOC/COCO — but the scale, the newer backbones, and the finding that a low confidence threshold (high recall) beats high precision are genuinely useful. The paper deserves a serious referee, but the headline numbers are optimistic for two reasons the authors should fix.\n\nWhat's good: 445 training runs across three foundation models, six inference architectures, and four datasets is real work. The counter-intuitive result that alpha=0.2 consistently beats alpha=0.8 for downstream mAP, even though label precision is much lower, is the kind of practical insight practitioners need. The paper is also honest about the LVIS and BDD gaps; it doesn't claim universal superiority. The class-level analysis showing rare classes like 'rider' and 'motor' get zero labels on BDD is a useful caveat.\n\nSoft spots, in order of severity:\n\n1. Selection on the same validation sets. The recommended YOLOW-0.2 configuration is chosen based on validation mAP (Sec. 3.4 and 3.5) and then reported as the final result in Tab. 11. That is selection on the test set, so the reported gaps to human labels (VOC 0.715 vs 0.756, etc.) are optimistic. The pattern is consistent enough that the qualitative conclusion likely survives, but a held-out split or nested validation is needed before taking the exact numbers at face value.\n\n2. The 'no leakage' assertion in Sec. 3.1 is unsupported. VOC, COCO, and LVIS draw heavily from Flickr, and YOLOW/YOLOE and GDINO were pre-trained on Objects365, GQA, Flickr30k, GoldG, and Cap4M. Without a near-duplicate analysis, the zero-shot premise isn't established. The stress-test note is right that BDD, the least likely contaminated set, shows the largest AL-vs-human drop; that is consistent with contamination, though not proof. The authors should run dedup or, at minimum, discuss the risk.\n\n3. No error bars or multiple seeds, and no code or labels released. Minor for a study this size, but it limits reproducibility.\n\nThe cost comparison relies on a 7 s/box estimate and a $0.036/box service price, which are reasonable order-of-magnitude numbers; the paper is transparent about where they come from.\n\nBottom line: the central claim — auto-labeling is viable on VOC/COCO-like data — holds up on the evidence presented, with caveats. This is a paper for practitioners and for researchers working on annotation-free detection. Send it to peer review, but require the authors to address the validation-selection issue and provide a leakage analysis. I'd bring it to reading group, mostly to debate the evaluation protocol.","headline":"Large, mostly honest empirical study of VLM auto-labeling with a useful low-threshold finding, but the headline numbers are selected on the same validation sets and the 'no leakage' claim needs evidence.","tokens_in":19888,"tokens_out":4458,"would_cite":true,"duration_ms":37583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that object detectors can be trained to near-human accuracy using pseudo labels generated by vision-language foundation models, cutting labeling cost by roughly five orders of magnitude.","keywords":["auto-labeling","object detection","vision-language models","pseudo ground truth labels","confidence threshold","YOLO-World","annotation cost","downstream model training"],"falsifier":"Run the same pipeline but choose the foundation model and confidence threshold α using only a held-out portion of each training set, then evaluate the trained detectors on untouched validation sets; if YOLOW-0.2 no longer lands within the reported 4–7% of human-label mAP50 on VOC and COCO, the viability claim is weakened.","tokens_in":18821,"feed_emoji":"🏷️","tokens_out":6518,"duration_ms":52463,"temperature":0.7,"pith_summary":"The paper tries to establish that object detectors can be trained without any human annotation by using pre-trained vision-language foundation models to generate pseudo \"ground truth\" labels, a process it calls auto-labeling. If true, this matters because conventional labeling of large detection datasets costs tens of thousands of dollars and thousands of hours, while auto-labeling the same data on a single GPU costs about a dollar and takes a couple of hours. The authors train lightweight detectors on these auto-labels across four datasets and report that the best configuration (YOLO-World at a 0.2 confidence threshold) reaches mAP50 of 0.715 on VOC and 0.460 on COCO, within roughly 4–7% of the same detectors trained on human labels. On harder domains—large-vocabulary LVIS and driving-scene BDD—the gap is larger, but the paper argues the near-zero cost still makes auto-labels the best starting point for most detection applications.","feed_headline":"Auto-labels rival human annotations at 1/100,000th the cost","feed_subtitle":"YOLO11n on YOLO-World pseudo-labels hits 0.715 mAP50 on VOC, near 0.756 from human labels.","key_machinery":"The engine of the approach is the zero-shot label generator $f^A(x_i, \\alpha, T)$: a pre-trained open-vocabulary detection model (YOLO-World, YOLOE, or Grounding DINO-T) that, given a set of class text prompts $T$ and a fixed confidence threshold $\\alpha$, emits bounding boxes and class indices as pseudo ground truth. The threshold $\\alpha$ is the lever that trades precision against recall, and the paper's core mechanistic finding is that downstream detector performance is best at low-to-mid $\\alpha$ (around 0.2), where recall is high even though label precision is modest, while high-precision settings ($\\alpha=0.8$) reliably degrade the trained detector. The downstream evaluation chain—training lightweight YOLO11 and RT-DETR models on the auto-labels with no pre-trained weights—is what converts label-quality metrics into the practical claim of viability.","core_discovery":"The central claim is that auto-labeling is a viable replacement for standard label acquisition in many object detection scenarios. Concretely, a YOLO11n trained on YOLO-World labels generated with confidence threshold α=0.2 attains validation mAP50 of 0.715 on VOC, 0.460 on COCO, 0.059 on LVIS, and 0.271 on BDD, versus 0.756, 0.496, 0.087, and 0.434 for the same model trained on human labels. The paper further reports that auto-labeling all four training sets costs about 1.27 GPU-hours and $1.18, compared with roughly 6,700 human labeling hours and $124,000 for an annotation service. A single configuration, YOLO-World with α=0.2, gives the most reliable downstream performance across all datasets, and across the measured settings downstream mAP tracks auto-label recall more than precision or even F1 score relative to human labels.","pith_inferences":["The reported cost and time figures assume a single forward pass on a rented L40S GPU; any human-in-the-loop verification or iterative labeling would add cost, though even a 100× cost increase stays far below annotation services.","The selection of α on the same validation splits used for reporting likely makes the headline gaps optimistic; a held-out protocol for choosing YOLOW-0.2 could widen the gap, especially on BDD where the difference is already 0.163 mAP50.","The class-level analysis suggests a testable fix: feeding more descriptive prompts for rare classes (e.g., 'rider' on BDD, 'car battery' on LVIS) should recover some of the unlabeled instances and shrink the downstream gap.","The same label-generation pipeline is a natural first step toward annotation-free instance segmentation, since bounding boxes from auto-labelers can bootstrap mask generators."],"forward_implications":["For common-object benchmarks like VOC and COCO, auto-labeling can substitute for human annotation with a 4–7% mAP50 penalty at roughly 1/100,000th of the labeling cost.","Redirecting the cost savings to a larger inference model can outperform a small model trained on human labels (e.g., YOLO11s with auto-labels beats YOLO11n with human labels by 2–5% on VOC and 9–11% on COCO).","Practitioners should set the confidence threshold low (α≈0.2) to favor recall, because high-precision settings produced the worst downstream model on every dataset tested.","For out-of-distribution or very large vocabulary applications (BDD, LVIS), auto-labeling still yields cheap working detectors, but the gap to human labels is larger and may require prompt engineering or verification.","Because the auto-labeling output has the same format as human labels, it drops directly into existing training pipelines and improves automatically as foundation models improve."],"supporting_citations":[{"why":"Supplies YOLO-World, the foundation model whose α=0.2 labels give the best downstream results in the paper.","marker":"[4]"},{"why":"Supplies YOLOE, the second auto-labeling foundation model and the best performer on LVIS.","marker":"[31]"},{"why":"Supplies Grounding DINO-T, the third auto-labeling model, whose labels run 12–300× slower.","marker":"[21]"},{"why":"Provides the VOC dataset and the mAP50 evaluation protocol used for the main comparisons.","marker":"[7]"},{"why":"Provides the COCO dataset, the moderate-complexity benchmark in the study.","marker":"[20]"},{"why":"Provides LVIS, the 1,203-class dataset that stresses auto-labeling recall.","marker":"[10]"},{"why":"Provides BDD, the autonomous-driving dataset whose viewpoint gap is the hardest for auto-labelers.","marker":"[35]"},{"why":"Provides the YOLO11 inference architectures, including the YOLO11n used for headline comparisons.","marker":"[14]"},{"why":"Provides RT-DETR, the transformer-based inference architecture used to test whether conclusions transfer across architectures.","marker":"[38]"},{"why":"The closest prior work that trains detectors on VLM-generated labels; this paper extends it with a broad benchmark and single-model configurations.","marker":"[23]"}],"fun_headline_variants":["Auto-labeling enables detector training with no human labels","Pseudo-labels from vision-language models rival human annotations","Zero-cost auto-labels train detectors near human-label accuracy","Auto-labeling cuts annotation cost by 100,000x for detection","Vision-language auto-labels match human labels for object detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported competitive gaps come from picking the auto-labeling configuration on the same validation splits used to report the numbers, so the real-world gap to human labels could be larger if the configuration had to be chosen on held-out data.","fun_headline_variants_meta":{"raw":{"variants":["Auto-labeling enables detector training with no human labels","Pseudo-labels from vision-language models rival human annotations","Zero-cost auto-labels train detectors near human-label accuracy","Auto-labeling cuts annotation cost by 100,000x for detection","Vision-language auto-labels match human labels for object detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3188,"prompt_tokens":940,"completion_tokens":2248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2165}},"tokens_in":556,"tokens_out":2248,"duration_ms":15031,"temperature":1.0,"reasoning_tokens":2165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:25:12.271623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline but choose the foundation model and confidence threshold α using only a held-out portion of each training set, then evaluate the trained detectors on untouched validation sets; if YOLOW-0.2 no longer lands within the reported 4–7% of human-label mAP50 on VOC and COCO, the viability claim is weakened.","supporting_citations":[{"cited_title":"Ultralytics yolo11","cited_arxiv_id":null,"evidence_quote":"Provides the YOLO11 inference architectures, including the YOLO11n used for headline comparisons."},{"cited_title":"Yolo-world: Real-time open-vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"Supplies YOLO-World, the foundation model whose α=0.2 labels give the best downstream results in the paper."},{"cited_title":"Grounding dino: Marry- ing dino with grounded pre-training for open-set object de- tection","cited_arxiv_id":null,"evidence_quote":"Supplies Grounding DINO-T, the third auto-labeling model, whose labels run 12–300× slower."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VOC dataset and the mAP50 evaluation protocol used for the main comparisons."},{"cited_title":"Lawrence Zitnick","cited_arxiv_id":null,"evidence_quote":"Provides the COCO dataset, the moderate-complexity benchmark in the study."},{"cited_title":"Lvis: A dataset for large vocabulary instance segmentation","cited_arxiv_id":null,"evidence_quote":"Provides LVIS, the 1,203-class dataset that stresses auto-labeling recall."},{"cited_title":"Detrs beat yolos on real-time object detection","cited_arxiv_id":null,"evidence_quote":"Provides RT-DETR, the transformer-based inference architecture used to test whether conclusions transfer across architectures."},{"cited_title":"Annotation-free object detection by knowledge-extraction training from visual-language models","cited_arxiv_id":null,"evidence_quote":"The closest prior work that trains detectors on VLM-generated labels; this paper extends it with a broad benchmark and single-model configurations."}],"review_version":1}