{"id":"a01ee935-e827-4d64-aa4e-7e697cf997b2","arxiv_id":"2501.09267","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On a robot-collected construction-site dataset of 10 MEP classes, fine-tuned YOLO11 Nano achieved F1 0.89 while zero-shot GSAM2, GDINO, and DETIC scored below 0.04.","lead":"This paper compares fine-tuned YOLO11 Nano with three zero-shot open-vocabulary detectors for finding MEP components on construction sites using robot-collected images. The fine-tuned model scored F1 0.89 while the best open-vocabulary model scored 0.03, a large gap favoring specialized detectors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-vocabulary models are compared with undisclosed prompts and confidence thresholds, so the near-zero F1 values in Table 2 may reflect evaluation choices rather than model capability.","rationale":"The reader's weakest assumption identifies exactly the most load-bearing gap in the paper: the open-vocabulary models are evaluated zero-shot, but the prompts and thresholds that define 'zero-shot' are never reported. Because the paper's headline numbers for Grounding SAM2, Grounding DINO, and DETIC are near zero, even a moderate improvement from prompt tuning or threshold selection would change the magnitude of the reported gap, and a large improvement would change the practical conclusion. The paper does not provide enough information to rule this out, so the conditional verdict is appropriate. I do not see the missing details as fatal: a properly specified reproduction could easily confirm the qualitative finding, since fine-tuned domain detectors are expected to beat zero-shot generalists on specialized vocabularies. However, as written, the comparison is under-specified at exactly the point where open-vocabulary models are most sensitive to input configuration. The reader's condition that these details must be reported is correct, and the verdict should remain CONDITIONAL until the reproduction check is run.","tokens_in":4895,"tokens_out":4611,"duration_ms":55009,"concrete_test":"Reproduce Table 2 using the released dataset and, for each open-vocabulary model, sweep confidence thresholds over [0.01, 0.5] and test at least five prompt variants per class, including the exact class names from Table 1, 'construction MEP equipment', and per-class paraphrases; select thresholds on the validation split and report the best F1 per model on the test split. If the best F1 remains below 0.1, the central conclusion stands; if it rises above 0.3 or approaches YOLO11's 0.89, the paper's 'not ready' claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim rests on comparing a fine-tuned detector (YOLO11 Nano) against three pre-trained open-vocabulary models in a zero-shot setting. Section 2 (Evaluation Procedure) states only that the open-vocabulary models were used without fine-tuning and that metrics are computed at IoU 0.5; it does not report the text prompts, class-name encoding, or detection confidence thresholds for Grounding SAM2, Grounding DINO, or DETIC. For open-vocabulary detectors, these choices are not incidental: Grounding DINO and Grounded SAM2 require text prompts and box/text thresholds that strongly control the precision-recall trade-off, and DETIC requires a vocabulary list. If the authors used generic prompts (e.g., \"MEP elements\") or default thresholds inappropriate for this domain, the near-zero F1 values could substantially underestimate what these models can achieve with construction-specific prompts or a small amount of prompt tuning. The abstract and conclusion's broad statement that open-vocabulary models are not ready for MEP detection therefore rests on an under-specified baseline. Since the paper itself suggests prompt design and fine-tuning as future work, the evidence as reported only supports a narrower claim: default, un-tuned open-vocabulary models underperform a fine-tuned domain detector. The gap is large, so the qualitative direction likely survives, but the exact F1 values and the \"ready\" framing are conditional on missing evaluation details.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether open-vocabulary detection models are ready for detecting MEP (mechanical, electrical, plumbing) components on construction sites. The authors collected a real-world dataset using a mobile ground robot, annotated 10 MEP component classes, and compared a fine-tuned YOLO11 Nano detector against three pre-trained open-vocabulary models (Grounding SAM2, Grounding DINO, and DETIC) used without fine-tuning. The reported results show a very large performance gap: YOLO11 Nano achieves an overall F1 of 0.89, while the open-vocabulary models achieve F1 scores between 0.014 and 0.032 on the same test split. The paper concludes that fine-tuned lightweight models still largely outperform open-vocabulary models in specialized environments and domain-specific tasks.","tokens_in":5298,"tokens_out":1708,"duration_ms":16714,"significance":"If the comparison is taken at face value, the paper provides a useful data point for practitioners in construction robotics: for this specific MEP detection task, a 2.6M-parameter fine-tuned detector massively outperforms much larger zero-shot open-vocabulary models. The main strength is the independently collected, annotated, and publicly shared dataset, which is a valuable resource for future work. The paper also reports deployment-relevant metrics (model size, inference speed on an embedded platform). However, the central quantitative claim rests on an under-specified evaluation protocol: the open-vocabulary models are used zero-shot with undisclosed prompts and confidence thresholds, no error bars are reported, and no attempt is made to tune or adapt the open-vocabulary models. The qualitative direction of the result is likely robust given the size of the gap, but the exact F1 values and the broad 'not ready' framing are conditional on missing details.","major_comments":[{"comment":"The evaluation procedure does not report the text prompts, class vocabularies, or detection confidence thresholds used for Grounding SAM2, Grounding DINO, and DETIC. For open-vocabulary detectors these choices strongly control the precision-recall trade-off and can change reported F1 by orders of magnitude. Without this information, the near-zero F1 values in Table 2 cannot be interpreted as a fair representation of what these models can achieve for MEP detection. The manuscript should specify the exact prompt strings (e.g., 'pipe fitting', 'cable tray fitting', or a combined prompt), the box/text thresholds, and any post-processing filters. If the authors used generic prompts or default thresholds, the conclusion should be narrowed to 'default, untuned open-vocabulary models' rather than 'open-vocabulary models are not ready'.","section":"Section 2, Evaluation Procedure; Table 2"},{"comment":"No error bars, repeated runs, or statistical significance tests are reported. The test split contains relatively few instances for some classes (e.g., Generator has 124 instances, Cable Tray Fitting has 317), so per-class F1 values such as 0.0 vs 0.005 may not be meaningful. Given the small class sizes and the single train/validation/test split, the authors should provide confidence intervals or at least report the number of true positives/false positives underlying the aggregate F1 scores.","section":"Section 3, Results and Discussion; Table 2"},{"comment":"The conclusion states that fine-tuned models 'significantly outperformed open-vocabulary models in detection accuracy, computational efficiency, and real-time applicability' and that open-vocabulary performance 'remains limited.' This overgeneralizes because the open-vocabulary models were only evaluated zero-shot with undisclosed prompts and thresholds, and no prompt engineering, prompt tuning, or fine-tuning was attempted. The paper's own future-work suggestions (fine-tuning and construction-related prompts) indicate that the evidence supports a narrower claim: a fine-tuned lightweight detector outperforms untuned open-vocabulary models in this specific setup. The abstract and conclusion should be revised to reflect this scope.","section":"Section 4, Conclusion"}],"minor_comments":[{"comment":"The statement 'The dataset is shared online' does not include a link or repository identifier; the reader cannot access the data without further searching.","section":"Section 2, Dataset"},{"comment":"The manuscript reports inference speed only for YOLO11 Nano (23.36 FPS on Jetson Orin Nano) and does not report inference times for the three open-vocabulary models on the same hardware. The claim that open-vocabulary models are 'less practical' for real-time use would be stronger with at least approximate runtime comparisons.","section":"Section 3, Real-world Applicability"},{"comment":"The sentence 'the model outperformed human annotations' is not supported by a quantitative measure; it may be more precise to say that the model detected objects that the annotators had missed, which is a qualitative observation.","section":"Section 3, Qualitative examples"},{"comment":"The row labeled 'All' reports total images as 8885, but the sum of per-class image counts exceeds this number because images contain multiple classes; this is fine, but the table caption should clarify that 'Images' counts are not disjoint across classes.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical comparison and the dataset contribution is genuine. The main concern is that the evaluation protocol for open-vocabulary models is under-specified, which weakens the central claim's generalizability. This is fixable with a revision that reports prompts, thresholds, error bars, and a more cautious conclusion. The paper is within scope for a vision/robotics venue, though the novelty is modest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper gives you a new, robot-collected MEP detection dataset (8885 images, 14k instances, 10 classes) and a benchmark that says fine-tuned YOLO11 Nano (2.6M params) beats zero-shot Grounding SAM2, Grounding DINO, and DETIC by enormous margins (F1 0.89 vs 0.02–0.03). I think the direction of that result is real — a fine-tuned domain detector should beat generic zero-shot models on a specialized task with unusual classes like valve, cable tray fitting, fire alarm detector. The dataset itself is the main contribution, and the real-world collection on a ground robot makes it more useful than yet another curated web scrape.\n\nThe paper does several things well. It reports per-class results, so you can see where open-vocabulary models collapse entirely (cable tray fitting: 0 for all). It compares model sizes honestly and checks real-time inference on a Jetson. It is also honest in the conclusion, noting that prompt design and fine-tuning of open-vocabulary models are future work.\n\nThe soft spots are concentrated in the open-vocabulary evaluation, and the stress-test note is right. Section 2 says only that the three models were used without fine-tuning; it never reports the text prompts, the vocabulary list for DETIC, or the box/text confidence thresholds. For Grounding DINO and Grounded SAM2, those thresholds control the precision–recall trade-off almost completely. If the authors used generic prompts or default thresholds, the near-zero F1 values in Table 2 may understate what these models can do with construction-specific prompts or a little prompt tuning. So the exact numbers are conditional, even though the qualitative story likely survives. The paper's own \"future work\" acknowledges this. Minor additions: no error bars (though the gap is so large it probably doesn't matter), no details on the annotation quality, and no dataset URL despite saying it is shared online.\n\nWho is this for? Construction robotics teams deciding whether to invest in a fine-tuned small detector versus trying an open-vocabulary model for MEP monitoring. They will get a clear, if under-specified, warning. A serious referee should engage with it, but the revision must report prompts, thresholds, and ideally a prompt-tuned baseline before the \"not ready\" framing is fully supported.\n\nRecommendation: send to peer review with a request for those details. The dataset and the benchmark are worth refereeing; the current reporting is not enough to take the F1 numbers at face value.","headline":"New robot-collected MEP dataset and benchmark showing fine-tuned YOLO beats zero-shot open-vocabulary models by a huge margin, but the open-vocabulary baselines are under-specified (no prompts, thresholds, or dataset URL).","tokens_in":5690,"tokens_out":2648,"would_cite":false,"duration_ms":40063,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned YOLO11 Nano outperforms open-vocabulary models for MEP detection on construction sites.","keywords":["MEP detection","open-vocabulary object detection","vision-language models","construction robotics","YOLO11","fine-tuning","ground robot","construction site monitoring"],"falsifier":"Run the same test images through Grounding DINO, Grounded SAM2, and DETIC again, this time with construction-specific prompts naming each of the 10 MEP classes and with confidence thresholds tuned on the validation split; if any of these models then reaches F1 near YOLO11 Nano's 0.89, the paper's central claim is contradicted. The simplest version: report the prompt texts and thresholds used, and show whether the reported F1 numbers change.","tokens_in":4716,"feed_emoji":"🏗️","tokens_out":7761,"duration_ms":65027,"temperature":0.7,"pith_summary":"Construction-site robots need to spot mechanical, electrical, and plumbing (MEP) components reliably, and open-vocabulary vision-language models promise to do this without task-specific training. This paper tests that promise with a dataset of 8,885 images collected by a ground robot on an active construction site, annotated across 10 MEP classes. A fine-tuned YOLO11 Nano detector with 2.6 million parameters reaches an overall F1 of 0.89, while three pre-trained open-vocabulary detectors — Grounded SAM2, Grounding DINO, and DETIC — used without fine-tuning score 0.018, 0.032, and 0.014. The authors conclude that fine-tuned lightweight detectors still substantially outperform open-vocabulary models in specialized environments, and that the open-vocabulary approach is not yet ready for MEP monitoring on construction sites.","feed_headline":"YOLO11 Nano beats open-vocabulary models on MEP detection","feed_subtitle":"On robot-captured site images, a 2.6M-parameter detector hits F1 0.89; zero-shot rivals stay below 0.04.","key_machinery":"The benchmark itself is the mechanism: a single robot-collected dataset, split into 70% training, 20% validation, and 10% testing, with 10 MEP classes and 14,064 annotated instances, and a fixed evaluation protocol of precision, recall, and F1 at 0.5 intersection over union on the test split. The central comparison is YOLO11 Nano, a closed-set detector fine-tuned for 200 epochs, versus three open-vocabulary models — Grounded SAM2, Grounding DINO, and DETIC — evaluated in their pre-trained, zero-shot form. Open-vocabulary here means the models can in principle detect objects named in text even if those classes were not seen during training, which is what makes the failure on specialized MEP parts informative. The paper also uses parameter counts to frame the gap: YOLO11 Nano has 2.6 million parameters, while Grounded SAM2 has 910 million, so the accuracy advantage also comes with a large computational-efficiency advantage.","core_discovery":"On a real-world dataset gathered from a teleoperated ground robot moving through an active construction site, the paper shows that a task-specific detector trained with just 2.6 million parameters can detect all ten MEP classes with F1 0.89, precision 0.87, and recall 0.901 at 0.5 IoU. Under the same test conditions, Grounded SAM2, Grounding DINO, and DETIC — all used pre-trained and without fine-tuning — reach F1 0.018, 0.032, and 0.014, and fail entirely on several classes such as cable tray fittings. The paper claims that this gap reflects a current limitation of open-vocabulary models on domain-specific, specialized visual tasks, and that the practical choice for real-time robotic MEP monitoring remains a fine-tuned lightweight detector.","pith_inferences":["The reported gap is specifically against zero-shot, default-configuration use of the open-vocabulary models; the paper does not report prompt texts or confidence thresholds, so a prompt-engineered or lightly fine-tuned version of these models could plausibly narrow the gap on the same test set.","If that happened, the practical takeaway would shift toward a hybrid: a fast fine-tuned detector for known MEP classes plus an open-vocabulary model as a fallback for novel or rare components.","Because the dataset is shared, one can track future vision-language models against the same 10 classes; a reasonable target would be matching YOLO11's F1 without per-class fine-tuning, which would amount to a definition of 'ready for MEP detection.'","The single-site data collection leaves open whether the fine-tuned model's advantage persists across different construction sites, lighting conditions, and camera viewpoints; that cross-site generalization question is not addressed by the paper."],"forward_implications":["For construction-site MEP monitoring with mobile robots, the results imply that a fine-tuned lightweight detector is the currently reliable option, at least until open-vocabulary models are adapted to the domain.","Because YOLO11 Nano runs at 23.36 FPS on an NVIDIA Jetson Orin Nano, the approach fits the compute budget of an onboard robot system, whereas the largest open-vocabulary model (910M parameters) is far harder to deploy on such hardware.","The open-vocabulary models' near-zero scores on classes like cable tray fittings indicate that generic pretraining data (COCO, LVIS) does not transfer to MEP components, so gains will likely require domain-specific data rather than larger generic models alone.","On the dataset released with the paper, future open-vocabulary models can be measured against these numbers, giving the community a concrete baseline to beat."],"supporting_citations":[{"why":"Supplies the fine-tuned detector (YOLO11 Nano) that sets the benchmark performance.","marker":"[12]"},{"why":"One of the three open-vocabulary models evaluated (Grounded SAM2), representing the largest model tested.","marker":"[13]"},{"why":"Provides Grounding DINO, the open-vocabulary model used as the main comparison baseline in the results.","marker":"[14]"},{"why":"Provides DETIC, the open-vocabulary detector with the weakest measured performance in the comparison.","marker":"[15]"},{"why":"COCO is the pre-training dataset for YOLO11 and one of the generic datasets whose class coverage is contrasted with MEP-specific components.","marker":"[16]"},{"why":"LVIS is cited as a generic dataset used by open-vocabulary models, whose lack of MEP classes is used to explain the low performance.","marker":"[17]"}],"fun_headline_variants":["YOLO11 Nano hits F1 0.89 on MEP; zero-shot models under 0.04","Fine-tuned YOLO11 Nano dominates zero-shot vision-language on MEP","For robot MEP detection, YOLO11 Nano beats zero-shot VLMs by 28x F1","Task-specific YOLO11 Nano beats open-vocabulary models by 28x on MEP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that running each open-vocabulary model with its default settings and no special wording is a fair test of what it can do for MEP detection.","fun_headline_variants_meta":{"raw":{"variants":["YOLO11 Nano hits F1 0.89 on MEP; zero-shot models under 0.04","Fine-tuned YOLO11 Nano dominates zero-shot vision-language on MEP","For robot MEP detection, YOLO11 Nano beats zero-shot VLMs by 28x F1","Task-specific YOLO11 Nano beats open-vocabulary models by 28x on MEP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001284,"raw_usage":{"total_tokens":5206,"prompt_tokens":863,"completion_tokens":4343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":4238}},"tokens_in":479,"tokens_out":4343,"duration_ms":27434,"temperature":1.0,"reasoning_tokens":4238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:05:49.965422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same test images through Grounding DINO, Grounded SAM2, and DETIC again, this time with construction-specific prompts naming each of the 10 MEP classes and with confidence thresholds tuned on the validation split; if any of these models then reaches F1 near YOLO11 Nano's 0.89, the paper's central claim is contradicted. The simplest version: report the prompt texts and thresholds used, and show whether the reported F1 numbers change.","supporting_citations":[{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Provides Grounding DINO, the open-vocabulary model used as the main comparison baseline in the results."},{"cited_title":"Detecting twenty- thousand classes using image-level supervision","cited_arxiv_id":null,"evidence_quote":"Provides DETIC, the open-vocabulary detector with the weakest measured performance in the comparison."},{"cited_title":"Microsoft coco: Com- mon objects in context","cited_arxiv_id":null,"evidence_quote":"COCO is the pre-training dataset for YOLO11 and one of the generic datasets whose class coverage is contrasted with MEP-specific components."}],"review_version":1}