{"id":"5c28ae65-5c8c-4918-9a15-141a78e1b541","arxiv_id":"2501.07396","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A YOLO-world detector plus a vision-language model can label military vehicle crops zero-shot, but post-hoc label alignment and missing baselines make the headline numbers unreliable.","lead":"This paper pairs an open-world detector with large vision-language models to identify military vehicles without training on those classes, testing 13 models across RGB, thermal, and synthetic imagery. The empirical comparison offers practical hints, but the evaluation method inflates the reported accuracy and leaves key baselines unmeasured.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-set accuracy rests on a post-hoc keyword-to-class mapping fitted to the test set; because the label vocabulary is chosen after seeing ground-truth labels, the reported zero-shot accuracies are not independent and may be substantially inflated.","rationale":"The reader's weakest assumption identifies the same fundamental issue: the open-set accuracy numbers are produced by a mapping fitted to the test set after model inference. This is not an independent zero-shot evaluation. Section III-B explicitly states that the strategy was to select the most recurring keyword from the reevaluation labels corresponding to each ground truth class. That procedure can only be executed with access to ground-truth class assignments, and it can reward a model for consistently outputting any label, correct or not. The reported accuracies of 70-73% for API models are therefore not trustworthy as evidence for zero-shot recognition of novel classes. Additional issues, such as the detector's recall never being measured and the absence of error bars, further weaken the practical claims, but the metric problem is the most load-bearing because it directly invalidates the central quantitative results. The paper's qualitative findings, such as chain-of-thought helping and API models outperforming smaller open-source models, may still be plausible, but they do not rescue the headline quantitative claim. The verdict should remain REJECT pending a re-evaluation with a fixed label vocabulary and detector-aware metrics.","tokens_in":12577,"tokens_out":2650,"duration_ms":29052,"concrete_test":"Recompute the open-set accuracy with a fixed label mapping chosen before seeing the test set: for example, exact match to the ground-truth class name, plus a synonym dictionary built from WordNet or a military vehicle glossary. Separately, run the same post-hoc 'most recurring keyword per class' procedure on uniformly random labels over the same output vocabulary; if this random baseline exceeds, say, 20-30%, the metric's inflation is confirmed. Report both values for Tables I-IV.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the evaluation protocol for open-set recognition in Section III-B. To score free-text outputs, the authors select, for each ground-truth class, the most recurring keyword appearing in the model's reevaluation labels, and then use that keyword as the accepted label for the class. Because this mapping is derived from the test set after the fact, the reported 'zero-shot' accuracies (e.g., 70-73% for API models in Tables II and III) are not independent predictions: the metric effectively chooses the best per-class response pattern from the test data. A model that emits a consistent but wrong label for a class would still be counted correct. This inflates all open-set numbers and undermines the central claim that the LVLMs recognize novel military vehicle classes in a zero-shot manner. The closed-set numbers are less affected, but the headline open-set results and the 'robust zero-shot ATR' conclusion rest on this post-hoc alignment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage pipeline for zero-shot automatic target recognition (ATR): YOLO-world is used for binary detection with the prompt keyword 'vehicle', and a large vision-language model (LVLM) labels each detected crop under one of three settings (open-set, closed-set, or chain-of-thought). The authors benchmark twelve LVLMs plus CLIP on four datasets (ADAS, AIS, DSIAC, and weather-degraded AIS) and report accuracy tables to support claims that the pipeline enables zero-shot recognition of novel military vehicle classes across modalities and distances.","tokens_in":12781,"tokens_out":5568,"duration_ms":51836,"significance":"If the quantitative results were valid, the paper would offer a useful, systematic comparison of LVLMs for military-vehicle recognition under domain shift, and the proposed division of labor (an open-vocabulary detector for localization, an LVLM for identification) is a reasonable design. The breadth of models and conditions, the qualitative demonstration of false-positive removal via LVLMs, and the chain-of-thought analysis are potentially valuable contributions. However, the open-set accuracy metric is computed through a post-hoc label-to-class mapping derived from the test ground truth, so the headline zero-shot numbers are not independent predictions, and the detection stage is never evaluated quantitatively. As a result, the central claim of a robust zero-shot ATR system is not supported by the evidence as presented.","major_comments":[{"comment":"The open-set recognition accuracy is computed by 'selecting the most recurring keyword from the reevaluation labels corresponding to each ground truth class.' This is a post-hoc mapping from the model's free-text outputs to ground-truth classes, inferred from the test set after the predictions are made. Consequently, the open-set accuracies in Tables I-IV are not zero-shot evaluations: a model that consistently emits a plausible but different label (e.g., 'boat' for 'tank') would be scored as correct, and the numbers are not comparable to standard fixed-vocabulary or exact-match metrics. Because the open-set results are the primary evidence for the paper's claim of zero-shot recognition of novel classes, this evaluation protocol must be redone with a mapping fixed in advance (e.g., a predefined synonym dictionary that does not use test labels) or replaced by a proper open-set recognition metric (e.g., exact match or semantic similarity with a fixed threshold).","section":"III-B"},{"comment":"The paper reports only recognition accuracy on the crops produced by the detector; it never reports detection precision, recall, missed-detection counts, or end-to-end system accuracy. Since the pipeline's overall ATR performance is bounded by the detection stage, the claim that the pipeline is 'a robust system for zero-shot ATR' (Abstract) is not quantitatively supported. Please provide per-dataset detection recall (and precision) for the binary 'vehicle' detection, and ideally report end-to-end recognition accuracy computed over all ground-truth objects rather than only over detected crops.","section":"V and IV-A"},{"comment":"The statement that binary detection with the keyword 'vehicle' 'produced the same level of localization performance as keyword detection' is supported only by qualitative examples in Fig. 4. A quantitative comparison (e.g., IoU or recall at a fixed confidence threshold) is needed to justify this design choice and to assess how the detector prompt affects downstream recognition. Without such measurements, the effect of the detection stage on the reported system performance is unknown.","section":"III-A"},{"comment":"The numerical results are presented without any measure of variance, number of repeated trials, or statistical significance. Because LVLM outputs are stochastic (especially for API models) and the open-set metric is sensitive to the choice of the keyword mapping, the authors should report means and standard deviations over multiple runs or at least describe the number of trials and the exact scoring procedure.","section":"Tables I-IV"}],"minor_comments":[{"comment":"The text contains a garbled sentence: 'Furthermore, for CoT recognition we use the following prompt of,For the closed-set recognition, we use the prompt,' which appears to be a duplication or editing error and should be corrected.","section":"III-B"},{"comment":"The header for the 3000-5000 m range lists 'Chain-of-Thought Open-set Closed-set' twice, making the column alignment unclear and the table hard to interpret.","section":"Table III"},{"comment":"The model name 'LLaV A' appears throughout; it should be 'LLaVA'.","section":"Throughout"},{"comment":"The exact number of test samples per dataset and condition, and the complete verbatim prompts for all three recognition settings, are not specified; please include them for reproducibility.","section":"IV"},{"comment":"Reference [30] is cited as a CLIP reference but is actually a Nature Reviews Methods Primers article; the CLIP paper is [29], so the citation appears misdirected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The open-set evaluation flaw is severe and goes to the heart of the paper's contribution; I would emphasize to the authors that the current open-set numbers cannot be used as zero-shot evidence. The absence of detection-stage metrics also needs to be addressed for any resubmission. The pipeline idea is worth pursuing, but the quantitative claims must be resubstantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a useful empirical sweep with a load-bearing evaluation flaw. The authors cascade YOLO-world (prompted with 'vehicle' for binary detection) with 13 LVLMs to label military vehicles in RGB, thermal, synthetic, and long-range imagery. The breadth is genuinely new: no prior work compares this many LVLMs on this kind of ATR data. The qualitative findings — API models beat open-source ones, chain-of-thought prompting helps, and performance degrades with distance — are plausible and worth having.\n\nThe problem is the open-set metric. In Section III-B the authors score free-text labels by selecting, for each ground-truth class, the most recurring keyword from the model's own outputs and then treating that keyword as the accepted label. That mapping is fitted to the test set after the fact. A model that consistently emits a wrong label for a class is still counted correct. So the 70-73% open-set accuracies in Tables II and III are not independent zero-shot measurements; they are permissive upper bounds. The closed-set numbers are less affected, but the paper's central claim — zero-shot ATR of novel classes — rests on the open-set results.\n\nThe detection stage is never quantified. They claim binary detection matches keyword detection for localization, but there are no precision/recall numbers, no analysis of missed crops, and no error bars on any accuracy table. The AIS dataset has only 200 test images, and some condition definitions are ambiguous.\n\nCredit where it's due: the false-positive filtering idea (Fig. 5) is a nice practical contribution, and the CoT qualitative example is instructive. The paper is clearly written and the authors are honest about limitations. But the evaluation protocol is load-bearing.\n\nBottom line: this deserves a serious referee, not a desk reject. The empirical effort is real and the pipeline is sensible, but the open-set metric needs to be redone with a fixed label vocabulary or exact matching, detection metrics need to be reported, and artifacts released. I'd send it back for major revision.\n\nRecommendation: engage with the paper, but don't cite the open-set numbers until they're recomputed.","headline":"Solid empirical sweep of LVLMs for military vehicle ATR, but the open-set accuracy metric is fitted post-hoc to the test set, so the headline zero-shot numbers are not trustworthy.","tokens_in":13274,"tokens_out":2425,"would_cite":false,"duration_ms":21612,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-world detector paired with a vision-language model labels military vehicles never seen in training.","keywords":["automatic target recognition","zero-shot learning","large vision-language models","open-world object detection","YOLO-world","chain-of-thought prompting","military vehicle recognition","thermal imagery"],"falsifier":"Fix the label-to-class mapping before inference: provide a predefined alias list for the eight classes in the long-range military dataset, run the open-set prompt, and score exact or fixed-list matches. If accuracy falls materially below the reported 67 to 73 percent at close range, the post-hoc keyword alignment is responsible for the headline numbers. A second check is to test on held-out vehicle classes that were not used when selecting the recurring-keyword mapping.","tokens_in":12308,"feed_emoji":"🎯","tokens_out":9118,"duration_ms":78371,"temperature":0.7,"pith_summary":"The paper proposes a two-stage pipeline for automatic target recognition (ATR) that separates localization from naming: an open-world detector crops candidate vehicles, and a large vision-language model (LVLM) assigns each crop a label without retraining. The authors argue this lets a system recognize military vehicle classes it has never seen across distance ranges, thermal imagery, and weather-degraded scenes. They report that the strongest API models reach roughly 58 to 73 percent accuracy depending on dataset and prompting protocol, well above a CLIP baseline, and that chain-of-thought prompting consistently helps. The intended payoff is an ATR system that can be pointed at novel categories and unknown environments and still produce useful labels.","feed_headline":"Vision-language models label military vehicles they never saw","feed_subtitle":"Two-stage pipeline reports up to 73 percent accuracy on novel vehicle classes, including thermal and rain-degraded images.","key_machinery":"The load-bearing object is the cascaded binary-detection-then-reevaluation pipeline. YOLO-world, an open-vocabulary detector, is prompted with a single class keyword ('vehicle') so that it acts as a class-agnostic localizer, sidestepping its unreliable confidence on novel classes. The LVLM reevaluation is the recognition engine, with three protocols: open-set (free naming), closed-set (forced choice from a label list with a novel option), and chain-of-thought (attribute description before labeling). The binary keyword choice is what makes the system zero-shot, because the detector never commits to a closed vocabulary.","core_discovery":"The paper's central claim is that combining detection and recognition as separate modules overcomes each module's weakness: YOLO-world, prompted with the single binary keyword 'vehicle', produces bounding boxes regardless of object class, and an LVLM labels each crop. In open-set prompting the model is asked to name the specific vehicle in one response; in closed-set prompting it chooses from a provided label list plus a 'novel' option; in chain-of-thought prompting it first describes attributes and then labels. On the military long-range dataset, the best API models score about 67 to 73 percent at 1,000 to 2,000 meter ranges, and on the synthetic desert dataset about 64 to 73 percent, with accuracy falling sharply at 3,000 to 5,000 meters. The paper also shows the pipeline can discard detector false positives and that LVLMs surpass the detector itself on recognition, though the improvement on thermal imagery is smaller.","pith_inferences":["Because the label-to-class mapping for open-set scoring is chosen after seeing the model's outputs, the reported numbers are ceilings; a deployment with a fixed vocabulary or exact-match rule would likely score lower.","A natural extension is to hold out a disjoint set of vehicle classes and measure human agreement with the model's proposed names instead of matching to ground truth.","The pipeline's upper bound is set by detector recall during binary localization, so improving class-agnostic proposals should transfer directly to ATR accuracy.","The smaller models' failures on degraded scenes suggest using strong LVLMs to label crops from weak detectors, then training compact specialized models on those labels."],"forward_implications":["Novel military vehicle types can be labeled without retraining or new annotation, as long as the detector can crop them.","Adding an LVLM reevaluation stage lets an existing ATR pipeline discard detector false positives by checking each crop's label plausibility.","Chain-of-thought prompting improves recognition relative to direct labeling, especially for similar vehicle classes.","Performance degrades predictably with range and weather, and the gap between API and open-source models is large.","The same two-stage recipe should transfer to other novel object domains where a detector can localize but cannot name."],"supporting_citations":[{"why":"Supplies the YOLO-world open-vocabulary detector used as the binary localizer in the detection phase.","marker":"[34]"},{"why":"Provides the CLIP baseline that all pipeline variants are compared against.","marker":"[29]"},{"why":"Defines the open-set ATR setting and the evaluation protocol adopted for open-set recognition.","marker":"[43]"},{"why":"GPT-4o is the strongest API model on most datasets and anchors the headline accuracy results.","marker":"[44]"},{"why":"Claude-3.5-Sonnet matches or exceeds GPT-4o on the synthetic dataset, defining the top end of reported performance.","marker":"[45]"},{"why":"Gemini-1.5-Pro is the third API model used in the cross-model comparison across prompting protocols.","marker":"[46]"},{"why":"LLaVA-1.5-7B represents the open-source LVLM tier, showing the capability gap with API models.","marker":"[47]"}],"fun_headline_variants":["LVLMs label unseen military vehicles with up to 73% accuracy","Detector plus LVLM combo recognizes unseen military vehicles","Zero-shot ATR pipeline pairs detectors with LVLMs for novel targets","LVLMs beat open-world detectors at recognizing unseen military vehicles","Vision-language models ID unknown military vehicles from detector crops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy depends on choosing the correct answer key after seeing the model's free-text labels, so the reported scores measure recall under a lenient label-matching rule rather than true zero-shot naming.","fun_headline_variants_meta":{"raw":{"variants":["LVLMs label unseen military vehicles with up to 73% accuracy","Detector plus LVLM combo recognizes unseen military vehicles","Zero-shot ATR pipeline pairs detectors with LVLMs for novel targets","LVLMs beat open-world detectors at recognizing unseen military vehicles","Vision-language models ID unknown military vehicles from detector crops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3353,"prompt_tokens":956,"completion_tokens":2397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2311}},"tokens_in":572,"tokens_out":2397,"duration_ms":19501,"temperature":1.0,"reasoning_tokens":2311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:42:36.308668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix the label-to-class mapping before inference: provide a predefined alias list for the eight classes in the long-range military dataset, run the open-set prompt, and score exact or fixed-list matches. If accuracy falls materially below the reported 67 to 73 percent at close range, the post-hoc keyword alignment is responsible for the headline numbers. A second check is to test on held-out vehicle classes that were not used when selecting the recurring-keyword mapping.","supporting_citations":[{"cited_title":"Yolo-world: Real-time open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO-world open-vocabulary detector used as the binary localizer in the detection phase."},{"cited_title":"Open-set automatic target recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the open-set ATR setting and the evaluation protocol adopted for open-set recognition."},{"cited_title":"Hello gpt-4o,","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the strongest API model on most datasets and anchors the headline accuracy results."},{"cited_title":"Introducing the next generation of Claude,","cited_arxiv_id":null,"evidence_quote":"Claude-3.5-Sonnet matches or exceeds GPT-4o on the synthetic dataset, defining the top end of reported performance."}],"review_version":1}