{"id":"9389c6ba-6d5d-47a4-8cf0-4c8f19b9fad1","arxiv_id":"2505.09139","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A prompt-ranking metric that subtracts semantic similarity to confounding classes selects higher-precision prompts for zero-shot vision-language object detection.","lead":"The authors introduce a scoring formula, the Contrastive Class Alignment Score, that ranks candidate prompts for detecting an object by how similar they are to the object's name and how different from confusing objects. On two zero-shot detection datasets, selecting the top-ranked prompt improves average precision compared with the plain class name, without retraining the detector.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CCAS ranking is never validated against OWLv2's own text encoder; the reported gains rest entirely on an unmeasured correlation between MiniLM embeddings and OWLv2 confusion behavior.","rationale":"I agree with the reader's weakest_assumption: the link between MiniLM similarity and OWLv2 precision is the load-bearing condition. The paper's own evidence is only two tasks, with no error bars and no per-prompt AP, so the link is asserted rather than demonstrated. The existence of two CCAS variants and a dataset-dependent winner adds a second, related fragility: 'the method' is not a single algorithm unless the variant choice and top-n are fixed by a rule. These are correctness and reproducibility risks, not fraud; the method might well work. The concrete check above would settle the proxy question. Since the reader already assigned CONDITIONAL, I see no need to change the verdict.","tokens_in":6666,"tokens_out":5516,"duration_ms":56142,"concrete_test":"For both reported tasks, release the full LLM-generated prompt lists and compute, for every candidate prompt, the OWLv2 AP and the CCAS_avg/CCAS_max scores. Calculate Spearman rank correlation between each CCAS score and AP. Also rank the same prompts using OWLv2's own text-encoder cosine similarity to the target and confounding class names, and compare the top-1 AP under that ranking to CCAS's top-1 AP. If CCAS's rank correlation is not significantly positive, or if OWLv2's native similarity ranking performs at least as well, the proxy that the whole pipeline depends on is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CCAS-ranked prompts improve OWLv2 detection precision without training or labels. The load-bearing step is the unstated identification, in Section III, of all-MiniLM-L6-v2 cosine similarity with the confusion behavior of OWLv2. This proxy is never tested directly: the paper reports AP only for baseline and the top-1/top-3/top-5/top-N prompt sets, not per-prompt AP for all candidates, so there is no evidence that CCAS ordering correlates with detector precision. No comparison is made against OWLv2's own text-encoder similarity, which would be the natural measure if the goal is to predict OWLv2 confusion. The two scoring variants (Eqs. 1-2) are also reported as separate methods, and the better variant differs by dataset (CCAS_max wins on goggles, CCAS_avg on stop signs); without a pre-registered rule for choosing the variant and n, the headline improvement may be a selection artifact rather than a property of a single automated pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated prompt-refinement pipeline for zero-shot object detection with vision-language models. For a target class and a set of manually chosen confounding classes, the method uses an LLM (GPT-4o) to generate N candidate prompts per class, embeds all prompts with the sentence transformer all-MiniLM-L6-v2, and ranks target-class prompts by a Contrastive Class Alignment Score (CCAS) that rewards similarity to the base class name while penalizing similarity to confounder prompts. The top-ranked prompts are then fed to OWLv2, and the paper reports average precision on two datasets: a safety-goggles dataset with confounders 'glasses' and 'sunglasses', and a self-driving-cars stop-sign dataset with confounders 'red light' and 'speed limit'. The reported results show AP improvements over the base class-name baseline for both datasets, with CCAS_max preferred on goggles and CCAS_avg preferred on stop signs.","tokens_in":6851,"tokens_out":4841,"duration_ms":50700,"significance":"If the central claim holds, the work offers a simple, interpretable, and training-free alternative to manual prompt engineering for zero-shot detection, with potential value in safety-critical applications where precision against visually similar distractors matters. The pipeline is appealingly transparent: the score is computed from text embeddings only, requires no detector training or labeled examples, and produces human-readable prompts. However, the significance is currently limited by the evaluation's narrow scope (two datasets, one detector), the absence of statistical validation, and the untested assumption that MiniLM cosine similarity predicts OWLv2's confusion behavior. The paper would be a useful contribution if these load-bearing points are addressed; as presented, the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The load-bearing assumption is that cosine similarity computed in all-MiniLM-L6-v2 embedding space predicts which candidate prompts will reduce false positives in OWLv2. This assumption is never directly tested: Tables III and IV report AP only for the baseline and for top-{1,3,5,N} prompt sets, not per-prompt AP for the full candidate pool, so there is no evidence that CCAS ordering correlates with detector precision. Please add per-prompt AP values or a rank-correlation test (e.g., Spearman) between CCAS scores and AP, and compare against a ranking obtained with OWLv2's own text encoder. Without such a validation, the central claim that CCAS identifies high-precision prompts is unsupported.","section":"Section III, Eqs. (1)-(2)"},{"comment":"The choice of scoring variant and prompt-count configuration appears post hoc: CCAS_max performs better on the goggles task, while CCAS_avg performs better on the stop-sign task, and N is set to 15 for one dataset and 25 for the other without a stated rule. Because two CCAS variants and several top-n thresholds are evaluated, the reported best improvements may reflect selection over a small grid of configurations rather than a property of a single automated pipeline. Please specify an a priori rule for selecting the variant and n (for example, based on the measured ambiguity of the base class) and validate that rule across additional datasets.","section":"Section IV, Tables III-IV"},{"comment":"The stop-sign baseline AP of 0.0069 is unusually poor, and the large relative gains over this baseline may not indicate a generally effective method. The manuscript should report absolute AP with error bars or significance tests across a broader set of classes with non-degenerate baselines before claiming, as the abstract does, that the method 'improves object detection accuracy' in general. At minimum, the paper should acknowledge that the stop-sign result is a single extreme case.","section":"Section IV, Table IV"},{"comment":"The Discussion concedes that when the base class name is already distinguishable, the baseline prompt should be preferred over CCAS-selected prompts, but it provides no criterion for detecting this situation. This concession directly limits the abstract's claim of a fully automated pipeline. Please either supply a decision rule (for example, a CCAS margin threshold that triggers fallback to the baseline) or revise the claim to scope the method to ambiguous classes.","section":"Section V, Discussion"}],"minor_comments":[{"comment":"The notation is inconsistent: the equations use cos(\\vec{t}_i, \\vec{T}) and cos(\\vec{t}_i, \\vec{c}_{m,k}), but the definition list gives \\vec{t}_{i,j} for the j-th candidate prompt and \\vec{t}_i for the base class name, and \\vec{T} is never defined. Please align the notation with the prose, which describes scores computed for each candidate prompt.","section":"Section III, Eqs. (1)-(2)"},{"comment":"The caption states that the pipeline is 'discussed in Algorithm and Experimental Evaluation,' but the manuscript contains no numbered Algorithm section. Please either add the algorithm pseudocode or correct the cross-reference.","section":"Figure 2 caption"},{"comment":"The paper does not state whether OWLv2 detections were thresholded at a fixed confidence score or how AP was computed (e.g., whether the Roboflow evaluation protocol was used). Please report the protocol so the AP numbers are reproducible.","section":"Section IV"},{"comment":"Reference [2] renders the dataset name as 'Sugarcrepe++' while the text uses 'SugarCrepe++'; please standardize the spelling.","section":"References"},{"comment":"The table headers 'CCAS scores averages' and 'CCAS scores maxes' are awkward; consider using 'CCAS_avg' and 'CCAS_max' to match Equations (1) and (2).","section":"Tables I-II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is short and under-powered for the strength of its claims, but the core idea is simple and potentially useful. The main risk is that the reported gains are a selection artifact over datasets, CCAS variants, and top-n choices, plus an unvalidated proxy relation between MiniLM embeddings and OWLv2 behavior. I do not see a fundamental flaw that would require rejection; the needed fixes are additional validation and a more disciplined evaluation protocol. The fit with the journal's scope is acceptable if the empirical claims are made more precise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Simple idea, clean execution, thin evidence. The paper proposes ranking LLM-generated detection prompts by a contrastive score in a sentence-embedding space - similarity to the target class minus similarity to chosen confounding classes - then feeding the top-ranked prompts to OWLv2. On the two reported tasks the gains are large: goggle AP goes from 0.26 to 0.54 (CCAS_max), stop-sign AP from 0.007 to 0.38 (CCAS_avg). That is a meaningful effect for a training-free, label-free pipeline, and the method is model-agnostic.\n\nThe specific CCAS formula appears new. The paper also honestly reports both scoring variants and acknowledges that sometimes the plain class name is already the best prompt. These two points earn credit.\n\nThe soft spots are real, and they are mostly about validation. The load-bearing claim - that MiniLM cosine similarity predicts how OWLv2 will confuse classes - is never tested directly. There are no per-prompt AP numbers for all candidates, so we cannot see whether CCAS ranking actually correlates with detector precision. The stop-sign baseline is nearly floor ('stop' as a prompt), which flatters the relative improvement. The better CCAS variant differs by dataset, and the paper gives no pre-registered rule for choosing between avg and max; as written, the best reported number is a post-hoc selection. The evaluation is exactly two small datasets, with no code, no prompt list for the stop-sign task, and no error bars or significance tests (though the detector is deterministic, so the last point is minor). The 'no labeled data' phrasing is generous: confounding classes are chosen by hand based on knowledge of the dataset.\n\nThis will be a useful read for people working on zero-shot detection in niche safety domains who want a low-cost prompt-refinement tool. It is not a large contribution to how VLMs are trained or benchmarked. A serious referee could push the authors to validate the proxy directly, add datasets, and settle the variant-selection question. I would send it to peer review rather than desk reject - the method is clean, the claims are testable, and the current evidence, though thin, is consistent with the hypothesis.","headline":"A clean, simple prompt-ranking method with large but under-validated gains on two detection tasks; the proxy assumption between MiniLM and OWLv2 is the main weakness.","tokens_in":7367,"tokens_out":4462,"would_cite":false,"duration_ms":40786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CCAS ranks prompt candidates by semantic closeness to a target class while penalizing similarity to confusable classes, and top-ranked prompts improve zero-shot detection accuracy with no training or labeled data.","keywords":["vision-language models","zero-shot object detection","automated prompt refinement","contrastive class alignment score","confounding classes","sentence transformer","average precision"],"falsifier":"Enumerate a larger benchmark of target-and-confounder class groups, compute CCAS for every LLM-generated prompt, and run OWLv2 on each prompt to measure average precision; if the rank correlation between CCAS and AP is zero or negative, or if the top-ranked prompt fails to beat the base class name on average, the central claim is refuted.","tokens_in":6456,"feed_emoji":"🎯","tokens_out":7453,"duration_ms":69815,"temperature":0.7,"pith_summary":"The paper proposes that prompt quality in zero-shot vision-language object detection can be judged before ever running the detector: generate many candidate phrases with an LLM for a target class and its confusable classes, embed them with a sentence transformer, and score each target prompt by how close it sits to the target class name and how far it sits from confounder prompts. This score, the Contrastive Class Alignment Score (CCAS), comes in an average and a maximum variant. The authors show that taking the top one to five CCAS prompts as detector inputs improves average precision over the bare class name on two confusable-class tasks: 'goggles' versus 'glasses' and 'sunglasses', and 'stop sign' versus 'red light' and 'speed limit'. No model finetuning and no labeled data are needed. The wider point is that a cheap, interpretable, text-only filter can replace manual prompt engineering for VLM-based detection.","feed_headline":"CCAS score finds high-precision prompts without training or labels","feed_subtitle":"Text-only ranking of prompt embeddings beats generic class names on confusable objects like goggles versus sunglasses.","key_machinery":"The load-bearing object is the Contrastive Class Alignment Score (CCAS), computed for each candidate prompt $t_i$ of a target class $T$ against $M$ confounding classes that each have $N$ prompts: $\\operatorname{CCAS}_{\\mathrm{avg}}(t_i)=\\cos(\\vec{t}_i,\\vec{T})-\\frac{1}{NM}\\sum_{m=1}^{M}\\sum_{k=1}^{N}\\cos(\\vec{t}_i,\\vec{c}_{m,k})$ and $\\operatorname{CCAS}_{\\mathrm{max}}(t_i)=\\cos(\\vec{t}_i,\\vec{T})-\\max_{m,k}\\cos(\\vec{t}_i,\\vec{c}_{m,k})$. The score lives entirely in text-embedding space, using cosine similarity from a sentence transformer, so it ranks prompts before any image is shown to the detector. That ordering is what turns an undifferentiated LLM-generated prompt list into a small set of high-precision prompts.","core_discovery":"The central claim is that an automatic, training-free prompt filter can disambiguate objects for a zero-shot detector. On the Safety Goggles and Self-Driving Cars evaluations, replacing the base prompt with a top-ranked CCAS prompt raises OWLv2 average precision from 0.2555 to 0.5415 for goggles (CCAS max, top 1) and from 0.006858 to 0.3835 for stop signs (CCAS avg, top 1). The method deliberately lets imperfect LLM prompts such as 'swimming goggles' and 'ski goggles' propagate through the ranking, to show that the scoring still selects useful prompts without manual cleaning. The authors interpret the gains as coming from phrases that are specific enough to match the target while staying semantically distant from the given confounding classes.","pith_inferences":["The paper does not test whether sentence-transformer cosine similarity tracks the internal similarity of the downstream detector; a natural extension is to compare CCAS rankings with per-prompt detector AP across a broader class set, or to replace the text-only proxy with detector-derived similarity scores.","The method inherits the user's choice of confounding classes; automating confounder discovery from dataset labels or an ontology would remove the last human step in the pipeline.","Because the score is text-only and detector-agnostic, it should transfer to other discrete-prompt vision tasks such as open-vocabulary segmentation or classification, where the same ambiguity-between-classes phenomenon appears."],"forward_implications":["If the central claim holds, a practitioner with a target class and a short list of confusable classes can get better detection precision by running the CCAS pipeline than by using the dataset class name, without touching model weights.","Because no training or labels are involved, the same filter can be dropped in front of any open-vocabulary detector that accepts discrete text prompts.","The observed drop from top-1 to top-3 and top-5 prompts implies that feeding fewer, higher-scoring prompts is preferable when precision is the goal.","CCAS avg and CCAS max are both viable, with CCAS max winning on goggles and CCAS avg winning on stop signs; the right variant may depend on how concentrated the confounder similarity is."],"supporting_citations":[{"why":"Supplies the Safety Goggles dataset used for one of the two zero-shot detection evaluations.","marker":"1"},{"why":"Supplies the Self-Driving Cars dataset used for the stop-sign evaluation.","marker":"[19]"},{"why":"Provides the LLM that generates the candidate prompt lists for target and confounding classes.","marker":"[20]"},{"why":"Provides the sentence-transformer embeddings from which CCAS cosine similarities are computed.","marker":"[21]"},{"why":"Supplies the open-vocabulary detector used to measure zero-shot detection average precision.","marker":"[22]"}],"fun_headline_variants":["CCAS: automatic prompt tuning for VLMs without extra data","Training-free prompt selection boosts VLM object detection","New score picks better prompts for confusing object classes","Auto prompt refinement for vision-language models via CCAS","Rank prompts by CCAS to disambiguate objects in VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that cosine similarity between sentence-transformer embeddings of prompts predicts which prompts will confuse the downstream OWLv2 detector; if this text-only proxy fails, the CCAS ranking need not improve detection.","fun_headline_variants_meta":{"raw":{"variants":["CCAS: automatic prompt tuning for VLMs without extra data","Training-free prompt selection boosts VLM object detection","New score picks better prompts for confusing object classes","Auto prompt refinement for vision-language models via CCAS","Rank prompts by CCAS to disambiguate objects in VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3687,"prompt_tokens":844,"completion_tokens":2843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":2764}},"tokens_in":460,"tokens_out":2843,"duration_ms":18536,"temperature":1.0,"reasoning_tokens":2764,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:38:16.692527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Enumerate a larger benchmark of target-and-confounder class groups, compute CCAS for every LLM-generated prompt, and run OWLv2 on each prompt to measure average precision; if the rank correlation between CCAS and AP is zero or negative, or if the top-ranked prompt fails to beat the base class name on average, the central claim is refuted.","supporting_citations":[{"cited_title":"Self-driving cars dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the Self-Driving Cars dataset used for the stop-sign evaluation."},{"cited_title":"Scaling open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the open-vocabulary detector used to measure zero-shot detection average precision."}],"review_version":1}