{"id":"af26739d-e459-4a82-bb23-08d4e13d72c3","arxiv_id":"2501.11485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SimLabel improves zero-shot OOD detection by scoring images based on their consistency with a set of semantically similar class labels.","lead":"This paper presents SimLabel, a post-hoc scoring method for out-of-distribution (OOD) detection that uses pretrained vision-language models. It improves detection by comparing each image to a class label and to a set of semantically similar labels, rather than to a single label alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 does not specify whether the ID images used to build the similar-class pool are the same evaluation images later scored; if they are, SimLabel-I's reported gains may be inflated by transductive access to the test set.","rationale":"The paper proposes SimLabel, a post-hoc consistency score that averages image-text affinity over a set of similar class labels. The central claim is empirical: this score separates ID from OOD better than MCM. The most load-bearing step is Algorithm 1, which constructs the similar-class pool D(lc) from a subset of ID images. If that subset is the same evaluation set that is later scored, then the score function itself has been adapted to the ID test distribution. This is not a violation of internal logic, but it would mean the reported gains reflect transductive fitting rather than a general consistency principle. The paper's own experiment section never clarifies the split; the phrase 'ID sample xID ∈ XID' in Algorithm 1 is under-specified. Because the absolute improvements in Table II are small, this ambiguity is material. I agree with the reader that this is the weakest assumption. I do not recommend changing the verdict because a clean re-run could still validate the method; the correct status is conditional on that clarification. The paper does provide useful evidence elsewhere: the method is evaluated on multiple backbones, on fine-grained datasets, and with a sensitivity analysis over k and alpha. Those pieces support the plausibility of the method but do not resolve the split question.","tokens_in":15882,"tokens_out":3691,"duration_ms":43319,"concrete_test":"Check the released code at https://github.com/ShuZou-1/SimLabel to identify which split is used for XID in Algorithm 1. If it is the same evaluation split, rerun Table II with D(lc) generated from a strictly disjoint set of ID images (e.g., ImageNet train split, or one random half of the val set) and evaluate SimLabel-I on the other half. If the AUROC/FPR95 gain over MCM shrinks materially, the reported improvement is not attributable to the consistency principle under a clean zero-shot protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SimLabel-I's central empirical claim is that its consistency score separates ID from OOD. That claim rests on the similar-class sets D(lc) produced by Algorithm 1. Algorithm 1 takes Xc ⊂ XID, computes per-image top-k similar labels, and aggregates by occurrence frequency. The paper never states whether XID is the training split, the validation split, or the same split used to compute the AUROC/FPR95 numbers in Table II. In the zero-shot setting there is no training split for the ID classes, so a reader naturally assumes the ImageNet validation set is used for both score construction and evaluation. If that is true, each scored ID image contributes to building the label graph that later decides its own ID/OOD score; the method is transductive rather than post-hoc zero-shot. The gains over MCM are modest (averaging roughly +1.28 AUROC and -7.15 FPR95 in Table II), so a test-set-derived label pool could plausibly explain them without the consistency hypothesis. A second, smaller issue: pseudo-labeling means Xc is selected by CLIP predictions, so D(lc) can mix images from several true classes; this is not necessarily fatal, but it makes the 'class prototype' interpretation weaker. The authors acknowledge long-tail imbalance but do not address this split ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SimLabel, a post-hoc zero-shot OOD detection score for pretrained vision-language models. For each in-distribution class label, SimLabel constructs a set of semantically similar labels using one of three strategies: label-set hierarchy, LLM-generated world knowledge, or image-text alignment on in-distribution images. The final affinity in Eq. (4) averages the CLIP similarity over these similar labels and adds it to the class-label similarity, and the score in Eq. (5) is the maximum softmax value of the resulting affinities. Experiments on the ImageNet-1k benchmark and on fine-grained datasets report improvements over the MCM baseline and several other post-hoc methods, with additional experiments on hard OOD tasks, other VLM backbones, and classification accuracy.","tokens_in":16117,"tokens_out":4810,"duration_ms":55877,"significance":"If the reported gains are robust and leakage-free, SimLabel is a simple and useful contribution: it shows that aggregating similarity over semantically related labels can improve ID/OOD separability without training any parameters. The paper is generally clearly written, the formulation is easy to reproduce from Eqs. (4) and (5), and the authors include useful ablations, including a version that uses only similar-class affinities (SimLabel-S) and zero-shot classification results. The main significance is currently limited by the modest and dataset-dependent gains, the lack of error bars, and a load-bearing ambiguity about whether Algorithm 1 uses the evaluation split to construct the similar-class pool.","major_comments":[{"comment":"Algorithm 1 does not specify what XID is. If XID is the same ImageNet validation split used to compute the AUROC/FPR95 numbers in Table II, then the similar-class sets D(lc) are constructed by pseudo-labeling and top-k selection on the very images that are later scored. This makes the procedure transductive rather than the claimed zero-shot post-hoc method, and it can inflate the reported improvements independently of the consistency hypothesis. The authors should state the split explicitly and, ideally, construct D(lc) using a disjoint set of ID images and re-report Tables II, III, and IV under that protocol.","section":"Sec. IV-C3, Algorithm 1"},{"comment":"The hyperparameters alpha=1 and k=6 are selected using the same ImageNet-1k benchmark on which the main results are reported, with no separate validation split described. Table VI and Fig. 5 are sensitivity analyses on the evaluation benchmark itself, so the reported numbers are selection results rather than an unbiased estimate of expected performance. Please fix hyperparameters on a held-out validation split or, at minimum, report the sensitivity of all reported datasets to alpha and k and justify the final choice under a predefined protocol.","section":"Secs. V-A and V-C, Table VI and Fig. 5"},{"comment":"The claimed improvement over the strongest baselines is not uniform across datasets. For example, SimLabel-I has lower AUROC than NPOS on SUN (90.35 vs. 90.44) and Textures (87.07 vs. 88.80), and substantially worse FPR95 on Textures (53.65 vs. 46.12). The paper averages over four OOD datasets and does not report variance or significance tests, which makes it difficult to tell whether the consistency mechanism, rather than dataset-specific label choices, drives the average gain. Please add per-dataset confidence intervals or repeated-run statistics and discuss the conditions under which the consistency score is expected to help.","section":"Table II"},{"comment":"The paper does not include a controlled comparison that isolates the effect of label-set construction from the consistency mechanism. Since SimLabel-I obtains D(lc) from image-text alignment on XID, its label pool is potentially tailored to the test distribution, whereas SimLabel-H and SimLabel-L use external text information. A control using random label sets of the same cardinality, or using hierarchy-derived sets matched for size, under the same data-split conditions, would directly test whether the observed gains are due to consistency over similar labels or to the information used to select them.","section":"Sec. IV-C3 and Eq. (4)"}],"minor_comments":[{"comment":"The function domains in Eq. (1) are swapped: f_T should map text prompts to R^D and f_I should map images to R^D as written in the text, and Eq. (2) has an unmatched closing parenthesis in the cosine-similarity expression.","section":"Sec. III, Eq. (1)"},{"comment":"The phrase \"set of similar class (donates D(xc))\" should read \"denotes D(xc)\", and the notation D(xc) is overloaded: it is used both for the per-image set of top-k labels and for the aggregated class-wise set D(lc). A clearer notation would avoid ambiguity.","section":"Sec. IV-C3"},{"comment":"Table VI reports FPR95 values but the metric is identified only in the surrounding text; adding a row or column label such as \"FPR95\" inside the table would make it self-contained.","section":"Table VI"},{"comment":"The caption contains a typo, \"ImgeNet\", and the text refers inconsistently to \"ImageNet-100\"; please unify the dataset name and spelling.","section":"Table V caption"},{"comment":"The description of MCM as using \"the maximum similarity\" is imprecise because MCM applies a softmax over the cosine similarities and uses the maximum softmax value; the paper should state this consistently with Sec. IV-A and Fig. 2.","section":"Sec. II"},{"comment":"The reference [14] is described as both \"CLIPEN\" and \"Dai et al.\"; please unify the citation name to avoid reader confusion.","section":"Sec. II and References"}],"recommendation":"major_revision","confidential_remarks":"The split ambiguity in Algorithm 1 is the main barrier to accepting this paper. The idea is plausible and the presentation is mostly clear, but the empirical claims need to be re-run with a construction/evaluation split that is explicitly stated and, ideally, disjoint. I would also ask the authors to avoid tuning alpha and k on the same benchmark used for reporting. If the gains persist under a clean protocol, the paper would be a solid contribution to the zero-shot OOD detection literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SimLabel is a genuinely new post-hoc score for zero-shot OOD detection: it augments the usual image-text affinity with an average over semantically similar class labels, then takes the softmax maximum. The three ways of building the similar-label pool (hierarchy, LLM, pseudo-alignment) are reasonable, and the authors back the core assumption with a clean experiment and a useful ablation (SimLabel-S) where the ground-truth label is removed. The paper also reports classification accuracy gains, which is a nice touch.\n\nWhere it gets soft: Algorithm 1 builds the similar-class pool from XID but never says which split that is. In zero-shot there is no training split, so the natural reading is that the validation set is used both to construct D(lc) and to report AUROC/FPR95. That makes the evaluation transductive: each scored image contributes to the label graph that decides its own score. The average gain over the strongest baseline is around 0.5-1 AUROC, so test-set-derived label selection could plausibly account for a chunk of that gain. The authors need to either use a separate held-out ID set for generating similar classes or clearly show the sensitivity to this choice. Also, no error bars anywhere, and SimLabel-H actually underperforms MCM on several datasets; the limitation section does acknowledge long-tail imbalance, but that underperformance deserves more than a one-line explanation given LLM and hierarchy variants are presented as part of the method.\n\nThe math is simple and correct under the stated assumption. The citation pattern is appropriate. The paper is honest about limitations. I think this deserves a serious referee, but the split ambiguity is load-bearing and must be fixed before the empirical claims are trustworthy. Send it to review with the requirement that the authors clarify or redo the Algorithm 1 split; also ask for error bars. If the split is clean, this is a modest but useful contribution.","headline":"New consistency-over-similar-labels OOD score, but an unspecified data split in Algorithm 1 leaves the main empirical claim open to test-set leakage.","tokens_in":16682,"tokens_out":2084,"would_cite":true,"duration_ms":22648,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that scoring an image against a label's semantically related neighbors, not just the label itself, sharpens the line between in-distribution and out-of-distribution images for pretrained vision-language models.","keywords":["out-of-distribution detection","vision-language models","CLIP","zero-shot OOD detection","post-hoc scoring","semantic label consistency","similar class generation","image-text alignment"],"falsifier":"Run SimLabel-I twice on the ImageNet-1k benchmark, once building each class's similar-class pool exactly as Algorithm 1 specifies and once building the pools only from a training or validation subset of the in-distribution images, holding $\\alpha$, $k$, and $\\tau$ fixed; if average AUROC/FPR@95 falls substantially under the disjoint protocol, test-image overlap rather than label consistency is carrying the reported gain, while comparable numbers would confirm the consistency hypothesis.","tokens_in":15674,"feed_emoji":"🔍","tokens_out":15403,"duration_ms":143757,"temperature":0.7,"pith_summary":"The paper argues that pretrained vision-language models such as CLIP separate in-distribution (ID) from out-of-distribution (OOD) images more reliably when the detection score checks consistency across semantically related class labels, not just the single best-matching label. Its observation is that an ID image typically shows high similarity to a whole cluster of adjacent labels, whereas an OOD image that happens to win one label does not show that cluster-wide agreement. The paper builds a post-hoc score, SimLabel, that adds to each class affinity the average similarity over a pool of similar classes, and reports that the best variant (pools chosen by image-text alignment) reaches 91.90 average AUROC and 36.46 FPR@95 on the ImageNet-1k benchmark, against 90.62 and 43.61 for the MCM baseline, with smaller gains on fine-grained and hard-OOD tasks. The paper's own limitation analysis (Sec. VI-D) concedes that the method assumes a fairly balanced label space and weights every similar class equally, leaving long-tailed distributions and uneven semantic distances as open problems. If the claim holds, the payoff is better zero-shot OOD detection with zero training, since the new score is a drop-in replacement for the standard MCM score.","feed_headline":"Similar-label consistency improves out-of-distribution detection","feed_subtitle":"The post-hoc score cuts false-positive OOD errors by about 7 points over MCM on ImageNet, with no retraining.","key_machinery":"The carrying mechanism is the consistency-adjusted affinity $A(x, l_c) = M(x, l_c) + \\alpha \\cdot \\frac{1}{|D(l_c)|}\\sum_{d \\in D(l_c)} M(x, d)$, combined with the softmax-maximum OOD score $S(x; L, \\tau) = \\max_{l_c \\in L} \\frac{e^{A(x,l_c)/\\tau}}{\\sum_c e^{A(x,l_c)/\\tau}}$. $M$ is the cosine similarity between a CLIP image embedding and a prompted class-label text embedding; $D(l_c)$ is that class's pool of similar labels; $\\alpha$ weights the neighborhood term. The pools are the second half of the machinery: text-hierarchy selection, LLM prompting for visually similar labels, and an image-text-alignment routine (Algorithm 1) that collects, over images pseudo-classified as $l_c$, the labels that appear most often among the top-$k$ matches and keeps the top-occurring ones. The argument is that this construction converts a one-shot maximum-similarity signal into a consistency check that in-distribution images pass across the whole neighborhood and OOD images fail.","core_discovery":"SimLabel's central claim is that the maximum-softmax score of a frozen vision-language model throws away the semantic structure of the label space. For every class $l_c$ the method builds a pool $D(l_c)$ of similar classes and defines an augmented affinity $A(x, l_c) = M(x, l_c) + \\alpha \\cdot \\frac{1}{|D(l_c)|}\\sum_{d \\in D(l_c)} M(x, d)$, where $M$ is the CLIP cosine similarity between image and prompted-label embeddings; the OOD score is the maximum over classes of the softmax of these affinities at temperature $\\tau$ (Eqs. 4-5). ID images are expected to score high across their class's pool, so the mean-over-neighbors term reinforces them, while an OOD image that wins a single label receives reinforcement only if its appearance happens to match the whole neighborhood, which the paper's experiments indicate it usually does not. Three pool-construction routes are tested: label hierarchy (SimLabel-H), LLM-generated visually similar classes (SimLabel-L), and counting, over ID images pseudo-labeled as $l_c$, which labels most often recur among their top-$k$ similarities (SimLabel-I), with the last performing best on the ImageNet-1k benchmark. The paper also reports that the same affinity improves zero-shot classification accuracy and transfers to other VLM backbones, and its stated scope is limited to relatively balanced label sets with equally weighted similar classes.","pith_inferences":["Editorial check: if Algorithm 1's $X_{ID}$ is the evaluation set, the reported margins over MCM are not a clean test of the consistency idea; a strictly disjoint pool (training or validation images only) is the decisive experiment, and the released code can settle which split is used.","The paper's pooled mean over similar classes is one summary of the top-$k$ similarity distribution; untested alternatives such as the variance or entropy of an image's similarities across its predicted class's neighborhood would probe the same consistency hypothesis without needing an explicit label pool.","A natural downstream application the paper does not evaluate is prediction refusal: if a class wins without neighborhood support, the same affinity could flag the input as low-confidence for abstention in zero-shot classifiers."],"forward_implications":["Any deployment already using the MCM score can swap in SimLabel with no retraining or OOD exemplars, because it is a post-hoc score over the same frozen image and text encoders.","The best variant (SimLabel-I) also improves zero-shot classification accuracy on ImageNet, ImageNetV2, and CUB-200, so the neighborhood term helps in-distribution recognition rather than only OOD separation.","The reported gains transfer across model families (CLIP, AltCLIP, GroupViT), across ID scales from ImageNet-1k to fine-grained sets, and to semantic-hard splits, which the paper offers as evidence of generality.","A similar-classes-only score (SimLabel-S) that drops the ground-truth label still detects OOD inputs far better than chance, supporting the paper's claim that adjacent labels share prototype structure.","A small pool of $k = 6$ similar classes per class suffices; the paper reports diminishing returns as $k$ grows, which justifies its hyper-parameter choice."],"supporting_citations":[{"why":"Supplies the MCM baseline that SimLabel extends and beats: the maximum-softmax image-label similarity score, the primary comparison across all benchmarks.","marker":"[13]"},{"why":"CLIP, the frozen vision-language model whose image and text encoders and cosine similarity provide the affinity $M(x, l_c)$ that every SimLabel score is built on.","marker":"[11]"},{"why":"CHiLS, the hierarchical label-set construction that SimLabel-H uses to select similar classes from the same super-class.","marker":"[31]"},{"why":"GPT-4, the LLM prompted to generate visually similar class labels for the SimLabel-L variant.","marker":"[43]"},{"why":"Defines the ImageNet-1k OOD benchmark (with iNaturalist, SUN, Places, and Texture as OOD sets) that the main experiments and reported averages use.","marker":"[38]"},{"why":"NegLabel, the external-negative-label method that SimLabel positions itself against, marking the contrast between external prompts and internal label consistency.","marker":"[16]"},{"why":"The strongest multi-modal OOD baseline in the comparisons; SimLabel-I reports beating it by 0.47 AUROC and 1.25 FPR95 on average.","marker":"[14]"}],"fun_headline_variants":["Similar-label consistency sharpens OOD detection","Post-hoc score boosts OOD detection via label neighborhoods","Leverage label structure for stronger OOD detection","SimLabel: neighbor-aware affinities cut false positives","Label-consistent affinities improve zero-shot OOD detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the in-distribution images used by the pool-construction routine (Algorithm 1, Sec. IV-C3) are not the same images whose out-of-distribution scores are later computed; the paper never states whether $X_{ID}$ is training or evaluation data, and if pools are built on the evaluation images, the reported margins could rest on that overlap rather than on label-consistency.","fun_headline_variants_meta":{"raw":{"variants":["Similar-label consistency sharpens OOD detection","Post-hoc score boosts OOD detection via label neighborhoods","Leverage label structure for stronger OOD detection","SimLabel: neighbor-aware affinities cut false positives","Label-consistent affinities improve zero-shot OOD detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1300,"prompt_tokens":1064,"completion_tokens":236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":680,"tokens_out":236,"duration_ms":3321,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:11:10.186553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SimLabel-I twice on the ImageNet-1k benchmark, once building each class's similar-class pool exactly as Algorithm 1 specifies and once building the pools only from a training or validation subset of the in-distribution images, holding $\\alpha$, $k$, and $\\tau$ fixed; if average AUROC/FPR@95 falls substantially under the disjoint protocol, test-image overlap rather than label consistency is carrying the reported gain, while comparable numbers would confirm the consistency hypothesis.","supporting_citations":[{"cited_title":"Delving into out-of- distribution detection with vision-language representations,","cited_arxiv_id":null,"evidence_quote":"Supplies the MCM baseline that SimLabel extends and beats: the maximum-softmax image-label similarity score, the primary comparison across all benchmarks."},{"cited_title":"Chils: Zero-shot image classification with hierarchical label sets,","cited_arxiv_id":null,"evidence_quote":"CHiLS, the hierarchical label-set construction that SimLabel-H uses to select similar classes from the same super-class."},{"cited_title":"Gpt-4 technical report,","cited_arxiv_id":null,"evidence_quote":"GPT-4, the LLM prompted to generate visually similar class labels for the SimLabel-L variant."},{"cited_title":"On the importance of gradients for detecting distributional shifts in the wild,","cited_arxiv_id":null,"evidence_quote":"Defines the ImageNet-1k OOD benchmark (with iNaturalist, SUN, Places, and Texture as OOD sets) that the main experiments and reported averages use."},{"cited_title":"Nega- tive label guided ood detection with pretrained vision-language models,","cited_arxiv_id":null,"evidence_quote":"NegLabel, the external-negative-label method that SimLabel positions itself against, marking the contrast between external prompts and internal label consistency."},{"cited_title":"Exploring large language models for multi-modal out-of-distribution detection,","cited_arxiv_id":null,"evidence_quote":"The strongest multi-modal OOD baseline in the comparisons; SimLabel-I reports beating it by 0.47 AUROC and 1.25 FPR95 on average."}],"review_version":1}