{"id":"1263dad3-9412-49e0-9007-89d69b9be7f3","arxiv_id":"2412.04925","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"S3 improves CLIP zero-shot classification by building a class-specific space of synonymous text embeddings, filtered by persistent homology, and comparing image embeddings to local centers in that space.","lead":"This paper introduces S3, a method that represents each image class as a 'synonymous semantic space' built from many LLM-generated textual descriptions instead of a single concept, and uses a point-to-local-center similarity to classify images with CLIP. The method reports higher zero-shot classification accuracy than existing prompt engineering and test-time adaptation methods across 17 benchmarks, but several hyperparameters were tuned on the evaluation datasets themselves.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on epsilon and N chosen from Pets test accuracy, with only a 1.5-point homology gain; a clean validation protocol is needed before the central claim is established.","rationale":"The reader's weakest assumption correctly identifies the global epsilon=0.9 threshold as a fragile point, and the manuscript's own supplementary material confirms that the threshold is recommended from Pets test accuracy. My stress test adds two load-bearing details: (1) the neighborhood size N and LLM choice are selected on the same type of benchmark evidence, so the full pipeline has multiple test-informed choices; and (2) the topological filtering is only responsible for about 1.5 points of average gain, meaning a modest hyperparameter misspecification could erase the method's claimed advantage over PE baselines. The persistent homology description in Eq. (3)-(4) is also internally loose: with a cosine-similarity threshold, connectivity decreases as epsilon increases, and maximal lifetime in 0-D persistence does not select the largest component at a single threshold; this makes the stated algorithm hard to reproduce independently. I am not rejecting the method: the idea is reasonable, the cost analysis is useful, and the architecture generalization table (Table S4) shows consistent gains. The issue is that the central empirical claim is not yet supported by an evaluation that separates genuine zero-shot improvement from hyperparameters tuned on the reported test sets. A threshold/N sweep with a proper validation split would settle this directly. The conditional verdict remains appropriate: accept only after the clean-protocol re-evaluation and code/text release.","tokens_in":15024,"tokens_out":5968,"duration_ms":71421,"concrete_test":"Reproduce the fine-grained and natural-distribution tables with a clean protocol: for each dataset, select epsilon in {0.70,0.75,...,0.95} and N in {5,10,20,30} on the official train split (or a random 50% split of the test set), then evaluate on the held-out split, repeating over 5 LLM-generation seeds and reporting mean +/- std. If the per-dataset optimal epsilon is not consistently near 0.9, or if fixed epsilon=0.9 loses more than the ~1.5-point homology gain relative to per-dataset selection, then the reported SOTA numbers are an artifact of Pets test-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical argument for S3 depends on two hyperparameters that are set by looking at test accuracy: Supplementary B fixes epsilon_max=0.9 and N in [10,30] from Pets (Figs. S1/S2), and the LLM choice is also made on the same benchmarks (Table S2). Since Pets is included in the reported 70.04 average, the headline number is partly a test-set-tuned result. More importantly, Section 3.2/Eq. (3)-(4) does not actually determine epsilon from persistent homology: increasing epsilon makes the Vietoris-Rips graph sparser, not more connected, and the 'generator with maximal lifespan' in 0-dimensional persistent homology is not the same as the largest connected component at one fixed cosine threshold. The only ablation for the topological filter (Table 4/S3) shows a 1.5-point average gain, with EuroSAT jumping 8.7 points. Thus the claimed advantage over DCLIP/CuPL could shrink or disappear if epsilon/N are chosen on a proper validation set or if the threshold varies by dataset/class density. Without a sensitivity analysis or robust selection rule, 'outperforms state-of-the-art' is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S3, a text-side enhancement to CLIP zero-shot classification. For each class, an LLM generates synonyms and descriptors, which are combined into 'synonymous texts'; these are embedded with CLIP's text encoder, and a Vietoris-Rips complex with a fixed cosine threshold is used to select the largest connected component as a 'synonymous semantic space.' Similarity between a test image and each class space is computed by a proposed point-to-local-center metric (nearest text embedding, then average of its N nearest neighbors). The paper reports gains over prompt-engineering and test-time-adaptation baselines on 10 fine-grained, 5 distribution-shift, and 2 segmentation benchmarks, plus a TTA variant called T S3. The supplementary material fixes epsilon_max=0.9 and N in [10,30] using Oxford Pets accuracy and selects Claude over GPT-4 using average accuracy over the same ten benchmarks.","tokens_in":15247,"tokens_out":5426,"duration_ms":57792,"significance":"If the results are robust, S3 offers a simple, cost-effective drop-in replacement for the text branch of CLIP classification, with a genuinely interesting structural idea: replacing point text embeddings with a semantic space addresses lexical variation. The paper's strengths include experiments across 17 benchmarks, multiple CLIP backbones, a token-cost analysis, and an implemented TTA extension. However, as submitted, the main claim is not yet cleanly supported because key hyperparameters are selected on test sets included in the reported averages, and the persistent-homology construction is not derived from the actual computation. The central idea is worth publishing if those issues are addressed.","major_comments":[{"comment":"The reported averages are not obtained from a clean validation protocol. The similarity threshold epsilon_max=0.9 and neighborhood size N in [10,30] are selected by maximizing accuracy on Oxford Pets, which is one of the ten datasets included in the reported 70.04 average, and the choice of Claude over GPT-4 is made using average accuracy on the same ten benchmarks (Table S2). Under this protocol, part of the observed improvement over baselines may be test-set selection. Please either report a validation-based selection rule (for example, selecting hyperparameters on a separate dataset or on held-out class splits) and give sensitivity of the 70.04/70.32 averages to epsilon and N, or clearly state that the published numbers are post-selection.","section":"§4.1, Supplementary B, Figs. S1–S2, Table S2"},{"comment":"The described procedure is not persistent homology as written. In the Vietoris-Rips complex defined by the condition <fi,fj> >= epsilon, increasing epsilon removes edges and makes the complex sparser, so the sentence 'As epsilon increases, the connectivity within R_epsilon(S_k) evolves' is backwards. Moreover, the largest connected component at a single fixed epsilon=0.9 is an ordinary graph component, not an object selected by the generator with maximal lifespan; the paper never computes birth/death times or shows how epsilon_max is determined from persistence. If epsilon_max is simply a tuned cosine threshold, then the persistent-homology claim in the abstract and in Section 3.2 is unsupported, and Eq. (4) is a definition rather than a topological derivation. Please rewrite the construction in standard filtration terms or remove the persistent-homology claim.","section":"§3.2, Eq. (3)–(4)"},{"comment":"The homology ablation shows an average gain of only 1.5 points and highly variable per-dataset gains (EuroSAT +8.72, but many datasets +0.1 to +0.5). This is the only evidence that the topological filter contributes beyond the LLM-generated text set. Please specify precisely what 'w/o homology' uses (all generated texts, or some other graph construction), report per-dataset optimal epsilon, and compare the largest-component selection against using all generated texts or a simple k-nearest-neighbor core at a fixed epsilon. Without this, the central method component is not isolated and the 1.5-point gain may be an artifact of the Pets-tuned threshold.","section":"Table 4, Table S3"},{"comment":"On the natural distribution benchmarks, S3's average gain over DCLIP and CuPL is only 0.09–0.13 points, which is likely within run-to-run or benchmark noise. The text states that S3 'surpasses most state-of-the-art baseline methods' and 'shows an overall improvement of 0.13%' over CuPL, but such a small margin does not support a strong state-of-the-art claim without confidence intervals or multiple-seed results. Please either temper the claim or provide statistical support for the fine-grained and natural-distribution comparisons.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The phrase 'bottle half' should read 'bottom half'; this appears in the discussion of the homology ablation.","section":"§4.5, Table 4"},{"comment":"The text says 'much more cost-effective than MVPR'; the method name is MPVR, as used in Table 1 and the related work section.","section":"§4.2"},{"comment":"The sentence 'Claude consistently outperformed GPT-4' is contradicted by Table S2, where GPT-4 is higher on Aircraft and EuroSAT; 'consistently' should be replaced by 'on average'.","section":"§4.5, Table S2"},{"comment":"The notation is ambiguous: R_epsilon(S_k) is a set of simplexes, so the union of all its elements is just the vertex set. Please define S0_k as the vertex set (or induced subgraph) of the largest connected component of the graph at threshold epsilon_max.","section":"§3.2, Eq. (4)"},{"comment":"The T S3 adaptation is described only verbally; the entropy-minimization objective, the definition of the learnable shift vector v_k, and the gradient update rule should be written out or explicitly referenced to TPT/TPS equations, otherwise the reported 70.32 result is not reproducible.","section":"§3.4"},{"comment":"The accuracy curves used to set epsilon_max and N show no error bars or multiple runs; since these figures are the basis for the main evaluation, reporting variance would strengthen the robustness of the chosen hyperparameters.","section":"Figures S1–S2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is legitimate: the headline numbers are tuned on test sets included in the averages, and the persistent-homology narrative is not backed by a persistence computation. I would recommend major revision with a clean validation protocol, a rewritten Section 3.2, and a more careful statistical treatment of the small margins on natural distribution benchmarks. The core idea—replacing a single text embedding with a semantic space built from LLM-generated synonyms—is promising and cost-effective, so it is worth another round if these issues are fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: S3 is a sensible, low-cost way to expand each CLIP class label into a small space of LLM-generated synonymous texts, and the point-to-local-center similarity is a neat trick. But the main comparison numbers are compromised because epsilon, N, and the LLM choice are all selected by looking at the test sets that go into the reported average. I can't sign off on the \"outperforms SOTA\" claim as it stands.\n\nWhat's actually new: the Rips-complex filtering of generated texts and the local-center metric. The filtering is a reasonable way to drop hallucinated or off-topic descriptors, and the metric gives a small but consistent bump over the other three point-to-space options in Table 5. The token cost is genuinely low: 60K tokens per 1K classes versus MPVR's 1000K. The open-vocabulary segmentation gains over CuPL and REAL are also consistent, which suggests the idea transfers beyond classification.\n\nThe soft spots, in order of severity. First, Supplementary B sets epsilon_max=0.9 and N in [10,30] using top-1 accuracy on Oxford Pets, and Table S2 picks Claude over GPT-4 on the same ten-dataset suite. Pets is part of the reported 70.04 average, and the LLM choice affects the whole average, so this is not a clean held-out evaluation. Second, the topological description is muddled. The Vietoris-Rips complex in Eq. (3) uses similarity >= eps, so increasing eps makes the complex sparser, not more connected. And the \"generator with maximal lifespan\" in 0-dimensional persistent homology does not determine the largest connected component at one fixed eps. The homology ablation gives a 1.5-point average gain, but EuroSAT jumps 8.7 points, a red flag that the threshold is doing something dataset-specific. Third, no code, no error bars, and the baselines are a mixture of their own reproductions and numbers taken from other papers. All of these issues are fixable with a proper validation split, a sensitivity analysis, and a clearer statement of what the homology step is actually contributing.\n\nWho this is for: people working on prompt engineering or test-time adaptation for CLIP. The idea is worth a serious referee even though the current evaluation is not convincing. I would send it to review with a request for clean validation and code, not desk-reject it.","headline":"The core idea is plausible and cheap, but the headline accuracy numbers are partly tuned on the test sets themselves, so the SOTA claim does not yet hold up to scrutiny.","tokens_in":15803,"tokens_out":2672,"would_cite":false,"duration_ms":28515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Representing each class as a space of synonymous texts improves zero-shot accuracy in vision-language models.","keywords":["zero-shot generalization","vision-language models","synonymous semantic space","Vietoris-Rips complex","persistent homology","test-time adaptation","fine-grained classification","lexical variation"],"falsifier":"Take a set of classes where an LLM is deliberately prompted to produce a large cluster of off-target synonyms for one class, then run S3 with the fixed 0.9 threshold: if that cluster becomes the largest connected component and classification accuracy collapses for that class while other classes stay unaffected, the global-threshold assumption is falsified. A quantitative variant is to compare the fixed 0.9 threshold against per-class thresholds chosen from each persistence diagram on a held-out set; if per-class thresholds outperform by a nontrivial margin, the paper's construction depends on ungeneralized tuning.","tokens_in":1752,"feed_emoji":"🧩","tokens_out":2549,"duration_ms":78946,"temperature":0.7,"pith_summary":"The paper argues that zero-shot vision-language models lose accuracy because each class is represented by a single text label, while real images are described by many synonymous phrases. It proposes $S^3$, which generates multiple synonymous texts per class with an LLM and packages them into one continuous, compact semantic space per class using the largest connected component of a Vietoris-Rips complex. Similarity between an image embedding and each class space is then measured with a point-to-local-center metric. On ten fine-grained benchmarks the method reports average top-1 accuracy of 70.04% versus 63.45% for the CLIP baseline, and it also improves natural-distribution and segmentation tasks. If correct, $S^3$ is a cheap, retraining-free upgrade to the text side of CLIP-style models.","feed_headline":"Synonym spaces beat single-label prompts for zero-shot CLIP","feed_subtitle":"Generated synonyms per class form compact text spaces, lifting fine-grained accuracy to 70% at a fraction of LLM cost.","key_machinery":"The key machinery is the Vietoris-Rips complex of text embeddings together with zero-dimensional persistent homology: a simplicial complex built by connecting text embeddings whose cosine similarity is at least a threshold $\\epsilon$, with the largest connected component at $\\epsilon_{\\max}$ serving as a compact, noise-filtered class representation. The complementary mechanism is the point-to-local-center metric, which finds the text embedding nearest to the image embedding, gathers its $N$ nearest neighbors in the class space, and uses their mean as the representative point for similarity scoring.","core_discovery":"The central discovery is that synonymous concepts for an image class do not collapse to a single point in CLIP's text embedding space; they form a continuous, roughly non-overlapping region per class. The paper constructs a synonymous semantic space $S^0_k$ for each class by prompting an LLM for synonyms and visual descriptors, embedding the combined synonymous texts with CLIP's text encoder, building a Vietoris-Rips complex on those embeddings, and keeping only the largest connected component at a similarity threshold to filter out LLM noise and hallucination. Prediction is done by measuring similarity between the image embedding and a local center of the nearest region in each class's space, rather than a point-to-point match. The paper reports that this surpasses prior prompt-engineering and test-time-adaptation methods, and that adding a test-time shift further improves results.","pith_inferences":["Because the method relies on a single global cosine-similarity threshold recommended as 0.9, a natural next test is per-class or per-dataset threshold selection from persistence diagrams; if that closes a meaningful gap, the fixed threshold is a tuning artifact rather than a stable property.","The pipeline is not inherently tied to CLIP's text encoder; the same LLM-synonym generation plus Vietoris-Rips largest-component construction could apply to other embedding models for retrieval or categorization, provided the corresponding image or query embeddings are comparably compact.","A stress test that corrupts a fraction of generated synonymous texts with unrelated but lexically similar phrases would show whether the largest connected component tracks the clean subset or merely trims the distribution, revealing whether persistent homology is genuinely removing hallucinations.","The paper's lexical-variation observations point toward a complementary direction: training or fine-tuning text encoders to separate synonymous classes more sharply could compound the gains from space-based matching."],"forward_implications":["Replacing a single text label with a synonym-derived semantic space lifts average top-1 accuracy from 63.45% to 70.04% on ten fine-grained benchmarks with CLIP ViT-B/16.","Adding test-time adaptation yields T-S3 at 70.32% fine-grained and 65.75% natural-distribution accuracy, exceeding all compared prompt-engineering and test-time methods.","The text-side construction transfers to open-vocabulary segmentation, raising average mIoU to 26.30% with MaskCLIP+ and 50.99% with LSeg+.","The LLM token cost is about $0.6 per 1,000 categories, roughly 6% of the cost of the strongest compared prompt-engineering method while achieving slightly higher accuracy.","The homology-based filtering step contributes an average gain of about 1.54 points over unfiltered synonym sets, showing that removing disconnected noise is doing measurable work."],"supporting_citations":[{"why":"Supplies the CLIP vision and text encoders that define the embedding space and the zero-shot baseline that S3 builds on.","marker":"[32]"},{"why":"Provides the descriptor-generation approach and code base that the paper adapts for generating synonymous texts, and serves as a prompt-engineering baseline.","marker":"[23]"},{"why":"Introduces LLM-generated synonyms for class labels, the direct predecessor that S3 extends from single-synonym replacement to a full semantic space.","marker":"[28]"},{"why":"Establishes the LLM-customized-prompt baseline that S3 compares against on fine-grained and natural-distribution benchmarks.","marker":"[30]"},{"why":"Provides the strongest prompt-engineering baseline and the cost comparison showing S3 achieves higher accuracy at roughly 6% of its LLM token cost.","marker":"[25]"},{"why":"Supplies the computational homology background for constructing the Vietoris-Rips complex used to form the synonymous semantic space.","marker":"[26]"},{"why":"Introduces persistent homology as a text-representation tool, motivating the zero-dimensional persistent-homology largest-connected-component selection.","marker":"[52]"},{"why":"Provides the topological data analysis formulation of birth, death, and persistence that underpins the noise-filtering step.","marker":"[43]"},{"why":"Defines the point-to-set similarity metric that the proposed point-to-local-center metric extends and compares against.","marker":"[51]"}],"fun_headline_variants":["Synonym spaces sharpen zero-shot CLIP","Per-class synonym spaces boost CLIP zero-shot","LLM-generated synonym spaces improve CLIP generalization","Vietoris-Rips semantic spaces lift CLIP zero-shot","From point to space: synonym embeddings for CLIP"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The load-bearing premise is that a single global cosine-similarity threshold, recommended as 0.9 after tuning on the Pets dataset, extracts the largest connected component that corresponds to the true class semantics and filters out LLM hallucinations across every class and all 17 datasets.","fun_headline_variants_meta":{"raw":{"variants":["Synonym spaces sharpen zero-shot CLIP","Per-class synonym spaces boost CLIP zero-shot","LLM-generated synonym spaces improve CLIP generalization","Vietoris-Rips semantic spaces lift CLIP zero-shot","From point to space: synonym embeddings for CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2788,"prompt_tokens":986,"completion_tokens":1802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1729}},"tokens_in":602,"tokens_out":1802,"duration_ms":13594,"temperature":1.0,"reasoning_tokens":1729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:07:34.175283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of classes where an LLM is deliberately prompted to produce a large cluster of off-target synonyms for one class, then run S3 with the fixed 0.9 threshold: if that cluster becomes the largest connected component and classification accuracy collapses for that class while other classes stay unaffected, the global-threshold assumption is falsified. A quantitative variant is to compare the fixed 0.9 threshold against per-class thresholds chosen from each persistence diagram on a held-out set; if per-class thresholds outperform by a nontrivial margin, the paper's construction depends on ungeneralized tuning.","supporting_citations":[{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP vision and text encoders that define the embedding space and the zero-shot baseline that S3 builds on."},{"cited_title":"Visual classification via description from large language models","cited_arxiv_id":null,"evidence_quote":"Provides the descriptor-generation approach and code base that the paper adapts for generating synonymous texts, and serves as a prompt-engineering baseline."},{"cited_title":"The neglected tails in vision-language models","cited_arxiv_id":null,"evidence_quote":"Introduces LLM-generated synonyms for class labels, the direct predecessor that S3 extends from single-synonym replacement to a full semantic space."},{"cited_title":"What does a platypus look like? generating customized prompts for zero-shot image classification","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-customized-prompt baseline that S3 compares against on fine-grained and natural-distribution benchmarks."},{"cited_title":"Meta-prompting for automating zero-shot visual recognition with llms","cited_arxiv_id":null,"evidence_quote":"Provides the strongest prompt-engineering baseline and the cost comparison showing S3 achieves higher accuracy at roughly 6% of its LLM token cost."},{"cited_title":"Computational homology","cited_arxiv_id":null,"evidence_quote":"Supplies the computational homology background for constructing the Vietoris-Rips complex used to form the synonymous semantic space."},{"cited_title":"Persistent homology: An introduction and a new text representation for natural language processing","cited_arxiv_id":null,"evidence_quote":"Introduces persistent homology as a text-representation tool, motivating the zero-dimensional persistent-homology largest-connected-component selection."},{"cited_title":"Topological data analysis","cited_arxiv_id":null,"evidence_quote":"Provides the topological data analysis formulation of birth, death, and persistence that underpins the noise-filtering step."},{"cited_title":"Progressive point to set metric learning for semi-supervised few-shot classification","cited_arxiv_id":null,"evidence_quote":"Defines the point-to-set similarity metric that the proposed point-to-local-center metric extends and compares against."}],"review_version":1}