Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

$S^3$: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Representing each class as a space of synonymous texts improves zero-shot accuracy in vision-language models.

desk verdict The core idea is plausible and cheap, but the headline accuracy numbers are partly tuned on the test sets themselves, so the SOTA claim does not yet hold up to scrutiny. read the letter →

arxiv 2412.04925 v1 pith:DRMFJVTZ submitted 2024-12-06 cs.CV

Xiaojie Yin , Qilong Wang , Bing Cao , Qinghua Hu This is my paper · ORCID
classification cs.CV
keywords zero-shotgeneralizationvision-languagemodelssynonymoussemanticspaceVietoris-Ripscomplexpersistenthomologytest-timeadaptationfine-grainedclassificationlexicalvariation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that zero-shot vision-language models lose accuracy because each class is represented by a single text label, while real images are described by many synonymous phrases. It proposes $S^3$, which generates multiple synonymous texts per class with an LLM and packages them into one continuous, compact semantic space per class using the largest connected component of a Vietoris-Rips complex. Similarity between an image embedding and each class space is then measured with a point-to-local-center metric. On ten fine-grained benchmarks the method reports average top-1 accuracy of 70.04% versus 63.45% for the CLIP baseline, and it also improves natural-distribution and segmentation tasks. If correct, $S^3$ is a cheap, retraining-free upgrade to the text side of CLIP-style models.

What carries the argument

The key machinery is the Vietoris-Rips complex of text embeddings together with zero-dimensional persistent homology: a simplicial complex built by connecting text embeddings whose cosine similarity is at least a threshold $\epsilon$, with the largest connected component at $\epsilon_{\max}$ serving as a compact, noise-filtered class representation. The complementary mechanism is the point-to-local-center metric, which finds the text embedding nearest to the image embedding, gathers its $N$ nearest neighbors in the class space, and uses their mean as the representative point for similarity scoring.

What would settle it

Take a set of classes where an LLM is deliberately prompted to produce a large cluster of off-target synonyms for one class, then run S3 with the fixed 0.9 threshold: if that cluster becomes the largest connected component and classification accuracy collapses for that class while other classes stay unaffected, the global-threshold assumption is falsified. A quantitative variant is to compare the fixed 0.9 threshold against per-class thresholds chosen from each persistence diagram on a held-out set; if per-class thresholds outperform by a nontrivial margin, the paper's construction depends on ungeneralized tuning.

Watch

Extended reading notes

Core claim

The central discovery is that synonymous concepts for an image class do not collapse to a single point in CLIP's text embedding space; they form a continuous, roughly non-overlapping region per class. The paper constructs a synonymous semantic space $S^0_k$ for each class by prompting an LLM for synonyms and visual descriptors, embedding the combined synonymous texts with CLIP's text encoder, building a Vietoris-Rips complex on those embeddings, and keeping only the largest connected component at a similarity threshold to filter out LLM noise and hallucination. Prediction is done by measuring similarity between the image embedding and a local center of the nearest region in each class's space, rather than a point-to-point match. The paper reports that this surpasses prior prompt-engineering and test-time-adaptation methods, and that adding a test-time shift further improves results.

Load-bearing premise

The load-bearing premise is that a single global cosine-similarity threshold, recommended as 0.9 after tuning on the Pets dataset, extracts the largest connected component that corresponds to the true class semantics and filters out LLM hallucinations across every class and all 17 datasets.

Editorial extensions

If this is right

  • Replacing a single text label with a synonym-derived semantic space lifts average top-1 accuracy from 63.45% to 70.04% on ten fine-grained benchmarks with CLIP ViT-B/16.
  • Adding test-time adaptation yields T-S3 at 70.32% fine-grained and 65.75% natural-distribution accuracy, exceeding all compared prompt-engineering and test-time methods.
  • The text-side construction transfers to open-vocabulary segmentation, raising average mIoU to 26.30% with MaskCLIP+ and 50.99% with LSeg+.
  • The LLM token cost is about $0.6 per 1,000 categories, roughly 6% of the cost of the strongest compared prompt-engineering method while achieving slightly higher accuracy.
  • The homology-based filtering step contributes an average gain of about 1.54 points over unfiltered synonym sets, showing that removing disconnected noise is doing measurable work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the method relies on a single global cosine-similarity threshold recommended as 0.9, a natural next test is per-class or per-dataset threshold selection from persistence diagrams; if that closes a meaningful gap, the fixed threshold is a tuning artifact rather than a stable property.
  • The pipeline is not inherently tied to CLIP's text encoder; the same LLM-synonym generation plus Vietoris-Rips largest-component construction could apply to other embedding models for retrieval or categorization, provided the corresponding image or query embeddings are comparably compact.
  • A stress test that corrupts a fraction of generated synonymous texts with unrelated but lexically similar phrases would show whether the largest connected component tracks the clean subset or merely trims the distribution, revealing whether persistent homology is genuinely removing hallucinations.
  • The paper's lexical-variation observations point toward a complementary direction: training or fine-tuning text encoders to separate synonymous classes more sharply could compound the gains from space-based matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes S3, a text-side enhancement to CLIP zero-shot classification. For each class, an LLM generates synonyms and descriptors, which are combined into 'synonymous texts'; these are embedded with CLIP's text encoder, and a Vietoris-Rips complex with a fixed cosine threshold is used to select the largest connected component as a 'synonymous semantic space.' Similarity between a test image and each class space is computed by a proposed point-to-local-center metric (nearest text embedding, then average of its N nearest neighbors). The paper reports gains over prompt-engineering and test-time-adaptation baselines on 10 fine-grained, 5 distribution-shift, and 2 segmentation benchmarks, plus a TTA variant called T S3. The supplementary material fixes epsilon_max=0.9 and N in [10,30] using Oxford Pets accuracy and selects Claude over GPT-4 using average accuracy over the same ten benchmarks.

Significance. If the results are robust, S3 offers a simple, cost-effective drop-in replacement for the text branch of CLIP classification, with a genuinely interesting structural idea: replacing point text embeddings with a semantic space addresses lexical variation. The paper's strengths include experiments across 17 benchmarks, multiple CLIP backbones, a token-cost analysis, and an implemented TTA extension. However, as submitted, the main claim is not yet cleanly supported because key hyperparameters are selected on test sets included in the reported averages, and the persistent-homology construction is not derived from the actual computation. The central idea is worth publishing if those issues are addressed.

major comments (4)
  1. [§4.1, Supplementary B, Figs. S1–S2, Table S2] The reported averages are not obtained from a clean validation protocol. The similarity threshold epsilon_max=0.9 and neighborhood size N in [10,30] are selected by maximizing accuracy on Oxford Pets, which is one of the ten datasets included in the reported 70.04 average, and the choice of Claude over GPT-4 is made using average accuracy on the same ten benchmarks (Table S2). Under this protocol, part of the observed improvement over baselines may be test-set selection. Please either report a validation-based selection rule (for example, selecting hyperparameters on a separate dataset or on held-out class splits) and give sensitivity of the 70.04/70.32 averages to epsilon and N, or clearly state that the published numbers are post-selection.
  2. [§3.2, Eq. (3)–(4)] The described procedure is not persistent homology as written. In the Vietoris-Rips complex defined by the condition <fi,fj> >= epsilon, increasing epsilon removes edges and makes the complex sparser, so the sentence 'As epsilon increases, the connectivity within R_epsilon(S_k) evolves' is backwards. Moreover, the largest connected component at a single fixed epsilon=0.9 is an ordinary graph component, not an object selected by the generator with maximal lifespan; the paper never computes birth/death times or shows how epsilon_max is determined from persistence. If epsilon_max is simply a tuned cosine threshold, then the persistent-homology claim in the abstract and in Section 3.2 is unsupported, and Eq. (4) is a definition rather than a topological derivation. Please rewrite the construction in standard filtration terms or remove the persistent-homology claim.
  3. [Table 4, Table S3] The homology ablation shows an average gain of only 1.5 points and highly variable per-dataset gains (EuroSAT +8.72, but many datasets +0.1 to +0.5). This is the only evidence that the topological filter contributes beyond the LLM-generated text set. Please specify precisely what 'w/o homology' uses (all generated texts, or some other graph construction), report per-dataset optimal epsilon, and compare the largest-component selection against using all generated texts or a simple k-nearest-neighbor core at a fixed epsilon. Without this, the central method component is not isolated and the 1.5-point gain may be an artifact of the Pets-tuned threshold.
  4. [§4.3, Table 2] On the natural distribution benchmarks, S3's average gain over DCLIP and CuPL is only 0.09–0.13 points, which is likely within run-to-run or benchmark noise. The text states that S3 'surpasses most state-of-the-art baseline methods' and 'shows an overall improvement of 0.13%' over CuPL, but such a small margin does not support a strong state-of-the-art claim without confidence intervals or multiple-seed results. Please either temper the claim or provide statistical support for the fine-grained and natural-distribution comparisons.
minor comments (6)
  1. [§4.5, Table 4] The phrase 'bottle half' should read 'bottom half'; this appears in the discussion of the homology ablation.
  2. [§4.2] The text says 'much more cost-effective than MVPR'; the method name is MPVR, as used in Table 1 and the related work section.
  3. [§4.5, Table S2] The sentence 'Claude consistently outperformed GPT-4' is contradicted by Table S2, where GPT-4 is higher on Aircraft and EuroSAT; 'consistently' should be replaced by 'on average'.
  4. [§3.2, Eq. (4)] The notation is ambiguous: R_epsilon(S_k) is a set of simplexes, so the union of all its elements is just the vertex set. Please define S0_k as the vertex set (or induced subgraph) of the largest connected component of the graph at threshold epsilon_max.
  5. [§3.4] The T S3 adaptation is described only verbally; the entropy-minimization objective, the definition of the learnable shift vector v_k, and the gradient update rule should be written out or explicitly referenced to TPT/TPS equations, otherwise the reported 70.32 result is not reproducible.
  6. [Figures S1–S2] The accuracy curves used to set epsilon_max and N show no error bars or multiple runs; since these figures are the basis for the main evaluation, reporting variance would strengthen the robustness of the chosen hyperparameters.

Circularity Check

2 steps flagged · score 4.0 of 10

Pets-tuned epsilon/N and test-set LLM choice contaminate the fine-grained average; the S3 construction itself is not circular.

  1. fitted input called prediction [Supplementary B (Figures S1/S2); Sec. 3.2 Eq. (4); Sec. 4.2 Table 1, Pets row]
    "Figure S1 illustrates the top-1 accuracy across various similarity thresholds on the Pets dataset. ... Consequently, we recommend setting the similarity threshold hyperparameter to 0.9 for optimal performance."

    The threshold epsilon_max used in Eq. (4) is not determined by persistent homology alone; Supplementary B sets it by maximizing top-1 accuracy on the Oxford Pets test set, and Figure S2 similarly picks the neighborhood size N from Pets accuracy. The same Pets test set then appears as the 'Pets' column in Table 1 (91.58) and contributes to the headline 70.04 average. The Pets result is therefore an optimized fit to that test set rather than an independent zero-shot prediction, so the comparison against DCLIP/CuPL/REAL/MPVR on Pets is partly forced by construction.

  2. other [Sec. 4.5 Table 4; Supplementary Table S2; Sec. 4.2 Table 1]
    "Claude consistently outperformed GPT-4 by an average of ∼1.2% on all datasets. Consequently, we selected Claude for synonym generation."

    The LLM that defines 'S3 (Ours)' is chosen after measuring zero-shot accuracy on the same ten fine-grained test sets reported in Table 1. The reported 70.04 average is therefore conditional on a model-selection step that used those test labels, rather than being the output of a fixed, precommitted pipeline. This is evaluation-set selection rather than a mathematical derivation cycle, but it makes the top-line fine-grained number partly fitted to the benchmark.

full rationale

The S3 construction itself is not circular: nothing in Eqs. (1)-(6) is fitted to the image labels of the evaluation benchmarks, and the method is tested against external CLIP zero-shot baselines and segmentation tasks. The circularity that exists is confined to the evaluation protocol. Supplementary B fixes epsilon_max=0.9 and N in [10,30] by maximizing top-1 accuracy on the Pets test set, and then the same Pets test-set number (91.58) is included in the headline 70.04 fine-grained average; this is a fitted-input-called-prediction loop for that dataset. In addition, the LLM used to generate the texts (Claude) is selected in Sec. 4.5 and Table S2 after comparing average accuracy on the same ten fine-grained test sets, so the top-line fine-grained average is partly conditioned on the benchmarks. The natural-distribution and segmentation results were not used for these selections and remain independent evidence. There is no load-bearing self-citation or imported uniqueness theorem: [51] is a co-authored reference used only for the standard point-to-set metric, and the method's core is empirical rather than a derivation from prior work. Score 4 reflects this partial, evaluation-side circularity without claiming the whole derivation reduces to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method's core depends on empirically motivated assumptions about CLIP embedding geometry and on two hyperparameters tuned on the evaluation data. No new physical entities are introduced.

free parameters (3)
  • epsilon_max (Vietoris-Rips similarity threshold) = 0.9
    Tuned on the Pets dataset to maximize accuracy; applied globally to all classes and datasets.
  • N (neighborhood size in point-to-local-center) = 10-30 range
    Recommended based on Pets; exact value used for each dataset not reported.
  • LLM choice = Claude-3.5-Sonnet
    Selected because it outperformed GPT-4 by 1.16% average on the 10 fine-grained test sets.
assumptions (3)
  • domain assumption Synonymous textual concepts form continuous, non-overlapping regions in CLIP's embedding space.
    Motivates the space construction; supported only by qualitative visualization on LAION-400M (Fig. 2c) and previous alignment claims.
  • ad hoc to paper The largest connected component of the Vietoris-Rips complex at epsilon_max contains the class's true semantic space and excludes hallucinated text.
    Introduced specifically for this method; no external or formal justification that this topological operation selects the desired subset.
  • domain assumption Averaging the N text embeddings nearest to the image gives a better similarity estimate than the centroid or a principal subspace.
    Empirically validated only on the same benchmarks; no principled derivation of why local averaging is optimal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $S^3$: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models." pith.science (2026). https://pith.science/paper/DRMFJVTZ

@misc{pith2026241204925,
  author       = {Pith},
  title        = {Pith review of: $S^3$: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRMFJVTZ}},
  note         = {Machine review of arXiv:2412.04925}
}
abstract

Recently, many studies have been conducted to enhance the zero-shot generalization ability of vision-language models (e.g., CLIP) by addressing the semantic misalignment between image and text embeddings in downstream tasks. Although many efforts have been made, existing methods barely consider the fact that a class of images can be described by notably different textual concepts due to well-known lexical variation in natural language processing, which heavily affects the zero-shot generalization of CLIP. Therefore, this paper proposes a \textbf{S}ynonymous \textbf{S}emantic \textbf{S}pace ($S^3$) for each image class, rather than relying on a single textual concept, achieving more stable semantic alignment and improving the zero-shot generalization of CLIP. Specifically, our $S^3$ method first generates several synonymous concepts based on the label of each class by using large language models, and constructs a continuous yet compact synonymous semantic space based on the Vietoris-Rips complex of the generated synonymous concepts. Furthermore, we explore the effect of several point-to-space metrics on our $S^3$, while presenting a point-to-local-center metric to compute similarity between image embeddings and the synonymous semantic space of each class, accomplishing effective zero-shot predictions. Extensive experiments are conducted across 17 benchmarks, including fine-grained zero-shot classification, natural distribution zero-shot classification, and open-vocabulary segmentation, and the results show that our $S^3$ outperforms state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2412.04925 by the authors.

Figure 1
Figure 1. Comparison of Methods. (a) CLIP: Point-to-point sim￾ilarity between image and label embeddings. (b) PE: Point-to￾point similarity between image and single concept embeddings. (c) TTA: Point-to-point similarity between image and shifted text embeddings. (d) S 3 (Ours): Similarity between image and seman￾tic spaces constructed from multiple synonymous concepts. tion by directly comparing embeddings of input images and… view at source ↗
Figure 2
Figure 2. (a) Lexical variation in LAION-400M dataset: Images of the same class with very similar visual embeddings correspond to significantly different text embeddings, which may even belong to different textual concepts. (b) Compactness: image v.s. text: Image embeddings (blue) are consistently more compact than text embeddings (red). The original data (light color) has been smoothed. (c) Synonymous concepts form semantic … view at source ↗
Figure 3
Figure 3. Overall architecture of S 3 . Given label of each class, our S 3 method generates synonymous texts by prompting LLMs, which are used to construct a synonymous semantic space by seeking the largest connected component in topological properties of semantic space. For a test image, similarities between image embedding and synonymous semantic spaces are calculated for zero-shot prediction. The overview of our S 3 method… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Generating Synonymous Texts. A class name (e.g., “sunflower”) and its dataset name (e.g., “flowers”) are given as inputs to the LLMs through two prompts. The first generates syn￾onyms (e.g., “sunflower”, “helianthus”), and the second provides descriptors (e.g., “large,…
Figure 5
Figure 5. Figure 5: Point-to-Space Similarity Metric: (a) Point-to-Set. (b) Point-to-Center. (c) Point-to-Subspace. (d) Point-to-Local-Center. in downstream tasks. To overcome above challenges, our generate diverse synonymous texts with prompting pow￾erful LLMs, and construct a synonymous…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adapting Vision-Language Models Without Labels: A Comprehensive Survey

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A survey that organizes unsupervised vision-language model adaptation by unlabeled-data availability into four paradigms: data-free transfer, domain transfer, episodic test-time, and online test-time adaptation.

Reference graph

Works this paper leans on

53 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization

    Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hus- sein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization. NeurIPS, 36, 2024. 2, 4

    work page 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Evaluating clip: towards characterization of broader capabilities and downstream implications

    Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Rad- ford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818,

    arXiv

  4. [4]

    A simple zero-shot prompt weight- ing technique to improve prompt ensembling in text-image models

    James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Bal- aji Lakshminarayanan. A simple zero-shot prompt weight- ing technique to improve prompt ensembling in text-image models. In International Conference on Machine Learning, pages 547–568, 2023. 1, 3

    work page 2023

  5. [5]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku

  6. [6]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In ECCV, pages 446–461, 2014. 3, 7

    work page 2014

  7. [7]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606–3613, 2014. 3, 7

    work page 2014

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 3, 7

    work page 2009

Show all 53 references
  1. [9]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88:303–338, 2010. 3, 8

    2010

  2. [10]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In CVPR Workshop, pages 178–178, 2004. 3, 7

    2004

  3. [11]

    Diverse data augmentation with diffusions for effective test-time prompt tuning

    Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In ICCV, pages 2704– 2714, 2023. 2, 3, 6, 7

    2023

  4. [12]

    Dall-e for detection: Language- driven compositional image synthesis for object detection

    Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi, Lau- rent Itti, and Vibhav Vineet. Dall-e for detection: Language- driven compositional image synthesis for object detection. arXiv preprint arXiv:2206.09592, 2022. 1

    2022 arXiv

  5. [13]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 3, 7

    2019

  6. [14]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349, 2021. 3, 8

    2021

  7. [15]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262–15271, 2021. 3, 7

    2021

  8. [16]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions. arXiv preprint arXiv:2311.05...

    2023 arXiv

  9. [17]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR,

  10. [18]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, pages 554–561, 2013. 3, 7

    2013

  11. [19]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation. arXiv preprint arXiv:2201.03546, 2022. 7

    2022 arXiv

  12. [20]

    Swapprompt: Test-time prompt adaptation for vision- language models

    Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision- language models. NeurIPS, 36, 2024. 2, 4

    2024

  13. [21]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 3, 7

    2013 arXiv

  14. [22]

    A survey on bias and fairness in machine learning

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) , 54 (6):1–35, 2021. 1

    2021

  15. [23]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. ICLR, 2022. 1, 3, 5, 6, 7

    2022

  16. [24]

    Task bias in vision-language models

    Sachit Menon, Ishaan Preetam Chandratreya, and Carl V on- drick. Task bias in vision-language models. arXiv preprint arXiv:2212.04412, 2022. 1

    2022 arXiv

  17. [25]

    Meta-prompting for automating zero-shot visual recognition with llms

    M Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger. Meta-prompting for automating zero-shot visual recognition with llms. In ECCV, pages 1–30, 2024. 1, 3, 6, 7

    2024

  18. [26]

    Computational homology

    K Mischaikow, T Kaczynski, and M Mrozek. Computational homology. Applied Mathematical Sciences, 157, 2004. 2, 5

    2004

  19. [27]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In In- dian Conference on Computer Vision, Graphics and Image Processing, pages 722–729, 2008. 3, 4, 7

    2008

  20. [28]

    The neglected tails in vision-language models

    Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision-language models. In CVPR, pages 12988–12997, 2024. 1, 2, 3, 4, 6, 7

    2024

  21. [29]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498–3505,

  22. [30]

    What does a platypus look like? generating customized prompts for zero-shot image classification

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pages 15691– 15701, 2023. 1, 3, 6, 7

    2023

  23. [31]

    Online zero-shot classification with clip

    Qi Qian and Juhua Hu. Online zero-shot classification with clip. In ECCV, pages 462–477. Springer, 2024. 6, 7

    2024

  24. [32]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  25. [33]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning, pages 8821–8831. PMLR, 2021. 1

    2021

  26. [34]

    Do imagenet classifiers generalize to im- agenet? In International Conference on Machine Learning, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019. 3, 7

    2019

  27. [35]

    Waffling around for performance: Visual classification with random words and broad concepts

    Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV, pages 15746–15757, 2023. 1, 3

    2023

  28. [36]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 1, 2, 4

    2021 arXiv

  29. [37]

    Laion-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for train- ing next generation image-text models. NeurIPS, 35:25278– 252...

    2022

  30. [38]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. NeurIPS, 35:14274–14289, 2022. 1, 2, 3, 6, 7

    2022

  31. [39]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  32. [40]

    Just shift it: Test-time prototype shifting for zero-shot gen- eralization with vision-language models

    Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot gen- eralization with vision-language models. arXiv preprint arXiv:2403.12952, 2024. 1, 2, 4, 6, 7

    2024 arXiv

  33. [41]

    Eigenfaces for recogni- tion

    Matthew Turk and Alex Pentland. Eigenfaces for recogni- tion. Journal of cognitive neuroscience , 3(1):71–86, 1991. 5

    1991

  34. [42]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurIPS, 32, 2019. 3, 8

    2019

  35. [43]

    Topological data analysis

    Larry Wasserman. Topological data analysis. Annual Review of Statistics and Its Application, 5(1):501–532, 2018. 5

    2018

  36. [44]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer So- ciety Conference on Computer Vision and Pattern Recogni- tion, pages 3485–3492. IEEE, 2010. 3, 7

    2010

  37. [45]

    Alip: Adaptive language-image pre-training with synthetic cap- tion

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic cap- tion. In ICCV, pages 2922–2931, 2023. 4

    2023

  38. [46]

    On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? In CVPR, pages 23783–23793, 2024

    Maxime Zanella and Ismail Ben Ayed. On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? In CVPR, pages 23783–23793, 2024. 2, 4, 6, 7

    2024

  39. [47]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18123–18133, 2022. 1

    2022

  40. [48]

    Siren’s song in the ai ocean: a survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yu- long Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. 2, 5

    2023 arXiv

  41. [49]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. IJCV, 127: 302–321, 2019. 3, 8

    2019

  42. [50]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pages 696–712. Springer,

  43. [51]

    Progressive point to set metric learning for semi-supervised few-shot classification

    Pengfei Zhu, Mingqi Gu, Wenbin Li, Changqing Zhang, and Qinghua Hu. Progressive point to set metric learning for semi-supervised few-shot classification. In IEEE Inter- national Conference on Image Processing , pages 196–200. IEEE, 2020. 5

    2020

  44. [52]

    Persistent homology: An introduction and a new text representation for natural language processing

    Xiaojin Zhu. Persistent homology: An introduction and a new text representation for natural language processing. In IJCAI, pages 1953–1959, 2013. 2, 5 S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models Supplementary Material A. Analy...

    1953

  45. [53]

    Therefore, we recommend setting the neighborhood size hyperparameter within this range to achieve the best results. C. Detailed Results on Ablation Study Selection of LLMs. Table S2 presents the detailed results of two leading text generation LLMs, GPT-4 and Claude for synonym...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.