Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLIP's contrastive pretraining leaves intra-modal distances miscalibrated, so image-to-image and text-to-text retrieval improve when queries are first inverted into the complementary modality.

desk verdict Empirically solid demonstration that CLIP intra-modal retrieval is miscalibrated and improves via inter-modal inversion; the mechanism is less proven than the effect. read the letter →

arxiv 2502.04263 v1 pith:OGIK6SXO submitted 2025-02-06 cs.CV cs.AIcs.LG

Marco Mistretta , Alberto Baldrati , Lorenzo Agnolucci , Marco Bertini , Andrew D. Bagdanov This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords CLIPintra-modalmisalignmentmodalitygapinversionimage-to-imageretrievaltext-to-textcontrastivelearningzero-shotclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that CLIP-style models, trained to match images to texts, never learn to compare images with images or texts with texts. As a result, off-the-shelf CLIP encoders give miscalibrated similarity scores for intra-modal tasks such as image-to-image retrieval. The authors show that inverting a query feature into the complementary modality — an image into text space, or a text into image space — and then comparing cross-modally consistently improves retrieval across more than fifteen datasets. They also show the reverse transformation hurts natively inter-modal tasks like zero-shot image classification, and that adding an intra-modal loss during pretraining or narrowing the modality gap reduces the effect. The finding matters because many applications reuse CLIP encoders for same-modality similarity without accounting for this bias.

What carries the argument

The load-bearing mechanism is optimization-based modality inversion. OTI (Optimization-based Textual Inversion) optimizes a small set of pseudo-tokens, appended to a fixed template like "a photo of", so that the CLIP text encoder reproduces a query image's embedding; OVI (Optimization-based Visual Inversion), introduced here, optimizes pseudo-patches so that the CLIP image encoder reproduces a query text's embedding. Both minimize the same cosine loss $\mathcal{L}_{\text{cos}} = 1 - c(\psi_I, \psi_T)$ with frozen encoders, at the single-feature level, with no external data or trained adapters. The key property is that the inverted feature stays on the complementary modality's manifold while retaining the query's semantic content; the paper shows retrieval peaks at partial inversion and degrades toward the intra-modal baseline as the loss approaches zero.

What would settle it

The paper's mechanism predicts a specific curve: image-to-image retrieval accuracy should rise as a query is partially inverted, peak before the cosine loss reaches zero, and fall back to the intra-modal baseline as the inverted feature drifts onto the native manifold. A monotone curve, or a peak at full inversion, would falsify the claim that inter-modal alignment, rather than the inversion itself, drives the improvement.

Watch

Extended reading notes

Core claim

The central discovery is that intra-modal misalignment — the uncalibrated distances between two image embeddings or two text embeddings produced by CLIP — is a real and exploitable defect of inter-modal contrastive pretraining. The paper claims that comparing a query's representation to gallery items in the complementary modality, after optimization-based modality inversion, outperforms the standard intra-modal baseline on image-to-image and text-to-text retrieval. It further claims the same inverted features hurt when used to make a natively inter-modal task (zero-shot image classification) intra-modal, which isolates the source of the gain as inter-modal alignment rather than a generic improvement from inversion. Finally, it claims that pretraining objectives that include an intra-modal term (SLIP) or that narrow the modality gap reduce the misalignment.

Load-bearing premise

The claim rests on the assumption that OTI and OVI invert a feature faithfully, preserving semantic content while changing only the modality manifold; if the inversion is instead a generic feature transform or denoiser, the retrieval gains would not specifically implicate intra-modal misalignment.

Editorial extensions

If this is right

  • Any application that uses raw CLIP image-image or text-text cosine similarity — k-NN classification, temporal consistency checks in video synthesis, visual memory retrieval — inherits the miscalibration and should expect gains from cross-modal reformulation.
  • For image-to-image retrieval, the inter-modal route via OTI gives consistent absolute mAP gains of about 2–3 points average across 15 datasets, with similar gains on text-to-text retrieval.
  • Zero-shot classification made intra-modal degrades, so gains from inversion are not a generic artifact of the inversion optimization but depend on preserving inter-modal alignment.
  • Adding an intra-modal self-supervised loss during pretraining (as in SLIP) nearly eliminates the inter-modal advantage, indicating a training-time remedy.
  • Closing the modality gap by raising the contrastive temperature removes the inter-modal advantage, although it also lowers overall task performance, so temperature alone is not a practical fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use the inter-modal versus intra-modal performance gap as a diagnostic for intra-modal misalignment in any contrastive vision-language model, not just CLIP.
  • The partial-inversion peak suggests a tunable interpolation between native and complementary representations; the paper's own combination experiment (native plus inverted features) already shows further gains at intermediate weights, which could be turned into a per-query calibration.
  • The paper's limitation note — 150 OTI and 1000 OVI optimization steps per query — implies the method is too slow for large galleries as-is; cheaper inversions or pretraining-time intra-modal losses are the practical pathways.
  • A testable prediction follows: if intra-modal misalignment is the cause, then per-sample gains from inter-modal retrieval should correlate with the distance of that sample's embedding from the complementary modality manifold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses the use of CLIP-style vision-language models as frozen feature extractors for intra-modal tasks such as image-to-image and text-to-text retrieval. The authors argue that CLIP's inter-modal contrastive training leaves intra-modal similarities uncalibrated, a phenomenon they call "intra-modal misalignment." To test this, they convert intra-modal tasks into inter-modal ones using two optimization-based inversion methods: OTI, which optimizes pseudo-tokens so that the text encoder produces a feature matching the query image feature, and OVI, which analogously optimizes pseudo-patches so that the image encoder produces a feature matching the query text feature. Experiments on 15 image retrieval datasets and several caption/text datasets across CLIP, OpenCLIP, SigLIP, and SLIP show consistent improvements for the inter-modal formulation over direct intra-modal similarities. Complementary experiments show that making a natively inter-modal task (zero-shot classification) intra-modal degrades performance, that SLIP's intra-modal loss reduces the effect, and that closing the modality gap via high-temperature fine-tuning removes the gain.

Significance. If the interpretation is correct, the work is significant because it identifies a systematic deficiency in a widely used practice and provides a simple, training-free recipe for avoiding it. The empirical design is a strength: results are reported across multiple model families, backbones, datasets, and loss functions, and the paper includes important controls (intra-OTI comparisons, zero-shot classification degradation, adapter- and captioning-based alternatives). The public code and reproducibility statement further support the contribution. The main open question is whether the inversion procedures are faithful enough to support the mechanistic claim that the gains come from exploiting CLIP's inter-modal alignment rather than from an arbitrary feature transformation. The existing controls make this interpretation plausible, but the manuscript does not yet directly demonstrate that OTI/OVI features lie on the complementary modality's natural manifold.

major comments (3)
  1. [Sec. 4.1, Eq. (2), Fig. 2(c)] The central mechanistic claim that OTI/OVI produce faithful cross-modal surrogates is not directly established. Removing the vocabulary regularization (Sec. 4.1) leaves the pseudo-tokens free in the continuous token-embedding space, and Eq. (2) alone does not constrain the input distribution of the text encoder. The distribution analysis in Fig. 2(c) compares aggregate cosine-similarity histograms, which is not a manifold-membership test. I suggest adding a direct check: report the distance of the optimized v* to the nearest real token embeddings, or compare OTI-inverted features to real text features using a nearest-neighbor or density-based criterion (e.g., the fraction of real captions among the top-K neighbors of the inverted feature). Without such evidence, the alternative explanation that OTI acts as a generic feature transform rather than a genuine modality inversion remains open.
  2. [Sec. 5.6, Table 4] The conclusion that closing the modality gap reduces intra-modal misalignment is not cleanly supported by the reported experiment. The τ = 1 fine-tuned model has substantially lower intra-modal baseline performance (average mAP drops from 39.3 to 27.0), so the absence of an inter-modal gain could be due to the overall weakness of the fine-tuned features rather than to the gap closure. The authors acknowledge the performance drop but still interpret the null effect causally. A cleaner control would be to reduce the modality gap without degrading downstream performance, for example by applying a projection-based gap-removal technique or by comparing models matched on intra-modal baseline accuracy.
  3. [Tables 1 and 2, abstract] The abstract and body use the word "significantly" to describe the improvements, but the paper reports no variance estimates or statistical tests. Many per-dataset gains are small (for example, CLIP B/32 on DTD changes from 14.5 to 14.4, and on EuroSAT from 47.9 to 47.2), and it is unclear whether these fluctuations are consistent across runs. I ask for standard deviations over at least three random seeds or a paired significance test across the 15 datasets to support the strength of the claim, or alternatively for a softer wording such as "consistently improves."
minor comments (4)
  1. [Figure 2] The x-axis label contains a typo: "Number of otimization steps" should be "Number of optimization steps."
  2. [Appendix B] The filtering procedure described in the Dogs-vs-Cats toy experiment is not fully specified: "filter out the minimal number of images that are incorrectly ranked" should define how the minimal subset is computed and whether the same procedure is applied independently to each query.
  3. [Appendix G] The NanoBEIR benchmark is mentioned without a formal citation; please provide a reference or a URL in the bibliography rather than only in a footnote.
  4. [Sec. 5.1] The statement that OTI-inverted features "cannot contain more informative content than the native ones" is intuitive but not a formal guarantee, since the optimization could in principle amplify or distort information; consider softening the wording or adding a small analysis of information preservation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is established by held-out retrieval evaluations and control experiments, not by construction or by load-bearing self-citation.

full rationale

The paper's claim that inter-modal comparison via OTI/OVI outperforms intra-modal similarity is an empirical result, not a derivation. OTI/OVI are fully specified in Algorithms 1 and 2 and are evaluated on held-out benchmarks in Tables 1, 2, A4, A5, A6, A7, and A8. No equation reduces to a fitted value: the cosine loss in Eq. (2) is an optimization objective, not a fitted parameter renamed as a prediction. The citations to Baldrati et al. (2023) and Agnolucci et al. (2024) are method provenance for OTI; the paper re-describes and evaluates the algorithm itself, so these self-citations are not load-bearing evidence. The paper's control experiments (intra-OTI in Table A8, zero-shot classification degradation in Tables 2 and A3, and adapter/captioning comparisons in Tables A6 and A7) actively test the alternative that inversion itself, rather than inter-modal alignment, causes the gains. The drift discussion in Sec. 4.3 and Fig. 2(c) is a post-hoc distributional analysis, not a definitional equivalence. The skeptical concern that OTI/OVI may produce off-manifold features is a correctness risk about semantic preservation, not a circularity, because the paper does not define 'intra-modal misalignment' in terms of the OTI output. The limitation statement about computational cost is a practical caveat, not a circular step. Overall, the derivation chain is self-contained and empirically grounded.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the empirical interpretation of the inversion as a faithful cross-modal surrogate, on the causal attribution of misalignment to the contrastive loss, and on a few hand-selected hyperparameters. There are no fitted theory parameters or invented physical entities beyond the named phenomenon of intra-modal misalignment.

free parameters (4)
  • Number of OVI pseudo-patches P per model = 1 to 4 depending on VLM (selected via Flickr30K validation, Table A1)
    Selected per model to maximize text-to-text retrieval mAP on a validation set; the reported OVI gains depend on this choice.
  • Number of OTI pseudo-tokens R = 1 (robustness choice; Fig. 2b shows R>1 can peak higher)
    Set to 1 to limit drift toward the image manifold; affects the magnitude of the reported image-to-image gains.
  • Optimization steps S for OTI and OVI = 150 (OTI), 1000 (OVI)
    Chosen by hand; Fig. 2 shows retrieval performance peaks early in optimization and degrades with additional steps, so the reported figures depend on this choice.
  • Fine-tuning temperature tau for modality gap experiment = 1.0 (gap closed) and 0.01 (reference)
    Used to test gap closure; the conclusion that gap closure removes the inter-modal improvement depends on these values and on the COCO fine-tuning procedure.
assumptions (4)
  • domain assumption CLIP's symmetric contrastive loss (Eq. 1) does not enforce intra-modal constraints, so intra-modal distances are uncalibrated.
    Central premise that motivates the study; supported by the geometry argument in Sec. 3 and the toy experiment in Appendix B, but treated as an axiom for interpreting the retrieval gains.
  • domain assumption The modality gap exists in CLIP and is caused by initialization and the contrastive loss, with magnitude related to temperature.
    Relied on from Liang et al. (2022) to motivate OTI/OVI and the modality gap experiment in Sec. 5.6.
  • ad hoc to paper A cosine-loss optimization of pseudo-tokens or pseudo-patches against a query feature yields a faithful cross-modal surrogate that preserves semantics.
    The weakest assumption; the interpretation of retrieval improvements as evidence of inter-modal alignment requires this to hold (Secs. 4.1 and 4.3).
  • domain assumption Captions associated with the same image are a valid ground truth for text-to-text retrieval.
    Used to construct the text-to-text task in Sec. 5.2; assumes near-synonymy of multiple captions of one image.
invented entities (1)
  • Intra-modal misalignment independent evidence
    purpose: Names the claimed phenomenon that CLIP's same-modality similarities do not reflect true image or text similarity
    Operationalized through retrieval experiments: inter-modal querying improves intra-modal tasks and intra-modal querying degrades inter-modal tasks, giving falsifiable handles outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion." pith.science (2026). https://pith.science/paper/OGIK6SXO

@misc{pith2026250204263,
  author       = {Pith},
  title        = {Pith review of: Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGIK6SXO}},
  note         = {Machine review of arXiv:2502.04263}
}
read the original abstract

Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

Figures

Figures reproduced from arXiv: 2502.04263 by the authors.

Figure 1
Figure 1. Motivation and overview. Left: The inter-modal contrastive loss used in pretraining enforces paired images and texts to be at a given distance r (i.e. rdog and rcat) but does not encourage intra-modal alignment. Consequently, intra-modal similarity scores might not correspond to those of actual images and texts (i.e. d1 < d2). Right: We show that the common practice of individually exploiting only one encoder is sub… view at source ↗
Figure 2
Figure 2. (a, b) Loss values and retrieval performance over OTI optimization steps for different numbers of pseudo-tokens R. (c) Distribution of pairwise image-image, text-image, and OTI-image cosine similarities. We consider the OTI-inverted features using four pseudo-tokens (R = 4) at two distinct optimization steps: the performance peak (step 17) and the final step (step 1000). In Figs. 2(a) and 2(b) we investigate how the… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MG-CLIP preserves CLIP's modality gap by adaptively limiting fine-tuning epochs and compensates for its limits with a visual-space classifier, improving class-incremental learning without replay.

  2. Global and Local Entailment Learning for Natural World Imagery

    cs.CV 2025-06 conditional novelty 6.0 of 10

    RCME enforces a transitivity constraint in radial embeddings, producing a hierarchical vision-language model that orders taxonomic labels better and improves hierarchical classification and retrieval.

  3. Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Current vision-language models are largely miscalibrated when they verbalize confidence, visual reasoning models such as o3 and o4-mini are better calibrated, and Visual Confidence-Aware Prompting reduces ECE on IsoBench.

Reference graph

Works this paper leans on

61 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval

    Lorenzo Agnolucci, Alberto Baldrati, Marco Bertini, and Alberto Del Bimbo. iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval . arXiv preprint arXiv:2405.02951, 2024

    arXiv 2024

  3. [3]

    nocaps: novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE International Conference on Computer Vision, pp.\ 8948--8957, 2019

    work page 2019

  4. [4]

    Zero-Shot Composed Image Retrieval with Textual Inversion

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-Shot Composed Image Retrieval with Textual Inversion . In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15338--15347, 2023

    work page 2023

  5. [5]

    Food-101--mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101--mining discriminative components with random forests. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp.\ 446--461. Springer, 2014

    work page 2014

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

    2020

  7. [7]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3606--3613, 2014

    2014

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

Show all 61 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con...

    2020

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Asirra: a captcha that exploits interest-aligned manual image categorization

    Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul. Asirra: a captcha that exploits interest-aligned manual image categorization. CCS, 7: 0 366--374, 2007

    2007

  4. [12]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7346--7356, 2023

    2023

  5. [13]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp.\ 178--178. IEEE, 2004

    2004

  6. [14]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

    2024

  7. [15]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  8. [16]

    Towards flexible perception with visual memory

    Robert Geirhos, Priyank Jaini, Austin Stone, Sourabh Medapati, Xi Yi, George Toderici, Abhijit Ogale, and Jonathon Shlens. Towards flexible perception with visual memory. arXiv preprint arXiv:2408.08172, 2024

    2024 arXiv

  9. [17]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12 0 (7): 0 2217--2226, 2019

    2019

  10. [18]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp.\ 4904--4916....

    2021

  11. [19]

    Deep visual-semantic alignments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3128--3137, 2015

    2015

  12. [20]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp.\ 554--561, 2013

    2013

  13. [21]

    Newsweeder: Learning to filter netnews

    Ken Lang. Newsweeder: Learning to filter netnews. In Machine learning proceedings 1995, pp.\ 331--339. Elsevier, 1995

    1995

  14. [22]

    Decap: Decoding CLIP latents for zero-shot captioning via text-only training

    Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding CLIP latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303.03032, 2023

    2023 arXiv

  15. [23]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021

    arXiv

  16. [24]

    Mind the Gap: Understanding the Modality Gap in Multi-Modal Contrastive Representation Learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the Gap: Understanding the Modality Gap in Multi-Modal Contrastive Representation Learning . Advances in Neural Information Processing Systems, 35: 0 17612--17625, 2022

    2022

  17. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

    2014

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  19. [27]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  20. [28]

    Image segmentation using text and image prompts

    Timo L \"u ddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 7086--7096, 2022

    2022

  21. [29]

    Learning word vectors for sentiment analysis

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp.\ 142--150, 2011

    2011

  22. [30]

    Fine-grained visual classification of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  23. [31]

    Slip: Self-supervision meets language-image pre-training

    Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European conference on computer vision, pp.\ 529--544. Springer, 2022

    2022

  24. [32]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp.\ 722--729. IEEE, 2008

    2008

  25. [33]

    Deep metric learning via lifted structured feature embedding

    Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4004--4012, 2016

    2016

  26. [34]

    Clip-guided vision-language pre-training for question answering in 3d scenes

    Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2023

  27. [35]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp.\ 3498--3505. IEEE, 2012

    2012

  28. [36]

    Eclipse: A resource-efficient text-to-image prior for image generations

    Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A resource-efficient text-to-image prior for image generations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9069--9078, 2024

    2024

  29. [37]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision...

    2015

  30. [38]

    Revisiting oxford and paris: Large-scale image retrieval benchmarking

    Filip Radenovi \'c , Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ond r ej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5706--5715, 2018

    2018

  31. [39]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision . In International conference on machine learning, p...

    2021

  32. [40]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  33. [41]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 22500-...

    2023

  34. [42]

    Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Representation Learning

    Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Representation Learning . arXiv preprint arXiv:2404.07983, 2024

    2024 arXiv

  35. [43]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Informa...

    2022

  36. [44]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2018

  37. [45]

    Towards Understanding the Modality Gap in CLIP

    Peiyang Shi, Michael C Welle, M rten Bj \"o rkman, and Danica Kragic. Towards Understanding the Modality Gap in CLIP . In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023

    2023

  38. [46]

    Improved deep metric learning with multi-class n-pair loss objective

    Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016

    2016

  39. [47]

    Clip models are few-shot learners: Empirical studies on vqa and visual entailment

    Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint arXiv:2203.07190, 2022

    2022 arXiv

  40. [48]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  41. [49]

    SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

    Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. SuS-X: Training-Free Name-Only Transfer of Vision-Language Models . In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2725--2736, 2023

    2023

  42. [50]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011

    2011

  43. [51]

    Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification

    Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27402--27411, 2024

    2024

  44. [52]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research

  45. [53]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 18123--18133, 2022

    2022

  46. [54]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11975--11986, 2023

    2023

  47. [55]

    Diagnosing and Rectifying Vision Models using Language

    Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and Rectifying Vision Models using Language . In International Conference on Learning Representations (ICLR), 2023

    2023

  48. [56]

    Avid: Any-length video inpainting with diffusion model

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7162--...

    2024

  49. [57]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp.\ 696--712. Springer, 2022 a

    2022

  50. [58]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130 0 (9): 0 2337--2348, 2022 b

    2022

  51. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  52. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  53. [61]

    a photo of

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.