Pith. sign in

REVIEW 4 cited by

Modeling Caption Diversity in Contrastive Vision-Language Pretraining

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.00740 v4 pith:TMBSFGI7 submitted 2024-04-30 cs.CV cs.AIcs.CLcs.LG

Samuel Lavoie , Polina Kirichenko , Mark Ibrahim , Mahmoud Assran , Andrew Gordon Wilson , Aaron Courville , Nicolas Ballas This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords llipimagezero-shotcaptionclippretrainingclassificationcontrastive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an image. In this work, we introduce Llip, Latent Language Image Pretraining, which models the diversity of captions that could match an image. Llip's vision encoder outputs a set of visual features that are mixed into a final representation by conditioning on information derived from the text. We show that Llip outperforms non-contextualized baselines like CLIP and SigLIP on a variety of tasks even with large-scale encoders. Llip improves zero-shot classification by an average of 2.9% zero-shot classification benchmarks with a ViT-G/14 encoder. Specifically, Llip attains a zero-shot top-1 accuracy of 83.5% on ImageNet outperforming a similarly sized CLIP by 1.4%. We also demonstrate improvement on zero-shot retrieval on MS-COCO by 6.0%. We provide a comprehensive analysis of the components introduced by the method and demonstrate that Llip leads to richer visual representations.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Contrastive vision-language learning with paraphrasing and negation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    SemCLIP jointly trains CLIP with paraphrasing and negation losses in a low-dimensional projection, improving original-over-negated retrieval accuracy on CC-Neg from 68.1% to 78.1%.

  2. Phi-Ground Tech Report: Advancing Perception in GUI Grounding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...

  3. Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new large-scale facial emotion caption dataset and a global-local contrastive training framework with positive mining improve zero-shot facial expression recognition.

  4. Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Proxy-FDA aligns the local neighborhood structure of pre-trained and fine-tuned feature spaces, generating synthetic proxies to reduce concept forgetting during fine-tuning.

Pith tools