Pith. sign in

REVIEW 3 major objections 4 minor 40 references

Learning Visual Composition through Improved Semantic Guidance

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A standard CLIP model, trained with recaptioned images and a pretrained language-model text tower under a single contrastive loss, achieves state-of-the-art scores on compositional benchmarks and detailed image retrieval, outperforming…

desk verdict The DOCCI gain is real; the ARO gain is mostly hard-negative fine-tuning, and the paper knows it. read the letter →

arxiv 2412.15396 v2 pith:CVVYJATE submitted 2024-12-19 cs.CV cs.AIcs.CLcs.IR

Austin Stone , Hagen Soltau , Robert Geirhos , Xi Yi , Ye Xia , Bingyi Cao , Kaifeng Chen , Abhijit Ogale
show 1 more author
Jonathon Shlens
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.IR
keywords visualcompositioncontrastivelearningCLIPrecaptioningimageretrievalsemanticguidancetextencodercompositionalbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that contrastive vision-language models behave like bag-of-words reasoners not because of their architecture but because their training text is too weak, and that fixing the text is sufficient to fix the model. It shows that recaptioning 1B web images with a grounded multimodal foundation model and swapping in a pretrained language model as the text tower lifts a standard CLIP model from 59%/63% to 92%/94% on ARO relations/attributes and from 58.4% to 94.5% recall@1 on DOCCI retrieval, surpassing bespoke architectures with a single contrastive loss and no architectural changes. If this is right, the bottleneck for compositional understanding is data quality, and scaling caption quality is a cheaper and more scalable path than designing specialized losses or cross-modal encoders. The paper also argues that COCO caption retrieval is too noisy to measure such progress: human annotation of its apparent failures shows 70.2% are actually acceptable matches.

What carries the argument

The load-bearing mechanism is the pairing of a grounded recaptioning pipeline with a mostly-frozen pretrained language-model text tower. Recaptioning replaces an average 7-word alt-text caption with a 57-word description generated by Gemini 1.5 Flash from the image plus the original alt-text and page title, with a prompt that instructs the model not to invent details; this turns a bag-of-words training target into one that names objects, attributes, relations, and spatial layout in natural language. The text tower is a pretrained Gemini 1.5 Flash-8B or Gemma-2B model whose last four layers are fine-tuned (with bidirectional attention) while the rest stay frozen, giving the contrastive objective a rich compositional language space to align image patches to. A supplementary mechanism is a set of 64M synthetic hard-negative captions generated in the style of the ARO benchmark, used for a short fine-tuning stage.

What would settle it

Train the identical recipe but swap one attribute or relation in half of the recaptioned captions so the caption no longer describes the image (human-verified), then measure ARO relations and attributes accuracy: if accuracy stays near 90%, the model is exploiting language statistics rather than grounded composition, while a sharp drop would confirm the model is reading image content.

Watch

Extended reading notes

Core claim

The central claim is that enriching the target semantic embedding is sufficient to make a standard contrastive learning model capture visual composition. Two minimal changes accomplish this: first, replacing noisy web alt-text with long, grounded captions generated by Gemini 1.5 Flash using the image, the alt-text, and the page title as grounding, with a prompt instructing the model not to invent details; second, using a pretrained Gemini 1.5 Flash-8B (or Gemma-2B) language model as the text tower, fine-tuning only the last four layers and switching to bidirectional attention. With these changes alone, a ViT-Base CLIP model trained with a single contrastive loss surpasses bespoke architectures on ARO and SugarCrepe and achieves 94.5% recall@1 on DOCCI-test retrieval without fine-tuning on DOCCI. Ablations on 100M images show the two changes are additive, that sentence sampling helps, and that hard-negative fine-tuning trades off DOCCI for ARO.

Load-bearing premise

The recaptioned training captions are factually accurate and grounded enough in the image that the contrastive objective learns true image-text correspondence rather than patterns in the generated language.

Editorial extensions

If this is right

  • The gains transfer across text encoders: both Gemini 1.5 Flash-8B and Gemma-2B deliver large DOCCI improvements over a from-scratch text encoder, so the effect is not specific to one language model.
  • COCO is a saturated retrieval benchmark: human raters judged 70.2% of the model's apparent failures to be acceptable matches, so detailed-caption benchmarks such as DOCCI are needed to measure compositional gains.
  • Adding a 50% JFT mixture raises zero-shot ImageNet accuracy from 68.4% to 79.1%, indicating that the lower ImageNet score reflects the training distribution rather than a failure of fine-grained discrimination.
  • Hard-negative fine-tuning lifts ARO relations from 65% to 93% and attributes from 82% to 94% while slightly hurting DOCCI, showing a trade-off between compositional discrimination and detailed retrieval.
  • The improved alignment transfers to a practical application: the model predicts human preference for text-to-image generation better than an open-source CLIP baseline (81.1% vs 71.3% weighted accuracy).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If caption quality is the main lever, the same recaptioning recipe should also improve other contrastive training families and filtering pipelines; the paper does not test this, but it is a direct consequence of the claim that the target text is the bottleneck.
  • The frozen LLM text tower itself encodes language plausibility, so part of the ARO/SugarCrepe gains may come from a text-side prior rather than from image grounding; the paper's ablations do not fully separate these two sources.
  • The 70.2% mislabeled-failure finding suggests that other models' COCO retrieval numbers would also rise under the same human re-scoring protocol, so some of the apparent gap between this model and prior work on COCO may be benchmark noise.
  • Because the model was trained at 256x256 without DOCCI fine-tuning while a compared method gains over 10 points from 448x448 fine-tuning, combining the recaptioned data with higher resolution and fine-tuning is an obvious deferred experiment with likely headroom.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes two simple, scalable modifications to a standard two-tower CLIP model: (1) recaption 1B WebLI images with Gemini 1.5 Flash using grounded prompts that incorporate alt-text and page title, and (2) replace the randomly initialized text tower with a pretrained LLM (Gemini Flash-8B or Gemma-2B), fine-tuning only the last four layers. The model is trained with a single contrastive loss, plus two data augmentations: random sentence sampling and 64M synthetic hard negatives generated to mimic ARO-style perturbations. The authors report state-of-the-art or highly competitive results on ARO, SugarCrepe, COCO, and DOCCI retrieval, with the largest gains on DOCCI (94.5% recall@1 on test). A series of ablations (Tables 5-8) isolates the contributions of recaptioning, text-encoder pretraining, sentence sampling, and hard-negative fine-tuning.

Significance. If the claims hold, this is an important demonstration that supervision quality can substitute for architectural complexity in compositional vision-language learning. The paper's strengths include the clean ablation structure, the use of DOCCI as a detailed retrieval benchmark, the human-annotation analysis showing that many COCO retrieval failures are actually acceptable matches, and the open Gemma-2B text-encoder variant that partially mitigates reliance on proprietary components. The central ARO claim, however, is entangled with a hard-negative fine-tuning stage that is explicitly crafted to match ARO's perturbation taxonomy, so the significance is conditional: the recaptioning and strong-text-encoder contributions are convincingly demonstrated on DOCCI (Table 5), but the broad claim of surpassing bespoke architectures on compositional benchmarks needs a cleaner separation of the hard-negative effect.

major comments (3)
  1. [Section 3.5, Table 8] The hard-negative fine-tuning stage is the dominant contributor to the ARO results in Table 1. Table 8 shows that adding hard negatives raises ARO relations from 65% to 93% and attributes from 82% to 94%, while slightly lowering DOCCI-full retrieval from 91.6 to 88.1. Since Figure 3 states that the synthetic negatives were produced with few-shot examples 'matching the style of ARO relations and attributes evaluation,' the Table 1 ARO numbers are partly trained to the evaluation distribution rather than being a clean demonstration of the proposed semantic guidance. The paper should either report the hard-negative ablation on a held-out composition benchmark whose perturbation style was not used to construct the negatives (SugarCrepe is a natural candidate and is not reported in Table 8), or explicitly limit the ARO claim.
  2. [Section 3.5, Tables 5-8] The paper's central claim is that two changes, recaptioning and a pretrained text encoder, suffice to close the compositional gap, but the final model also includes 64M synthetic hard negatives, and the ablations show that this third ingredient is required for the ARO headline. The abstract and introduction ('simple approach boosts performance of CLIP substantially and surpasses all bespoke architectures') should be aligned with this decomposition. In addition, the hard-negative ablation in Table 8 is reported only on DOCCI-full, COCO, and ARO; no DOCCI-test numbers are given for the hard-negative stage, so it is unclear whether the strong DOCCI-test result in Table 3 is affected by the same trade-off. Please report a complete configuration-to-benchmark matrix for the main results.
  3. [Section 2.1, Appendix A] The recaptioning step is motivated by reducing hallucination, but the paper provides no quantitative evidence that the generated captions are factually grounded in the images. The only indirect evidence is Table 6, where removing alt-text grounding costs 7.7 points on COCO (39.2 vs 31.5) but only 1.0 point on DOCCI-full (90.3 vs 89.3). Because the large DOCCI gains are the strongest support for the recaptioning claim, a sample-based verification of caption-image factuality (e.g., human ratings or an object-consistency check) would materially strengthen the paper. Without it, one cannot exclude the possibility that the visual encoder learned spurious correlations that happen to match DOCCI-style captions.
minor comments (4)
  1. [Table 6 caption] The caption states 'Default and short caption length contain 133.4 and 354.8 words, respectively,' which appears to have the two values reversed or mislabeled, since the 'short' condition is intended to be more concise and Figure 4 reports a mean of 57 words for the default captions; please clarify.
  2. [All tables] All results are reported from single training runs without seeds, confidence intervals, or error bars; given the large claimed differences (e.g., 65% to 93% in Table 8), reporting variance across a few seeds would help assess reliability.
  3. [Section 2] The paper relies on internal 1B WebLI data, Gemini 1.5 Flash recaptioning, and the Gemini Flash-8B text encoder, none of which are publicly available. The open Gemma-2B variant only partially mitigates this. Please state explicitly what artifacts (captions, checkpoints, or code) can be released to support reproducibility.
  4. [Section 2.1] There is a typo in the reference to Figure 2: 'a strong multimodal foundation model (Figure 2.' is missing a closing parenthesis, and the sentence should end properly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central recaptioning and strong-text-encoder claims are validated on external benchmarks, and the ARO hard-negative stage is transparent data augmentation rather than a derivation from its own inputs.

full rationale

The paper's central claim—that recaptioning web-scale alt-text with a grounded multimodal model and using a pretrained LLM text tower improves compositional understanding in a standard two-tower CLIP model—is supported by external benchmark evaluations (DOCCI, COCO, SugarCrepe, ImageNet) and by ablations that vary captions while holding images fixed. None of these results is equivalent to the method's inputs by construction. The most plausible circularity concern is the hard-negative fine-tuning stage: Figure 3 states that synthetic negatives are generated with few-shot examples 'matching the style of ARO relations and attributes evaluation,' and Table 8 shows this stage raises ARO relations from 65% to 93% while slightly harming DOCCI. This is a legitimate benchmark-specificity and generalization concern, but it is not circular in the sense required here: the model is not trained on ARO test data, the ARO numbers are not algebraically forced by the synthetic data distribution, and the paper is explicit that hard-negative fine-tuning is a separate data augmentation whose removal significantly harms ARO. The contribution of recaptioning and the strong text encoder is independently visible on DOCCI (Table 5: 53.5 to 91.6 without hard-negative fine-tuning), so the central derivation does not reduce to the ARO-style augmentation. Self-citations to Gemini models, WebLI, and ALIGN are tool/data citations rather than load-bearing uniqueness theorems or unverified premises; the method's effectiveness is measured against external benchmarks rather than asserted from those citations. The paper also candidly flags limitations of ARO (language-only baselines solve 71%/82%, Section 3.1) and of COCO (70.2% of scored failures are human-acceptable matches, Section 3.2), which further indicates that the headline results are not being presented as definitionally guaranteed. Overall, no circular step meets the evidentiary bar of exhibiting a specific reduction of a prediction to its fitted inputs or to a self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method introduces no new theoretical entities, but it relies on proprietary data and models and on benchmark-specific data augmentation. The counted free parameters are training choices, not fitted model parameters.

free parameters (5)
  • Number of text encoder layers unfrozen = 4 (last 4 layers)
    Chosen via experiments as best trade-off between compute and performance (Section 2.2).
  • Hard negative example count = 2M in Section 2.1; 64M in Section 3.1 and Figure 3 caption
    The paper reports inconsistent counts for the hard-negative augmentation, which affects the claimed recipe's reproducibility.
  • Sentence sampling range = 1 to 10 sentences
    Randomly sample between one and ten sentences from long captions as targets (Section 2.1, Table 7).
  • Caption length (ablation default) = 133.4 words (default) vs 354.8 words (short) per Table 6 note; main 1B set mean 57 words
    Caption length is a design choice affecting retrieval; the paper reports inconsistent word counts across sections.
  • Training steps and batch sizes = 150k steps batch 65,536; 500 steps batch 4,096
    Training schedule fixed; no sweep reported.
assumptions (5)
  • domain assumption Contrastive learning is a sufficient training signal for visual composition when the semantic target is rich.
    Section 1 states this as a posit: 'we posit that contrastive learning is a sufficient training signal for multimodal learning so long as the target semantic embedding is rich enough.'
  • domain assumption ViT-Base has enough capacity to capture visual composition.
    Section 1: 'we posit that underlying visual architecture (e.g. ViT) contains sufficient parameters and scale to capture visual composition.'
  • ad hoc to paper Gemini 1.5 Flash recaptions are factually reliable and grounded.
    The paper uses prompt text to minimize hallucination but does not quantify caption-image alignment (Section 2.1, Appendix A).
  • domain assumption ARO, SugarCrepe, and DOCCI retrieval are valid measures of compositional understanding.
    The benchmarks are used as ground truth for evaluation without questioning their construct validity beyond noting ARO's language-only baseline.
  • ad hoc to paper Training on synthetic hard negatives that mimic ARO perturbation style is a valid augmentation rather than benchmark overfitting.
    Figure 3 says negatives are generated 'matching the style of ARO relations and attributes evaluation'; Table 8 shows these are essential for ARO performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Visual Composition through Improved Semantic Guidance." pith.science (2026). https://pith.science/paper/CVVYJATE

@misc{pith2026241215396,
  author       = {Pith},
  title        = {Pith review of: Learning Visual Composition through Improved Semantic Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVVYJATE}},
  note         = {Machine review of arXiv:2412.15396}
}
read the original abstract

Visual imagery does not consist of solitary objects, but instead reflects the composition of a multitude of fluid concepts. While there have been great advances in visual representation learning, such advances have focused on building better representations for a small number of discrete objects bereft of an understanding of how these objects are interacting. One can observe this limitation in representations learned through captions or contrastive learning -- where the learned model treats an image essentially as a bag of words. Several works have attempted to address this limitation through the development of bespoke learned architectures to directly address the shortcomings in compositional learning. In this work, we focus on simple, and scalable approaches. In particular, we demonstrate that by substantially improving weakly labeled data, i.e. captions, we can vastly improve the performance of standard contrastive learning approaches. Previous CLIP models achieved near chance rate on challenging tasks probing compositional learning. However, our simple approach boosts performance of CLIP substantially and surpasses all bespoke architectures. Furthermore, we showcase our results on a relatively new captioning benchmark derived from DOCCI. We demonstrate through a series of ablations that a standard CLIP model trained with enhanced data may demonstrate impressive performance on image retrieval tasks.

Figures

Figures reproduced from arXiv: 2412.15396 by the authors.

Figure 1
Figure 1. Summary of results. Left: Previous state-of-the-art results in multimodal embeddings have limited understanding of the composition of images [14, 26]. Right: The goal of this work is to learn multimodal embeddings which reflect a strong under￾standing of the composition of visual and semantic information. Images and captions are red and blue, respectively. is that the visual representations are not rich enough to ca… view at source ↗
Figure 2
Figure 2. Summary of methodology. All images were recaptioned using a multimodal foundation model grounded on the image and the alt-text for the image on the web page. In this case, we show captions generated using Gemini 1.5 Flash [30]. We highlight aspects of the new caption in blue that leverage the alt-text or demonstrate a capability. Note how the generated captions leverage information provided by the alt-text or perfor… view at source ↗
Figure 4
Figure 4. Recaptioning increases caption length by 8X. While the median alt text caption length is just 7 words, the detailed cap￾tions generated by Gemini Flash 1.5 increase this to 57 words. ing steps with reduced batch size of 4096 using data aug￾mentation on the “hard negative” examples (Section 2.1). The main model training is done on 256 accelerators and the fine-tuning is done with 16 accelerators. We use Adam optimize… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Recaptioning improves caption log-likelihood. Alt text on the web is often unnatural (example: “bigtimerush nyc 007”), leading to low log-likelihood with a median of -223. In contrast, the captions from Gemini Flash 1.5 substantially im￾prove median log-likelihood to -…
Figure 6
Figure 6. Figure 6: Qualitative examples of limitations of image retrieval COCO captions. Three examples showcasing where the model recalled image was incorrect, but is well described by the caption. Top and Middle Rows: Caption and associated ground truth image. Bottom Row: Top recalled …
Figure 7
Figure 7. Figure 7: Caption length comparison for ablation captions. Concise captions are significantly shorter than our default cap￾tions. Using Gemini-8B and removing alt-text conditioning have little impact on the length [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages

  1. [1]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. 8

    work page 2016

  2. [2]

    TIPS: Text-image pretraining with spatial awareness

    Anonymous. TIPS: Text-image pretraining with spatial awareness. In Submitted to The Thirteenth International Conference on Learning Representations , 2024. under re- view. 5, 8, 12

    work page 2024

  3. [3]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR,

  4. [4]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 2, 7

    arXiv 2022

  5. [5]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Pi- otr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschan- nen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjo- erd van Steenkiste, Gamal...

    work page 2023

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009. 6

    work page 2009

  7. [7]

    Virtex: Learning visual representations from textual annotations, 2021

    Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations, 2021. 8

    work page 2021

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. 1, 2

    2021

Show all 40 references
  1. [9]

    Improving clip training with language rewrites, 2023

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites, 2023. 8

    2023

  2. [10]

    Datacomp: In search of the next generation of multimodal datasets, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Or- gad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussm...

    2023

  3. [11]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In Thirty- Seventh Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2023. 3, 4

    2023

  4. [12]

    Mate: Meet at the embedding – connect- ing images with long texts, 2024

    Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. Mate: Meet at the embedding – connect- ing images with long texts, 2024. 5, 8, 12

    2024

  5. [13]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Le...

    2021

  6. [14]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. 1, 2

    2021

  7. [15]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. 3

    2014 arXiv

  8. [16]

    Veclip: Improving clip training via visual-enriched captions,

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Improving clip training via visual-enriched captions,

  9. [17]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. 1, 4, 5, 8, 12

    2022

  10. [18]

    On er- roneous agreements of CLIP image embeddings

    Siting Li, Pang Wei Koh, and Simon Shaolei Du. On er- roneous agreements of CLIP image embeddings. arXiv preprint arXiv:2411.05195, 2024. 8

    2024 arXiv

  11. [19]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312,

    arXiv

  12. [20]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general-purpose plausibility estimation model for common- sense statements, 2023. 4

    2023

  13. [21]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. 8

    2022

  14. [22]

    DOCCI: Descriptions of Connected and Con- trasting Images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Ja- son Baldridge. DOCCI: Descriptions of Connected and Con- trasting Images. In arXiv:2404.19753, 2024. 1, 2, 5, 7, 8

    2024 arXiv

  15. [23]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, J...

    2024

  16. [24]

    TripletCLIP: Improving compositional reasoning of CLIP via synthetic vision-language negatives

    Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and Yezhou Yang. TripletCLIP: Improving compositional reasoning of CLIP via synthetic vision-language negatives. arXiv preprint arXiv:2411.02545, 2024. 8

    2024 arXiv

  17. [25]

    Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V . Le. Combined scaling for zero-shot transfer learn- ing, 2023. 6

    2023

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  19. [27]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  20. [28]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  21. [29]

    Laion-5b: An open large-scale dataset for training next generation image-text models, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Lai...

    2022

  22. [30]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024. 2, 3, 4, 7, 12

    2024

  23. [31]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. 2, 3, 7

    2024

  24. [32]

    Image captioners are scalable vision learners too, 2023

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiao- hua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too, 2023. 4, 8

    2023

  25. [33]

    Show and tell: Lessons learned from the 2015 mscoco image captioning challenge

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence , 39(4):652–663,

    2015

  26. [34]

    Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V . Le. Self-training with noisy student improves imagenet clas- sification, 2020. 8

    2020

  27. [35]

    CoCa: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 2, 4, 5, 6, 7, 8, 12

    2022

  28. [36]

    When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023. 1, 2, 3, 4, 8, 13

    2023

  29. [37]

    Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts, 2022

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts, 2022. 1, 4, 5, 8, 12

    2022

  30. [38]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12104–12113, 2022. 6, 7

    2022

  31. [39]

    Lit: Zero-shot transfer with locked-image text tuning, 2022

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning, 2022. 6

    2022

  32. [40]

    Long-clip: Unlocking the long-text capability of clip, 2024

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip, 2024. 5, 8, 12 Appendix A. Prompts used for recaptioning For gathering our main set of 1B image captions, we used the following prompt. “The image present...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.