Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Learning CLIP with one sampled sentence per synthetic caption, plus an asymmetric captioner, sets new retrieval records on MSCOCO and Flickr30K.

desk verdict Solid, well-executed empirical paper on synthetic-caption CLIP training; the headline retrieval numbers are new and the comparisons are controlled, but the single-sentence sampling design is under-tested because the ablation never isolates it against the generative loss. read the letter →

arxiv 2411.16828 v1 pith:VWQO3XCA submitted 2024-11-25 cs.CV

Yanqing Liu , Xianhang Li , Zeyu Wang , Bingchen Zhao , Cihang Xie This is my paper · ORCID
classification cs.CV
keywords CLIPsyntheticcaptionscontrastivepretraininginversescalingeffectautoregressivecaptionerzero-shotretrievalvision-languageLLaVAtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that long, richly detailed synthetic captions should not be fed to a CLIP text encoder in full. Instead, the authors show an inverse effect: with synthetic captions, shortening the text to roughly one randomly sampled sentence consistently improves zero-shot cross-modal retrieval across model sizes, opposite to what happens with web-crawled captions. To keep the full synthetic caption useful, they add an asymmetric autoregressive decoder that takes the image and the short web-crawled caption as input and predicts the full synthetic caption, mimicking the recaptioning process. On this recipe, CLIPS sets state-of-the-art recall@1 scores on MSCOCO (76.4% image-to-text) and Flickr30K (96.6%) with a ViT-H/14 encoder, and swapping its vision encoder into LLaVA improves most multimodal benchmarks. The result matters because it suggests synthetic captions are best exploited by splitting their role: short text for contrastive alignment, full text for generation.

What carries the argument

The load-bearing object is the inverse effect of synthetic captions: with recaptioned text, shorter inputs are consistently better for contrastive learning, so the text encoder is fed one randomly sampled sentence from each synthetic caption alongside the original web caption, with a multi-positive contrastive loss. The second mechanism is an asymmetric captioner: an autoregressive decoder with learnable tokens, image tokens, and web-caption tokens concatenated under a combination mask, trained to predict the full-length synthetic caption. Together they separate the two roles of synthetic captions, short fragments for alignment and full text for generative modeling, so the model exploits the complete caption without forcing the text encoder to process long sequences.

What would settle it

Train the full CLIPS pipeline at ViT-L/14 scale for 10,000 epochs while sweeping input token length (16, 32, 64, 128); if a length other than roughly one sentence yields higher MSCOCO or Flickr30K recall@1, the central inverse-effect claim for large-scale training fails.

Watch

Extended reading notes

Core claim

The central discovery is a strong inverse effect in learning with synthetic captions: reducing token length from 128 to 16 consistently improves CLIP retrieval performance across ViT-S, ViT-B, and ViT-L, with sub-caption and block masks working best, and a single randomly sampled sentence, about 32 tokens, being the strongest encoder input. Building on this, the paper proposes CLIPS, which combines a multi-positive contrastive loss over the original web-crawled caption and the short synthetic sub-caption with an asymmetric autoregressive decoder. Unlike the symmetric CoCa design, the decoder conditions on image features plus web-crawled caption tokens and predicts the full-length synthetic caption, using learnable tokens and a combination mask. This setup yields gains over CLIPA, CoCa, SigLIP, and CLOC at matched model sizes, and the resulting vision encoder transfers to LLaVA, improving eight of nine evaluated benchmarks.

Load-bearing premise

The design assumes that the one-sentence/32-token optimum observed for ViT-S, ViT-B, and ViT-L at 2,000 epochs transfers unchanged to ViT-L/14 and ViT-H/14 trained for 10,000 epochs on the same recaptioned data.

Editorial extensions

If this is right

  • CLIP pretraining with synthetic captions should use one sampled sentence per caption in the contrastive loss rather than the full recaptioned text.
  • The same framework lifts smaller models to the retrieval level of larger baselines: CLIPS-B/16 matches CLIPA and CoCa L/16 on MSCOCO and Flickr30K.
  • Long synthetic captions are more valuable as generation targets than as contrastive inputs, so the information discarded by short-sentence sampling is recovered through the captioner.
  • A CLIPS-trained vision encoder can replace a standard CLIP encoder in an MLLM like LLaVA and improve most evaluated benchmarks, not only retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same inverse effect may apply to other recaptioning pipelines, so caption length could be treated as a tunable hyperparameter rather than a fixed design choice.
  • Editorial inference: the asymmetric decoder's success suggests web captions act as cheap summaries of the richer synthetic text; this could be tested by replacing web captions with machine-generated summaries and measuring whether retrieval degrades.
  • Editorial inference: the framework should combine with caption-diversity methods, since the one-sentence sampler already injects stochasticity; whether diversity and shortness are complementary is a direct extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CLIPS, a CLIP training framework for learning from long synthetic captions. Two design changes are introduced: (1) instead of feeding the full synthetic caption to the text encoder, the authors randomly sample a single sentence from each synthetic caption and use it in a multi-positive contrastive loss alongside the original web-crawled caption; (2) an asymmetric autoregressive decoder predicts the full-length synthetic caption from the image and the web-crawled caption, imitating the recaptioning process. The authors report consistent gains over reproduced CLIPA and CoCa baselines on MSCOCO and Flickr30K zero-shot retrieval, new state-of-the-art numbers at ViT-L/14 and ViT-H/14, and improved LLaVA-1.5 performance when the visual encoder is swapped in. The paper includes ablations of the sub-caption strategy, multi-positive loss, generative loss, fusion type, and token lengths.

Significance. If the reported results hold, the paper makes a useful empirical contribution: it shows that long synthetic captions can be exploited more effectively by shortening the contrastive text input while recovering the omitted information through a generative decoder. The controlled comparison in Table 1, where CLIPA and CoCa are reproduced on the same data mixture, is a genuine strength, as are the component ablations in Table 4. The observed 'inverse effect' for synthetic captions is clearly presented and could inform future CLIP-style pretraining designs. The authors also openly acknowledge weaker ImageNet zero-shot accuracy and provide a plausible explanation. However, the load-bearing claim that single-sentence sampling is beneficial in the full model is not directly tested, and the headline numbers are single runs without released code or checkpoints, so the significance of the specific SOTA gains is currently moderate.

major comments (3)
  1. [Section 4.3, Table 4] The ablation sequence is Baseline -> +SC -> +SC&MP -> +SC&MP&GL, so the marginal benefit of the sub-caption strategy (+1.4% I->T R@1) is measured only against Plain CLIP, without the multi-positive loss or the generative decoder. Since the generative loss already supervises the full-length synthetic caption, it is possible that the text encoder no longer suffers from long contrastive captions once the decoder is present; in that case, the single-sentence sampling would not be load-bearing for the final model. Please add an ablation that applies the multi-positive loss and generative loss with full-length synthetic captions in the contrastive branch, i.e., compare Baseline+MP+GL versus Baseline+SC+MP+GL. This is necessary to support the claim in Section 3.3 that feeding only a single sampled sentence is one of the two key designs.
  2. [Section 3.2 and Section 4.1] The inverse effect shown in Figure 3 is measured on ViT-S/16, ViT-B/16, and ViT-L/16 trained for 2,000 ImageNet-equivalent epochs at 112x112 resolution, with token lengths of 128, 64, 32, and 16. The headline SOTA experiments in Table 2 use ViT-L/14 and ViT-H/14 trained for 10,000 epochs at 84x84 resolution, with the text input padded to length 80. The paper does not verify that a single randomly sampled sentence remains optimal at this larger scale, higher resolution schedule, or different model size. If the optimal fragment length depends on any of these factors, the main results could be specific to this recipe rather than reflecting a general principle. Please either provide a small-scale or mid-scale sweep of the number of sampled sentences under the final training setup, or explicitly state this transfer as a limitation and temper the claim that the single-sentence design is a robust principle.
  3. [Section 4.2, Tables 2 and 3] All headline results are reported as single runs without error bars, multiple seeds, or significance tests, and code/checkpoints are not released on the project page. Given that several SOTA differences are small (e.g., CLIPS-H/14 versus CLOC-H/14 on MSCOCO I->T R@1 differs by 0.7 points, and several LLaVA benchmark differences are a few points), the claim of new state-of-the-art results would be materially strengthened by either reporting variance over at least two or three runs or releasing the trained checkpoints so that the results can be independently verified. This is a standard expectation for empirical SOTA claims in the current literature.
minor comments (5)
  1. [Section 3.3 and Footnote 2] The text says 'all ViTs generally achieve the strongest performance at an input token length of 32', but Footnote 2 and Figure 4a show that padding the single sampled sentence to 80 gives an additional ~1% improvement. Please clarify whether 32 refers to the effective number of real tokens before padding or the target token length, and reconcile the statement with the 80-token choice used in the main experiments.
  2. [Section 3.2, Eq. (4)] The formal definition of Subcaption(S, L) is ambiguous: it shows only one concatenation step, but the prose describes an iterative process of selecting sub-captions and checking the length again. Please rewrite the equation or pseudocode to make the iterative selection explicit, including the handling of ties when |S_i| = L.
  3. [Section 4.3, Table 7] The header of Table 7 is difficult to read: the checkmarks under 'Causal mask', 'L-tokens', and 'Content' are not self-explanatory, and the rows labeled 'first' and 'random' need a short explanation in the caption. Please add a caption and, if possible, row labels that describe each configuration in words.
  4. [Section 4.1] Table 4 uses a 'mixed ratio of 0.6' baseline, while Table 1 and Section 4.2 use an 80% web-captions / 20% synthetic-captions mixture. Please state explicitly how the mixed ratio in the ablation relates to the main setup, since the marginal contributions in Table 4 may not carry over exactly to the 80/20 setting.
  5. [Section 4.3, 'Generated sequence length'] The sentence 'Due to the short length of the original captions, we rely entirely on synthetic captions to train CoCa' appears to contradict the previous paragraph, which says web-crawled captions were maintained for contrastive learning. Please clarify which inputs are used for the contrastive branch and which for the generative branch in these experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity — the two design changes are empirically ablated and benchmarked externally; self-citations supply data and training recipes but do not force the conclusions.

full rationale

The paper's central claims are the single-sentence sub-caption input to the text encoder and the asymmetric autoregressive decoder that predicts full synthetic captions. Neither is derived from its own output by construction. The inverse effect shown in Sec. 3.2 is a new experiment (Figure 3) on ViT-S/B/L with four token-reduction strategies, not an imported conclusion from the authors' prior inverse-scaling paper [31]; the paper explicitly notes a different 'tipping point' when moving from web-crawled to synthetic captions. The ablation in Table 4 independently isolates the sub-caption contribution (+1.4% I-to-T), the multi-positive loss, and the generative loss, and the final SOTA numbers are evaluated on external benchmarks (MSCOCO, Flickr30K, ImageNet, LLaVA tasks) against CLIP, CoCa, SigLIP, CLOC, and OpenAI-CLIP baselines. The references to Recap-DataComp-1B [30] and the efficient training recipe [31] are self-citations, but they are used as infrastructure and training protocol, not as evidence for the specific performance gains; the dataset and recipe are shared across all compared methods, so they do not by themselves produce CLIPS's relative improvements. No equation in the manuscript reduces a predicted quantity to a fitted input, no fitted parameter is relabeled as a prediction, and no uniqueness theorem from the authors is invoked to forbid alternatives. The closest concern is that the 32-token/single-sentence choice was selected using MSCOCO retrieval in Figure 3 and then reported on MSCOCO; that is a benchmark-selection and generalization risk, not a circularity reduction, and therefore does not affect the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The framework relies on standard contrastive and generative losses; the main hand-set knobs are loss weights, token lengths, and data mixture. The key unproven premise is that the small-scale inverse-effect observation transfers to the largest models and longest training runs.

free parameters (5)
  • contrastive loss weight alpha = 1
    Set by hand in Eq. 11; no sensitivity analysis is reported for the trade-off between contrastive and generative loss.
  • generative loss weight beta = 2
    Set by hand in Eq. 11; no sensitivity analysis is reported for the trade-off between contrastive and generative loss.
  • text encoder input token length = 80
    Footnote 2: padding a single sampled sub-caption to 80 tokens gives about 1% extra retrieval gain over 32; the optimal value is data-dependent.
  • decoder output token length and number of learnable tokens = 128
    Set to match full synthetic caption length; Figure 4b shows longer generated sequences help, but 128 is a chosen cap.
  • web-to-synthetic caption mixture ratio = 80%/20%
    Follows the Recap-DataComp-1B recipe [30] for main experiments; the ablation baseline uses a 0.6 synthetic ratio, so mixture ratio affects comparisons.
assumptions (4)
  • standard math InfoNCE contrastive loss (Eq. 1) and autoregressive next-token loss (Eq. 10) are valid optimization objectives for representation learning.
    Used as background from CLIP and CoCa; not derived in this paper.
  • domain assumption Web captions and synthetic sub-captions are both positive descriptions of the same image.
    The multi-positive loss in Eq. 5 assumes one-to-many matching; if a sampled synthetic sentence describes hallucinated content, it injects a false positive.
  • ad hoc to paper The inverse effect observed on ViT-S/B/L at 2,000 epochs transfers to ViT-L/14 and ViT-H/14 at 10,000 epochs.
    Single-sentence sampling is chosen from Figure 3 and applied to the SOTA runs without a dedicated large-scale sweep.
  • domain assumption Full-length synthetic captions are good generative targets and modeling them as an auxiliary loss helps rather than hurts.
    Table 5 shows synthetic targets beat web targets, but the result is empirical and only tested on Recap-DataComp-1B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions." pith.science (2026). https://pith.science/paper/VWQO3XCA

@misc{pith2026241116828,
  author       = {Pith},
  title        = {Pith review of: CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWQO3XCA}},
  note         = {Machine review of arXiv:2411.16828}
}
read the original abstract

Previous works show that noisy, web-crawled image-text pairs may limit vision-language pretraining like CLIP and propose learning with synthetic captions as a promising alternative. Our work continues this effort, introducing two simple yet effective designs to better leverage richly described synthetic captions. Firstly, by observing a strong inverse effect in learning with synthetic captions -- the short synthetic captions can generally lead to MUCH higher performance than full-length ones -- we therefore fed only partial synthetic captions to the text encoder. Secondly, we incorporate an autoregressive captioner to mimic the recaptioning process -- by conditioning on the paired image input and web-crawled text description, the captioner learns to predict the full-length synthetic caption generated by advanced MLLMs. Experiments show that our framework significantly improves zero-shot performance in cross-modal retrieval tasks, setting new SOTA results on MSCOCO and Flickr30K. Moreover, such trained vision encoders can enhance the visual capability of LLaVA, showing strong improvements on a range of MLLM benchmarks. Our project page is https://ucsc-vlaa.github.io/CLIPS/.

Figures

Figures reproduced from arXiv: 2411.16828 by the authors.

Figure 1
Figure 1. The pipeline of our proposed CLIPS. We introduce two simple yet effective designs—1) only a subpart of the synthetic caption is used in contrastive learning and 2) a captioner to pre￾dict the full synthetic caption based on the web-crawled caption and the image—to better leverage synthetic captions. Our method registers new SOTA results on MSCOCO, achieving 76.4% in text retrieval and 57.2% in image retrieval. subst… view at source ↗
Figure 2
Figure 2. Visualization of four different token reduction strategies. These strategies can improve the model’s learning efficiency on [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The inverse scaling effect of synthetic captions. Unlike the performance drop from reducing token length in original captions, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study on input and output token lengths. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mining Contextualized Visual Associations from Images for Creativity Understanding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A scalable pipeline generates 1.7 million increasingly abstract MSCOCO captions, and fine-tuning CLIP on them improves zero-shot retrieval in poetry and metaphor tasks.

  2. MobileCLIP2: Improving Multi-Modal Reinforced Training

    cs.CV 2025-08 conditional novelty 5.0 of 10

    MobileCLIP2 combines DFN-trained teachers, a fine-tuned CoCa captioner, and new 5-stage FastViT variants to set state-of-the-art ImageNet-1k zero-shot accuracy at low latency.

  3. OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    OpenVision is a fully open, from-scratch family of CLIP-style vision encoders that matches or surpasses OpenAI's CLIP as the visual backbone of LLaVA-style multimodal models.

  4. OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

    cs.CV 2025-09 conditional novelty 4.0 of 10

    OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

Reference graph

Works this paper leans on

55 extracted references · 19 canonical work pages · cited by 4 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    nocaps: novel object caption- ing at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. nocaps: novel object caption- ing at scale. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 8948–8957, 2019. 7

    work page 2019

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  4. [4]

    Contrastive localized language- image pre-training

    Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive localized language- image pre-training. arXiv preprint arXiv:2410.02746, 2024. 2, 6, 7

    arXiv 2024

  5. [5]

    Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- \sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. 2

    arXiv 2024

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2

    arXiv 2023

  7. [7]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 7

    arXiv 2022

  8. [8]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer,

Show all 55 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  2. [10]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024. 2

    2024 arXiv

  3. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 7

    2009

  4. [12]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  6. [14]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 36, 2024. 1, 2

    2024

  7. [15]

    Data fil- tering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- tering networks. arXiv preprint arXiv:2309.17425, 2023. 7

    2023 arXiv

  8. [16]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 7

    2023 arXiv

  9. [17]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Sys...

    2024

  10. [18]

    Synthclip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.01832, 2024

    Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arXiv preprint arXiv:2402.01832, 2024. 2

    2024 arXiv

  11. [19]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 7

    2019

  12. [20]

    Open- clip

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip. github, 2021. 7

    2021

  13. [21]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  14. [22]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International conference on machine learning, pages 5583–5594. PMLR, 2021. 2

    2021

  15. [23]

    Veclip: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In European Confer- ence on Computer Vision, pages 111–127. Springer, ...

    2025

  16. [24]

    Modeling caption diversity in contrastive vision-language pretraining

    Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mah- moud Assran, Andrew Gordon Wildon, Aaron Courville, 10 and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint arXiv:2405.00740, 2024. 2

    2024 arXiv

  17. [25]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 2

    2021

  18. [26]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 1, 2

    2022

  19. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  20. [28]

    Visualbert: A simple and perfor- mant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 2

    1908 arXiv

  21. [29]

    Clipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within a $10,000 budget; an extra $4,000 unlocks 81.8% accuracy

    Xianhang Li, Zeyu Wang, and Cihang Xie. Clipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within a $10,000 budget; an extra $4,000 unlocks 81.8% accuracy. arXiv preprint arXiv:2306.15658, 2023. 2

    2023 arXiv

  22. [30]

    What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    arXiv

  23. [31]

    An inverse scal- ing law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scal- ing law for clip training. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3, 4, 6

    2024

  24. [32]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. 7

    2023

  25. [33]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 2

    2024 arXiv

  26. [34]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  27. [35]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 7

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  29. [37]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. arXiv preprint arXiv:2311.18765, 2023. 2, 4

    2023 arXiv

  30. [38]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019. 2

    2019

  31. [39]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning. In Find- ings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. ...

    2022

  32. [40]

    Introducing meta llama 3: The most capable openly available llm to date

    AI Meta. Introducing meta llama 3: The most capable openly available llm to date. Meta AI, 2024. 2

    2024

  33. [41]

    Improving multimodal datasets with image captioning

    Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Se- woong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1, 2

    2024

  34. [42]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 3

    2018 arXiv

  35. [43]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt, 2022. 2

    2022

  36. [44]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  39. [47]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 7

    2019

  40. [48]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learn- i...

    2022

  41. [49]

    Mofi: Learning image represen- tations from noisy entity annotated images

    Wentao Wu, Aleksei Timofeev, Chen Chen, Bowen Zhang, Kun Duan, Shuangning Liu, Yantao Zheng, Jonathon Shlens, 11 Xianzhi Du, Zhe Gan, et al. Mofi: Learning image represen- tations from noisy entity annotated images. arXiv preprint arXiv:2306.07952, 2023. 7

    2023 arXiv

  42. [50]

    Alip: Adaptive language-image pre-training with synthetic cap- tion

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic cap- tion. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 2922–2931, 2023. 2

    2023

  43. [51]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 2

    2021 arXiv

  44. [52]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 1, 2, 5, 6, 7

    2022 arXiv

  45. [53]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  46. [54]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2, 6, 7

    2023

  47. [55]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In European Confer- ence on Computer Vision, pages 73–90. Springer, 2025. 1, 2, 3, 4, 5 12

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.