Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen LLM can replace CLIP's text encoder and beat it with far less data.

desk verdict A genuinely useful frozen-LLM-as-text-tower recipe for CLIP-style training, with one real confound: no same-capacity trainable text encoder control, so the central attribution is not yet secure. read the letter →

arxiv 2411.11927 v3 pith:BWSDD7SI submitted 2024-11-18 cs.CV

Anjia Cao , Xing Wei , Zhiheng Ma This is my paper · ORCID
classification cs.CV
keywords language-imagepre-trainingfrozenlargelanguagemodelstextencodercontrastivelearninglongcaptionsmultilingualretrievalpromptdistillationdataefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard CLIP text encoder, limited to 77 tokens and weak multilingual coverage, is the real bottleneck in data-efficient language-image pre-training, and that a frozen large language model removes that bottleneck. FLAME prompts a frozen LLM with a small set of single-word 'facet' questions -- main object, attribute, action, scene atmosphere, color -- and aligns each resulting embedding against the image with a symmetric contrastive loss. Trained on just CC3M, it beats the previous state of the art by 4.9 points in ImageNet top-1 accuracy; trained on YFCC15M, it outperforms a CLIP trained on WIT-400M in long-context retrieval and in zero-shot retrieval across all 36 Crossmodal-3600 languages. If correct, this means long captions can be used directly as training labels, without chunking, caption synthesis, or positional-encoding tricks, while keeping training cost near CLIP's through offline embedding.

What carries the argument

The load-bearing mechanism is multifaceted prompt distillation over a frozen decoder-only LLM. A set of seven prompts shares one prefix containing the caption, then forces a single-word response for a distinct facet (entity category, entity attribute, interaction, scene context, atmosphere, color); the final-token hidden state for each prompt becomes a text embedding. Facet-decoupled attention packs all prompts into one sequence with a mask that blocks attention across facets, giving a 3.6x speedup over separate forward passes. Offline caching of these embeddings moves the LLM out of the training loop entirely, so the trainable part is only the visual encoder plus an MLP projection.

What would settle it

Train a CLIP-style model with a trainable text transformer of comparable capacity to Mistral-Nemo on the same CC3M and YFCC15M caption sets used for FLAME, with the same multi-facet prompt decomposition. If its ImageNet top-1, long-context recall, and Crossmodal-3600 results match or exceed FLAME's, the claim that frozen LLMs plus prompt distillation drive the gains is falsified, because the improvement would then be attributable to text-encoder scale rather than to the framework.

Watch

Extended reading notes

Core claim

The central claim, stated on the authors' terms, is that 'frozen LLMs can effectively enhance language-image pre-training through rich semantic representations,' overturning the conventional wisdom that frozen text encoders are suboptimal. FLAME feeds the full long caption to the LLM and appends K prompts, each asking for a single-word answer about one semantic facet; the hidden state at each prompt's final token is used as a separate text embedding, and the image is contrastively aligned to all K embeddings. A facet-decoupled attention mask concatenates the prompts into one forward pass so facets do not leak into each other, and because the LLM is frozen, all text embeddings are computed once offline. The authors report that this design yields higher zero-shot classification, short- and long-context retrieval, multilingual retrieval, and linear-probe accuracy than CLIP-style models trained on the same or much larger data, and that training time stays competitive with CLIP.

Load-bearing premise

The reported gains are credited to the FLAME design, so the claim rests on the assumption that a similarly large trainable text encoder would not achieve the same results on the same small datasets; every comparison pairs FLAME with far smaller CLIP-style encoders, and only LLMs are swept in the backbone ablation.

Editorial extensions

If this is right

  • CLIP-style pre-training can consume full long captions directly, eliminating the need to split captions into 77-token chunks or synthesize shorter rewrites.
  • A model trained only on English text can generalize zero-shot to 36 languages at retrieval time, because the frozen LLM supplies multilingual text representations.
  • The visual encoder inherits finer semantics: on CC3M, FLAME raises ImageNet top-1 by 4.9 points over the prior state of the art, and the average gain over 10 downstream zero-shot classification sets is 9.3 points.
  • Training cost stays near CLIP's because text embeddings are precomputed offline; the extra cost is a one-time preprocessing step.
  • Scaling the visual backbone from ViT-B/16 to ViT-L/14 continues to improve zero-shot and linear-probe accuracy, so the approach is not tied to one visual encoder size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains persist when a trainable text encoder of equal capacity is used, the framework's principle generalizes beyond LLMs: any rich frozen text model could act as a fixed label source, shrinking the need for curated image-text pairs.
  • The cross-lingual transfer suggests visual features inherit the LLM's language coverage; one testable extension is to probe low-resource languages deliberately excluded from the LLM's pretraining corpus.
  • The one-word-per-facet constraint likely under-exploits the LLM's compositional reasoning; allowing short phrases per facet could improve compositionality benchmarks such as SugarCrepe without hurting single-word retrieval.
  • Because the text space is fixed, FLAME could serve as a stable initialization for downstream vision-language models, with the frozen LLM embeddings acting as a semantic anchor during later fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes FLAME, a language-image pre-training framework that uses a frozen large language model (Mistral-Nemo) as the text encoder in a CLIP-style contrastive setup. Two technical components are introduced: multifaceted prompt distillation, which extracts several semantic embeddings from a single long caption using a set of hierarchical prompts, and facet-decoupled attention, which enables single-pass inference over multiple prompts with masked attention and supports offline pre-computation of text embeddings. Experiments on CC3M and YFCC15M report large improvements over CLIP and DreamLIP in zero-shot classification, long-context retrieval (ShareGPT4V-val, Urban-1k), multilingual retrieval (Crossmodal-3600), and short-context retrieval (MSCOCO, Flickr30k), together with ablations on the number of prompts, backbone choices, and semantic levels.

Significance. If the reported results are trustworthy, the paper makes a useful empirical contribution: it shows that a frozen, high-capacity LLM can serve as the text encoder for data-efficient language-image pre-training, naturally handling long and multilingual inputs without architectural modification of the text side. The paper also provides a practical efficiency mechanism (facet-decoupled attention plus offline embedding) that makes the approach computationally feasible, and it ships code and several ablations that test the design choices. The cross-modal gains are large and consistent across diverse benchmarks, which is a genuine strength. However, the central attribution of these gains to the specific FLAME design is weakened by the absence of a same-capacity trainable text-encoder control, and the reliance on a re-implemented, unreleased synthetic caption dataset limits the verifiability of head-to-head comparisons with DreamLIP.

major comments (2)
  1. [§4.2–4.3, Tables 1–3 and 7] Every headline comparison pairs FLAME's frozen Mistral-Nemo text encoder (~12B parameters) with CLIP-family text towers that are roughly two orders of magnitude smaller, and Table 7 sweeps only among LLMs (Mistral-7B, Mistral-Nemo, Llama-3.1-8B). No trainable text encoder of comparable capacity is trained or evaluated, so the reported gains (e.g., 4.9% ImageNet top-1 on CC3M, 44.4% multilingual recall@1, 34.6% long-context recall@1) could in principle be explained by LLM scale and prior linguistic pre-training alone, independent of the frozen design or the multifaceted prompt distillation. To support the paper's challenge to the conventional wisdom about frozen text encoders, please add a control experiment that trains a same-capacity trainable text encoder on the same data with the same objective, or that fine-tunes (e.g., with LoRA) the same LLM text encoder in the same framework. Reporting the same metrics for that control would directly separate the effect of freezing from the effect of LLM capacity; without it, the central attribution is not fully secured.
  2. [§4.1, Dataset Details] The synthetic captions used for training are a re-implementation of DreamLIP's pipeline with MiniCPM-Llama3-V 2.5, not the exact released DreamLIP captions, and the generated captions are not yet released. Since Tables 1–3 compare against published DreamLIP numbers, differences in caption quality or distribution could account for part of the reported gap. Please release the synthetic caption sets, or retrain DreamLIP on the same re-implemented captions under the same schedule and report both results, so that the comparison is apples-to-apples.
minor comments (5)
  1. [§3.2, Prompt template] The prompt template in Section 3.2 uses color annotations (blue for the shared prefix, red for the extraction position) that are not visible in the printed text; please spell out which part of the prompt is the shared prefix and where the final hidden state is extracted, so the construction is unambiguous.
  2. [References, [21]] The paper cites Mistral-Nemo as [21], but reference [21] is the Mistral 7B technical report; please provide the correct citation for Mistral-Nemo or disambiguate the two models.
  3. [§4.1, Implementation Details] The text says 'we apply a single prompt for inference' for short-text inputs and identifies it as 'the first prompt at the scene level', but the supplementary prompt list marks the scene-level prompt as default for both long and short inputs; please clarify why a different prompt is not used for short inputs and whether the choice affects results.
  4. [Table 6 and §4.5] The number of prompts K=7 is selected after observing performance on the evaluation benchmarks (ImageNet, MSCOCO, Flickr30k, ShareGPT4V-val, Urban-1k). This is a form of test-set peeking; please state explicitly that K was chosen on a held-out validation set or report the sensitivity of the choice to the evaluation split.
  5. [Figure 6] The embedding and training overhead figure is referenced in the text but not reproduced in this version; please ensure the figure is included in the final submission and that the y-axis and baseline definitions are clearly labeled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FLAME's reported gains are empirical evaluations against external benchmarks, with no fitted parameter or self-citation chain presented as a prediction.

full rationale

FLAME's central claim is that a frozen LLM text encoder, combined with multifaceted prompt distillation, outperforms prior CLIP-style models under data-limited language-image pre-training. This is an empirical claim evaluated on external benchmarks (ImageNet, COCO, Flickr30k, Crossmodal-3600, Urban-1k, ShareGPT4V-val, and others), not a derivation from the method's own equations. The text embeddings are produced by a frozen Mistral-Nemo LLM plus a trainable visual encoder and projection trained with the contrastive objective in Eq. (1); no parameter is fitted to the benchmark numbers and then reported as a prediction. The prompt set and prompt count are design choices ablated in Table 6, and choosing 7 prompts after observing saturation is a model-selection procedure, not a circular reduction: the reported numbers are model outputs on held-out test sets, not the quantity being optimized. The manuscript's self-citations are incidental (e.g., reference [4] appears in a generic list of LLMs, and reference [5] appears as a ViT-B/16 citation) and are not load-bearing for the central claim. The absence of a same-capacity trainable text-encoder control is a legitimate attribution confound, but it does not make the claim true by construction. No step in the paper reduces, by the paper's own equations or by a self-citation chain, to its own inputs. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entity. Its load-bearing assumptions are behavioral properties of LLMs (last-token embeddings, prompt sensitivity, multilingual ability) and the quality of the re-implemented caption data. The main free choices are the prompt set and prompt count, selected partly by ablation, plus the inference-time prompt rule for short text.

free parameters (3)
  • Number of prompts K = 7 (out of 9 designed)
    Selected after ablations in Table 6 across ShareGPT4V-val, Urban-1k, MSCOCO, Flickr30k, ImageNet and downstream datasets; 7 gives the best balanced performance.
  • Prompt templates and semantic hierarchy = 9 templates, 7 used by default
    Hand-designed according to entity/interaction/scene decomposition; not derived from a theory, and results could shift with different prompts.
  • Inference prompt choice for short text = Scene-level prompt 'this image description means in just one word'
    The paper states a single prompt is used for short-text inference; this is a design choice not derived from the training objective.
assumptions (4)
  • domain assumption Final-token hidden states of a causal LLM, prompted to respond with one word, yield discriminative text embeddings.
    Relied on in Sec. 3.2 and the prompt templates; supported by prior LLM embedding works [22,44,61,65] but not proven here.
  • domain assumption Facet-decoupled attention prevents semantic leakage between prompts while sharing the prefix KV cache.
    Introduced in Sec. 3.3; the paper does not measure how much cross-facet leakage remains or whether it matters.
  • domain assumption Synthetic captions generated by MiniCPM-Llama3-V 2.5 with DreamLIP-style prompts are adequate and comparable to DreamLIP's training captions.
    Sec. 4.1 says the synthesis pipeline is re-implemented; differences in generated captions could affect the comparison.
  • domain assumption Contrastive learning against fixed, precomputed text embeddings transfers to zero-shot classification and retrieval.
    The whole training setup in Sec. 3.1 assumes the frozen text embedding space is a good target space for visual features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training." pith.science (2026). https://pith.science/paper/BWSDD7SI

@misc{pith2026241111927,
  author       = {Pith},
  title        = {Pith review of: FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWSDD7SI}},
  note         = {Machine review of arXiv:2411.11927}
}
read the original abstract

Language-image pre-training faces significant challenges due to limited data in specific formats and the constrained capacities of text encoders. While prevailing methods attempt to address these issues through data augmentation and architecture modifications, they continue to struggle with processing long-form text inputs, and the inherent limitations of traditional CLIP text encoders lead to suboptimal downstream generalization. In this paper, we propose FLAME (Frozen Large lAnguage Models Enable data-efficient language-image pre-training) that leverages frozen large language models as text encoders, naturally processing long text inputs and demonstrating impressive multilingual generalization. FLAME comprises two key components: 1) a multifaceted prompt distillation technique for extracting diverse semantic representations from long captions, which better aligns with the multifaceted nature of images, and 2) a facet-decoupled attention mechanism, complemented by an offline embedding strategy, to ensure efficient computation. Extensive empirical evaluations demonstrate FLAME's superior performance. When trained on CC3M, FLAME surpasses the previous state-of-the-art by 4.9% in ImageNet top-1 accuracy. On YFCC15M, FLAME surpasses the WIT-400M-trained CLIP by 44.4\% in average image-to-text recall@1 across 36 languages, and by 34.6% in text-to-image recall@1 for long-context retrieval on Urban-1k. Code is available at https://github.com/MIV-XJTU/FLAME.

Figures

Figures reproduced from arXiv: 2411.11927 by the authors.

Figure 1
Figure 1. Conceptual comparison of text streams. FLAME leverages frozen large language models (LLMs) to directly process long captions. With multifaceted prompts, this framework extracts diverse semantic embeddings, achieving data efficiency. Preserving LLMs’ inherent capabilities enables multilingual generalization. vision. CLIP-style models have demonstrated state-of-the￾art performance across a wide range of downstream tas… view at source ↗
Figure 2
Figure 2. FLAME overview. This framework harnesses the so￾phisticated long-text comprehension capabilities of large language models to conduct language-image pre-training directly on long captions. Based on multifaceted prompts, it extracts a diverse array of representations embedded within the long caption, thereby en￾hancing semantic alignment. processing long-form text and understanding multiple lan￾guages, while enabling … view at source ↗
Figure 3
Figure 3. Facet-decoupled attention. By streamlining all prompts with a shared prefix and applying this facet-decoupled attention mask, the overhead of feature extraction is greatly reduced. The positions in red indicate the features to be extracted. • Constrained Output Space: All prompts are designed to generate single-word responses, forcing the LLM to dis￾till complex visual concepts into concise, discriminative features.… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Multilingual zero-shot retrieval recall@1 results on Crossmodal-3600 (text-to-image retrieval). Despite being trained solely on English datasets, FLAME achieves outstanding average performance across all 36 languages, surpassing mSigLIP, which is trained on the multili…
Figure 5
Figure 5. Figure 5: Semantic interpretability. Based on vocabulary map￾ping, FLAME achieves patch-to-word translation with competent interpretability of language-image alignment. We apply average pooling to reduce the number of words for a clearer presentation. vocabulary. As shown in [P…
Figure 6
Figure 6. Figure 6: Embedding and training overhead. FDA denotes facet-decoupled attention. While online training is hindered by inefficiencies, our approach of freezing LLMs allows for offline embedding to lower the overhead. Our FDA further boosts the efficiency of offline embedding, re…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.

Reference graph

Works this paper leans on

65 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 3

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023

    arXiv 2023

  4. [4]

    Is a 3d-tokenized llm the key to reliable autonomous driving? arXiv preprint arXiv:2405.18361, 2024

    Yifan Bai, Dongming Wu, Yingfei Liu, Fan Jia, Weixin Mao, Ziheng Zhang, Yucheng Zhao, Jianbing Shen, Xing Wei, Tian- cai Wang, et al. Is a 3d-tokenized llm the key to reliable autonomous driving? arXiv preprint arXiv:2405.18361, 2024. 3

    arXiv 2024

  5. [5]

    Ar- trackv2: Prompting autoregressive tracker where to look and how to describe

    Yifan Bai, Zeyang Zhao, Yihong Gong, and Xing Wei. Ar- trackv2: Prompting autoregressive tracker where to look and how to describe. In CVPR, 2024. 5

    work page 2024

  6. [6]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mos- bach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024. 2, 3, 4

    arXiv 2024

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. 3

    work page 2020

  8. [8]

    Cross-lingual and multilingual clip

    Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multilingual clip. In LREC, 2022. 2, 3

    work page 2022

Show all 65 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    2023 arXiv

  2. [10]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. In ICLR, 2023. 6

    2023

  3. [11]

    Altclip: Altering the language encoder in clip for extended language capabilities

    Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu. Altclip: Altering the language encoder in clip for extended language capabilities. In ACL Findings, 2023. 3

    2023

  4. [12]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. In CVPR,

  5. [13]

    An image is worth 16x16 words: Transform- ers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In ICLR, 2021. 5

    2021

  6. [14]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yon- glong Tian. Improving clip training with language rewrites. In NeurIPS, 2023. 2, 3, 7, 1

    2023

  7. [15]

    Pyramidclip: Hierarchical feature alignment for vision-language model pretraining

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Ron- grong Ji, and Chunhua Shen. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining. In NeurIPS, 2022. 3

    2022

  8. [16]

    Softclip: Softer cross-modal alignment makes clip stronger

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. In AAAI, 2024. 3, 6

    2024

  9. [17]

    Hiclip: Contrastive language-image pre- training with hierarchy-aware attention

    Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-image pre- training with hierarchy-aware attention. In ICLR, 2023. 3, 7

    2023

  10. [18]

    Sugarcrepe: Fixing hack- able benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hack- able benchmarks for vision-language compositionality. In NeurIPS, 2023. 2

    2023

  11. [19]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Ha- jishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. If you use this software, please cite it as below. 5

    2021

  12. [20]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 3

    2021

  13. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  14. [22]

    Scaling sentence embeddings with large language models

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645, 2023. 2, 3, 4

    2023 arXiv

  15. [23]

    Misalign, contrast then distill: Rethinking misalign- ments in language-image pre-training

    Bumsoo Kim, Yeonsik Jo, Jinhyung Kim, and Seunghwan Kim. Misalign, contrast then distill: Rethinking misalign- ments in language-image pre-training. In ICCV, 2023. 3

    2023

  16. [24]

    Jina clip: Your clip model is also your text retriever

    Andreas Koukounas, Georgios Mastrapas, Michael G¨unther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mo- hammad Kalim Akram, Joan Fontanals Mart´ınez, Saahil Og- nawala, et al. Jina clip: Your clip model is also your text retriever. arXiv preprint arXiv:2405.20204, 2024. 2

    2024 arXiv

  17. [25]

    From scarcity to efficiency: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen- Nee Chuah, Yinfei Yang, et al. From scarcity to efficiency: Improving clip training via visual-enriched captions. arXiv preprint arXiv:2310.07699, 2023. 2, 3

    arXiv

  18. [26]

    Uni- clip: Unified framework for contrastive language-image pre- training

    Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uni- clip: Unified framework for contrastive language-image pre- training. In NeurIPS, 2022. 3

    2022

  19. [27]

    Gecko: Versatile text embed- dings distilled from large language models

    Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Ka- padia, Wen Ding, et al. Gecko: Versatile text embed- dings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024. 3

    2024 arXiv

  20. [28]

    Meta-task prompting elicits embedding from large language models

    Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. Meta-task prompting elicits embedding from large language models. In ACL, 2024. 3

    2024

  21. [29]

    Scene graph generation: A comprehensive survey

    Hongsheng Li, Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Xia Zhao, Syed Afaq Ali Shah, and Mohammed Bennamoun. Scene graph generation: A comprehensive survey. Neurocomputing, 566:127052, 2024. 4

    2024

  22. [30]

    Grounded language- image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language- image pre-training. In CVPR, 2022. 1

    2022

  23. [31]

    An inverse scaling law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In NeurIPS, 2023. 1

    2023

  24. [32]

    Scene graph generation from objects, phrases and region captions

    Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xi- aogang Wang. Scene graph generation from objects, phrases and region captions. In ICCV, 2017. 4

    2017

  25. [33]

    Supervi- sion exists everywhere: A data efficient contrastive language- image pre-training paradigm

    Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervi- sion exists everywhere: A data efficient contrastive language- image pre-training paradigm. In ICLR, 2022. 3, 5

    2022

  26. [34]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR, 2023. 3

    2023

  27. [35]

    Language quantized autoencoders: Towards unsupervised text-image alignment

    Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. In NeurIPS, 2023. 2

    2023

  28. [36]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. arXiv preprint arXiv:2311.18765, 2023. 2, 3, 6, 7

    2023 arXiv

  29. [37]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 1

    2017 arXiv

  30. [38]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In ECCV, 2022. 3

    2022

  31. [39]

    Gen- erative representational instruction tuning

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Gen- erative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024. 3

    2024 arXiv

  32. [40]

    Docci: De- scriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: De- scriptions of connected and contrasting images. In ECCV,

  33. [41]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 5, 6, 7, 3

    2021

  34. [42]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1

    2022

  35. [43]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. 5

    2018

  36. [44]

    Repetition improves lan- guage model embeddings

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves lan- guage model embeddings. arXiv preprint arXiv:2402.15449,

    arXiv

  37. [45]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 1

    2023 arXiv

  38. [46]

    Crossmodal-3600: A massively multilingual multi- modal evaluation dataset

    Ashish V Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multi- modal evaluation dataset. In EMNLP, 2022. 2, 6, 1

    2022

  39. [47]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In CVPR, 2022. 2

    2022

  40. [48]

    Stablerep: Synthetic images from text-to- image models make strong visual representation learners

    Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to- image models make strong visual representation learners. In NeurIPS, 2023. 3

    2023

  41. [49]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  42. [50]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurIPS, 2021. 2

    2021

  43. [51]

    Nllb-clip–train performant multilin- gual image retrieval model on a budget

    Alexander Visheratin. Nllb-clip–train performant multilin- gual image retrieval model on a budget. arXiv preprint arXiv:2309.01859, 2023. 2, 3, 6, 7

    2023 arXiv

  44. [52]

    Improving text embeddings with large language models.arXiv preprint arXiv:2401.00368,

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Ran- gan Majumder, and Furu Wei. Improving text embeddings with large language models.arXiv preprint arXiv:2401.00368,

    arXiv

  45. [53]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022. 1

    2022

  46. [54]

    Chinese clip: Con- trastive vision-language pretraining in chinese

    An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese clip: Con- trastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335, 2022. 2, 3, 6, 7

    2022 arXiv

  47. [55]

    Alip: Adaptive language-image pre-training with synthetic caption

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. In ICCV,

  48. [56]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 3

    2021 arXiv

  49. [57]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 3

    2022 arXiv

  50. [58]

    Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms

    Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. In NeurIPS, 2023. 2

    2023

  51. [59]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR,

  52. [60]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 1, 6

    2023

  53. [61]

    Simple techniques for enhancing sentence embeddings in generative language models

    Bowen Zhang, Kehua Chang, and Chunping Li. Simple techniques for enhancing sentence embeddings in generative language models. In ICIC, 2024. 2, 3, 4

    2024

  54. [62]

    Long-clip: Unlocking the long-text capability of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In ECCV, 2024. 2, 3, 5, 6

    2024

  55. [63]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In ECCV, 2024. 2, 3, 5, 6, 7, 8, 1

    2024

  56. [64]

    Beyond text: Frozen large language models in visual signal comprehension

    Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models in visual signal comprehension. In CVPR, 2024. 2

    2024

  57. [65]

    yi”. After thinking step by step, the category of the main object in this image means in just one word:

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Promptreps: Prompting large lan- guage models to generate dense and sparse representations for zero-shot document retrieval. In EMNLP, 2024. 2, 3, 4 FLAME : Frozen Large Language Models Enable Data-Effic...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.