{"id":"93249d0a-63a2-42ec-9071-7b8b5382672b","arxiv_id":"2411.11927","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Frozen LLM text encoders, combined with multi-prompt hidden-state extraction and cached embeddings, make CLIP-style pre-training data-efficient, long-context aware, and multilingual.","lead":"FLAME replaces the CLIP text encoder with a frozen large language model, so image-text pre-training can use long captions and generalize to many languages even with small training sets. On CC3M and YFCC15M it reports substantially higher retrieval and classification accuracy than prior CLIP-style models, with code released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing same-capacity trainable text-encoder control leaves the central claim undersupported: gains may stem from LLM scale/pretraining rather than FLAME's frozen+multi-facet design.","rationale":"The reader's weakest_assumption precisely identifies the missing same-capacity trainable text-encoder control, and I agree this is the most load-bearing issue. The paper's strongest claim is that frozen LLMs can effectively enhance language-image pre-training, and every comparison in Tables 1-3, 4, and 5 uses small CLIP-family text encoders as baselines. The only text-encoder ablation (Table 7) is among LLMs, so the effect of LLM scale/pretraining is not separated from the proposed mechanism. This is a correctness risk, not a disagreement with consensus: the frozen-LLM approach may genuinely work, but the evidence as presented cannot rule out that the improvements simply reflect using a much larger, better-pretrained text encoder. The proposed test—training a comparable-capacity trainable text encoder on the same data—would directly resolve this. The secondary concern about synthetic caption reproducibility is real but would not by itself change my verdict; combined with the primary confound it strengthens the case for conditional acceptance. Since the reader already recommends CONDITIONAL, my read does not change the verdict.","tokens_in":24265,"tokens_out":4689,"duration_ms":48198,"concrete_test":"Train a text encoder with comparable capacity to Mistral-Nemo (~12B parameters) on the same CC3M/YFCC15M data, using the same multi-facet prompts and the same ViT-B/16 visual encoder, with the text encoder trainable (not frozen). Compare zero-shot ImageNet top-1, Crossmodal-3600 recall@1, and Urban-1k retrieval against FLAME. If the trainable encoder matches or exceeds FLAME, the gains are not attributable to the frozen design or multi-facet prompts; if it underperforms, the frozen LLM's pretraining is the key ingredient. A cheaper complementary test: fine-tune Mistral-Nemo (unfreeze it) with the same prompts and data; if the unfrozen variant outperforms FLAME, the claim that freezing is necessary or beneficial is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that frozen LLMs can effectively serve as text encoders for data-efficient language-image pre-training. But every headline comparison pairs FLAME (frozen Mistral-Nemo, ~12B parameters) with CLIP-family text encoders that are roughly two orders of magnitude smaller (~63M parameters for the ViT-B/16 text tower). Table 7 only sweeps among LLMs (Mistral-7B, Mistral-Nemo, Llama-3.1-8B); no trainable text encoder of comparable capacity is trained or evaluated. Consequently, the reported gains—4.9% ImageNet top-1 on CC3M, 44.4% average multilingual recall@1, 34.6% long-context recall@1—could be explained by the scale and prior linguistic knowledge of the frozen LLM alone, independent of the proposed multi-facet prompt distillation or the fact that the encoder is frozen. The claim that the specific FLAME design causes the improvement therefore rests on a missing control. A secondary reproducibility issue compounds this: the synthetic captions are a re-implementation of DreamLIP's pipeline (with MiniCPM-Llama3-V 2.5) rather than the exact released captions, and they are not yet released, so baseline comparability is not fully verifiable. Both issues are concrete and addressable, but until a same-capacity trainable text encoder is tested, the central attribution is not secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLAME, a language-image pre-training framework that uses a frozen large language model (Mistral-Nemo) as the text encoder in a CLIP-style contrastive setup. Two technical components are introduced: multifaceted prompt distillation, which extracts several semantic embeddings from a single long caption using a set of hierarchical prompts, and facet-decoupled attention, which enables single-pass inference over multiple prompts with masked attention and supports offline pre-computation of text embeddings. Experiments on CC3M and YFCC15M report large improvements over CLIP and DreamLIP in zero-shot classification, long-context retrieval (ShareGPT4V-val, Urban-1k), multilingual retrieval (Crossmodal-3600), and short-context retrieval (MSCOCO, Flickr30k), together with ablations on the number of prompts, backbone choices, and semantic levels.","tokens_in":24517,"tokens_out":5863,"duration_ms":58870,"significance":"If the reported results are trustworthy, the paper makes a useful empirical contribution: it shows that a frozen, high-capacity LLM can serve as the text encoder for data-efficient language-image pre-training, naturally handling long and multilingual inputs without architectural modification of the text side. The paper also provides a practical efficiency mechanism (facet-decoupled attention plus offline embedding) that makes the approach computationally feasible, and it ships code and several ablations that test the design choices. The cross-modal gains are large and consistent across diverse benchmarks, which is a genuine strength. However, the central attribution of these gains to the specific FLAME design is weakened by the absence of a same-capacity trainable text-encoder control, and the reliance on a re-implemented, unreleased synthetic caption dataset limits the verifiability of head-to-head comparisons with DreamLIP.","major_comments":[{"comment":"Every headline comparison pairs FLAME's frozen Mistral-Nemo text encoder (~12B parameters) with CLIP-family text towers that are roughly two orders of magnitude smaller, and Table 7 sweeps only among LLMs (Mistral-7B, Mistral-Nemo, Llama-3.1-8B). No trainable text encoder of comparable capacity is trained or evaluated, so the reported gains (e.g., 4.9% ImageNet top-1 on CC3M, 44.4% multilingual recall@1, 34.6% long-context recall@1) could in principle be explained by LLM scale and prior linguistic pre-training alone, independent of the frozen design or the multifaceted prompt distillation. To support the paper's challenge to the conventional wisdom about frozen text encoders, please add a control experiment that trains a same-capacity trainable text encoder on the same data with the same objective, or that fine-tunes (e.g., with LoRA) the same LLM text encoder in the same framework. Reporting the same metrics for that control would directly separate the effect of freezing from the effect of LLM capacity; without it, the central attribution is not fully secured.","section":"§4.2–4.3, Tables 1–3 and 7"},{"comment":"The synthetic captions used for training are a re-implementation of DreamLIP's pipeline with MiniCPM-Llama3-V 2.5, not the exact released DreamLIP captions, and the generated captions are not yet released. Since Tables 1–3 compare against published DreamLIP numbers, differences in caption quality or distribution could account for part of the reported gap. Please release the synthetic caption sets, or retrain DreamLIP on the same re-implemented captions under the same schedule and report both results, so that the comparison is apples-to-apples.","section":"§4.1, Dataset Details"}],"minor_comments":[{"comment":"The prompt template in Section 3.2 uses color annotations (blue for the shared prefix, red for the extraction position) that are not visible in the printed text; please spell out which part of the prompt is the shared prefix and where the final hidden state is extracted, so the construction is unambiguous.","section":"§3.2, Prompt template"},{"comment":"The paper cites Mistral-Nemo as [21], but reference [21] is the Mistral 7B technical report; please provide the correct citation for Mistral-Nemo or disambiguate the two models.","section":"References, [21]"},{"comment":"The text says 'we apply a single prompt for inference' for short-text inputs and identifies it as 'the first prompt at the scene level', but the supplementary prompt list marks the scene-level prompt as default for both long and short inputs; please clarify why a different prompt is not used for short inputs and whether the choice affects results.","section":"§4.1, Implementation Details"},{"comment":"The number of prompts K=7 is selected after observing performance on the evaluation benchmarks (ImageNet, MSCOCO, Flickr30k, ShareGPT4V-val, Urban-1k). This is a form of test-set peeking; please state explicitly that K was chosen on a held-out validation set or report the sensitivity of the choice to the evaluation split.","section":"Table 6 and §4.5"},{"comment":"The embedding and training overhead figure is referenced in the text but not reproduced in this version; please ensure the figure is included in the final submission and that the y-axis and baseline definitions are clearly labeled.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical case is strong but the missing same-capacity trainable text-encoder control is a genuine gap for the 'frozen' claim, and the unreleased re-implemented captions make the DreamLIP comparison hard to audit. Both issues are addressable with additional experiments or releases, so I recommend major revision rather than rejection. The model-selection aspect (prompt count chosen on test benchmarks) is a wider community issue and, while worth flagging, does not by itself change my verdict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: FLAME is a real and useful combination—frozen LLM as the text tower in contrastive image-language pre-training, with multi-prompt facet extraction and a facet-decoupled attention mask that makes single-pass inference possible. The reported gains are large and consistent across retrieval, classification, multilingual, and long-context benchmarks. That is not noise.\n\nThe genuinely new part is the package: prior work used LLMs for text embeddings generally (LLM2Vec, PromptReps) and long synthetic captions for CLIP (DreamLIP), but nobody has put a frozen LLM directly into the contrastive loop with multi-facet prompting. The facet-decoupled attention is a neat efficiency trick, and the offline embedding strategy is what makes the whole thing practical. The ablations on prompt count and backbone choice are sensible, and the semantic interpretability section (patch to word via the LLM's projection head) is a small but nice bonus. The authors also ship code, which is more than many papers do.\n\nThe soft spot is the one the stress-test flags, and I think it is real: every headline comparison pairs FLAME (frozen Mistral-Nemo, ~12B) with CLIP text towers around 63M. Table 7 sweeps only among LLMs. There is no trainable text encoder of comparable capacity. So the central claim—that the frozen + multi-facet design is what drives the gains—is not fully secured. A big part of the improvement could be scale and prior linguistic knowledge of the LLM alone. This is not a fatal flaw; the method still works as a recipe. But it changes the strength of the claim from \"FLAME's design is superior\" to \"a frozen 12B LLM beats a 63M text tower, and FLAME is a reasonable way to harness it.\" That is still worth publishing, but the authors should train a comparable-capacity trainable encoder (even a soft prompt / LoRA version of the same LLM) as a control.\n\nSecondary issues are minor: the synthetic captions are a re-implementation of DreamLIP's pipeline, not released yet, so exact reproduction depends on their release. Prompt-count selection on the evaluation suite is a minor model-selection concern; the plateau behavior in Table 6 makes it less worrying.\n\nBottom line: this is a serious paper. The direction is promising, the experiments are broad, and the missing control is addressable. I would send it to peer review and make the same-capacity text encoder control a required addition, not a desk reject.","headline":"A genuinely useful frozen-LLM-as-text-tower recipe for CLIP-style training, with one real confound: no same-capacity trainable text encoder control, so the central attribution is not yet secure.","tokens_in":25076,"tokens_out":2280,"would_cite":true,"duration_ms":52942,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen LLM can replace CLIP's text encoder and beat it with far less data.","keywords":["language-image pre-training","frozen large language models","text encoder","contrastive learning","long captions","multilingual retrieval","prompt distillation","data efficiency"],"falsifier":"Train a CLIP-style model with a trainable text transformer of comparable capacity to Mistral-Nemo on the same CC3M and YFCC15M caption sets used for FLAME, with the same multi-facet prompt decomposition. If its ImageNet top-1, long-context recall, and Crossmodal-3600 results match or exceed FLAME's, the claim that frozen LLMs plus prompt distillation drive the gains is falsified, because the improvement would then be attributable to text-encoder scale rather than to the framework.","tokens_in":24038,"feed_emoji":"🧊","tokens_out":7587,"duration_ms":67471,"temperature":0.7,"pith_summary":"The paper claims that the standard CLIP text encoder, limited to 77 tokens and weak multilingual coverage, is the real bottleneck in data-efficient language-image pre-training, and that a frozen large language model removes that bottleneck. FLAME prompts a frozen LLM with a small set of single-word 'facet' questions -- main object, attribute, action, scene atmosphere, color -- and aligns each resulting embedding against the image with a symmetric contrastive loss. Trained on just CC3M, it beats the previous state of the art by 4.9 points in ImageNet top-1 accuracy; trained on YFCC15M, it outperforms a CLIP trained on WIT-400M in long-context retrieval and in zero-shot retrieval across all 36 Crossmodal-3600 languages. If correct, this means long captions can be used directly as training labels, without chunking, caption synthesis, or positional-encoding tricks, while keeping training cost near CLIP's through offline embedding.","feed_headline":"A frozen LLM as text encoder beats CLIP with far less data","feed_subtitle":"Prompting for object, action, and scene facets lifts ImageNet top-1 by 4.9 points on just 3M captions.","key_machinery":"The load-bearing mechanism is multifaceted prompt distillation over a frozen decoder-only LLM. A set of seven prompts shares one prefix containing the caption, then forces a single-word response for a distinct facet (entity category, entity attribute, interaction, scene context, atmosphere, color); the final-token hidden state for each prompt becomes a text embedding. Facet-decoupled attention packs all prompts into one sequence with a mask that blocks attention across facets, giving a 3.6x speedup over separate forward passes. Offline caching of these embeddings moves the LLM out of the training loop entirely, so the trainable part is only the visual encoder plus an MLP projection.","core_discovery":"The central claim, stated on the authors' terms, is that 'frozen LLMs can effectively enhance language-image pre-training through rich semantic representations,' overturning the conventional wisdom that frozen text encoders are suboptimal. FLAME feeds the full long caption to the LLM and appends K prompts, each asking for a single-word answer about one semantic facet; the hidden state at each prompt's final token is used as a separate text embedding, and the image is contrastively aligned to all K embeddings. A facet-decoupled attention mask concatenates the prompts into one forward pass so facets do not leak into each other, and because the LLM is frozen, all text embeddings are computed once offline. The authors report that this design yields higher zero-shot classification, short- and long-context retrieval, multilingual retrieval, and linear-probe accuracy than CLIP-style models trained on the same or much larger data, and that training time stays competitive with CLIP.","pith_inferences":["If the gains persist when a trainable text encoder of equal capacity is used, the framework's principle generalizes beyond LLMs: any rich frozen text model could act as a fixed label source, shrinking the need for curated image-text pairs.","The cross-lingual transfer suggests visual features inherit the LLM's language coverage; one testable extension is to probe low-resource languages deliberately excluded from the LLM's pretraining corpus.","The one-word-per-facet constraint likely under-exploits the LLM's compositional reasoning; allowing short phrases per facet could improve compositionality benchmarks such as SugarCrepe without hurting single-word retrieval.","Because the text space is fixed, FLAME could serve as a stable initialization for downstream vision-language models, with the frozen LLM embeddings acting as a semantic anchor during later fine-tuning."],"forward_implications":["CLIP-style pre-training can consume full long captions directly, eliminating the need to split captions into 77-token chunks or synthesize shorter rewrites.","A model trained only on English text can generalize zero-shot to 36 languages at retrieval time, because the frozen LLM supplies multilingual text representations.","The visual encoder inherits finer semantics: on CC3M, FLAME raises ImageNet top-1 by 4.9 points over the prior state of the art, and the average gain over 10 downstream zero-shot classification sets is 9.3 points.","Training cost stays near CLIP's because text embeddings are precomputed offline; the extra cost is a one-time preprocessing step.","Scaling the visual backbone from ViT-B/16 to ViT-L/14 continues to improve zero-shot and linear-probe accuracy, so the approach is not tied to one visual encoder size."],"supporting_citations":[{"why":"Defines the CLIP contrastive framework and the WIT-400M-trained CLIP baseline that FLAME outperforms in retrieval and multilingual tests.","marker":"[41]"},{"why":"DreamLIP is the prior state-of-the-art on CC3M/YFCC15M; its long/short caption synthesis pipeline is re-implemented for FLAME's training data and its results are the main accuracy baseline.","marker":"[63]"},{"why":"Long-CLIP is the long-context baseline FLAME compares against; it extends CLIP via positional-encoding interpolation and supplies the Urban-1k and ShareGPT4V benchmarks.","marker":"[62]"},{"why":"LaCLIP represents the data-augmentation approach (language rewrites) and the cited conventional wisdom that frozen text encoders are suboptimal.","marker":"[14]"},{"why":"LiT is the locked-text-encoder result that FLAME directly challenges by showing frozen LLMs can work if the frozen model is large and prompted well.","marker":"[59]"},{"why":"Mistral 7B is the LLM family used as the frozen text encoder; Mistral-Nemo is the default backbone in all main experiments.","marker":"[21]"},{"why":"LLM2Vec shows decoder-only LLMs can be effective text encoders, providing the methodological basis for using causal LLMs with prompt-based embeddings.","marker":"[6]"},{"why":"MLLM-A is a caption-synthesis baseline used in classification and retrieval comparisons, showing the data-centric alternative FLAME avoids.","marker":"[36]"}],"fun_headline_variants":["Frozen LLM text encoder outshines CLIP with a fraction of data","Facet prompts let frozen LLM beat CLIP on far less data","Frozen LLM as text encoder: data-efficient CLIP rival","Frozen LLM text encoder boosts multilingual retrieval beyond CLIP","Data-efficient pretraining with frozen LLM text encoder beats CLIP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains are credited to the FLAME design, so the claim rests on the assumption that a similarly large trainable text encoder would not achieve the same results on the same small datasets; every comparison pairs FLAME with far smaller CLIP-style encoders, and only LLMs are swept in the backbone ablation.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM text encoder outshines CLIP with a fraction of data","Facet prompts let frozen LLM beat CLIP on far less data","Frozen LLM as text encoder: data-efficient CLIP rival","Frozen LLM text encoder boosts multilingual retrieval beyond CLIP","Data-efficient pretraining with frozen LLM text encoder beats CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001436,"raw_usage":{"total_tokens":5822,"prompt_tokens":1011,"completion_tokens":4811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":4717}},"tokens_in":627,"tokens_out":4811,"duration_ms":33073,"temperature":1.0,"reasoning_tokens":4717,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:33:18.084107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CLIP-style model with a trainable text transformer of comparable capacity to Mistral-Nemo on the same CC3M and YFCC15M caption sets used for FLAME, with the same multi-facet prompt decomposition. If its ImageNet top-1, long-context recall, and Crossmodal-3600 results match or exceed FLAME's, the claim that frozen LLMs plus prompt distillation drive the gains is falsified, because the improvement would then be attributable to text-encoder scale rather than to the framework.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP contrastive framework and the WIT-400M-trained CLIP baseline that FLAME outperforms in retrieval and multilingual tests."},{"cited_title":"Dreamlip: Language- image pre-training with long captions","cited_arxiv_id":null,"evidence_quote":"DreamLIP is the prior state-of-the-art on CC3M/YFCC15M; its long/short caption synthesis pipeline is re-implemented for FLAME's training data and its results are the main accuracy baseline."},{"cited_title":"Long-clip: Unlocking the long-text capability of clip","cited_arxiv_id":null,"evidence_quote":"Long-CLIP is the long-context baseline FLAME compares against; it extends CLIP via positional-encoding interpolation and supplies the Urban-1k and ShareGPT4V benchmarks."},{"cited_title":"Improving clip training with language rewrites","cited_arxiv_id":null,"evidence_quote":"LaCLIP represents the data-augmentation approach (language rewrites) and the cited conventional wisdom that frozen text encoders are suboptimal."},{"cited_title":"Lit: Zero-shot transfer with locked-image text tuning","cited_arxiv_id":null,"evidence_quote":"LiT is the locked-text-encoder result that FLAME directly challenges by showing frozen LLMs can work if the frozen model is large and prompted well."}],"review_version":1}