{"id":"52c131d9-ca84-4e09-a366-3ea1a82a11d4","arxiv_id":"2412.03513","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Knowledge-CLIP distills Llama 2 embeddings into CLIP and uses k-means soft concept labels to slightly improve CLIP text and image encoder scores on three benchmarks.","lead":"This paper introduces Knowledge-CLIP, a method that trains CLIP's text and image encoders using a large language model (Llama 2) as a teacher, adding knowledge distillation and k-means concept labels to the standard contrastive objective. The reported gains over CLIP are small, especially for image encoders, and the supporting experiments lack ablations and error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Image-encoder gains are confounded by the jointly modified text encoder; no fixed-text control isolates image-encoder improvement.","rationale":"The reader's stated weakest assumption is that k-means clusters of Llama 2 embeddings correspond to semantic attributes; that is a legitimate mechanism-level concern, but the more load-bearing problem is that the image-encoder evaluation is confounded by the jointly modified text encoder. The reader's rationale does mention this confound ('AWA2/CUB evaluation uses the modified text encoder as well'), but the reader does not list it as the weakest assumption. I agree with the reader's overall REJECT verdict because the central claim that both encoders improve is not established by the provided experiments. The concrete fixed-text test would directly settle whether the image encoder contributes any independent improvement, and the contrastive-only ablation would separate the new losses from continued fine-tuning. Until such controls are reported, the small AWA2/CUB gains cannot be attributed to Knowledge-CLIP's image encoder specifically.","tokens_in":6213,"tokens_out":3881,"duration_ms":40658,"concrete_test":"Re-evaluate AWA2 and CUB attribute prediction using the original frozen CLIP text encoder to encode all prompts, for both the original CLIP image encoder and the Knowledge-CLIP image encoder, keeping every other evaluation step identical. If Knowledge-CLIP's image encoder does not outperform CLIP's image encoder on top-1 accuracy under this fixed-text protocol, then the reported gains are attributable to the text encoder, not the image encoder. Also train a contrastive-loss-only fine-tuned CLIP on the same 500k subset as an ablation to rule out continued-pretraining effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 compares original CLIP (both encoders unchanged) against Knowledge-CLIP (both encoders modified). The AWA2/CUB evaluation protocol encodes each attribute prompt with the model's own text encoder, so the reported +0.9 and +0.5 top-1 gains could come entirely from the distilled text encoder rather than from the image encoder. The paper provides no control that fixes the text encoder while swapping only the image encoder, and no ablation that trains CLIP with the contrastive loss alone on the same 500k subset. Continued fine-tuning on that subset could itself explain the small gains. The paper's own Section 4.3 calls the image improvement 'not strong,' and no error bars are reported. Because the central claim is that both encoders improve, and the image-encoder half is evaluated through a pipeline that includes the modified text encoder, the evidence does not isolate the image encoder's contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Knowledge-CLIP, a method to fine-tune CLIP by distilling text-embedding knowledge from Llama 2, learning soft concept labels obtained by k-means clustering of Llama 2 caption embeddings, and continuing with the CLIP contrastive objective. The training loss is a weighted sum of a text-distillation MSE term, a concept-label cross-entropy term, and a contrastive term. The authors report that the resulting text encoder improves exact-match on a CC3M reconstruction task from 35.8% (CLIP) to 44.4%, and that the image encoder improves attribute-prediction top-1 accuracy from 55.8% to 56.7% on AWA2 and from 78.3% to 78.8% on CUB, while acknowledging that the image gains are not strong.","tokens_in":6384,"tokens_out":5124,"duration_ms":54874,"significance":"If the central claims were established, Knowledge-CLIP would be a lightweight and practical recipe for injecting LLM-derived conceptual knowledge into CLIP without additional paired data. The method is clearly specified: the three loss terms in Eqs. (1)-(3) are simple, the experimental setup is small (500k pairs, 100 epochs), and the paper honestly reports the modest size of the image-encoder gains in Section 4.3. However, the current experimental design does not isolate the contribution of the image encoder, does not validate the k-means concept labels beyond a single qualitative figure, and does not include ablations or error bars. The central claim that both encoders are improved is therefore not yet supported by the evidence presented.","major_comments":[{"comment":"The image-encoder evaluation is confounded. Knowledge-CLIP differs from CLIP in both the text encoder and the image encoder, and the AWA2/CUB prompts are encoded with the model's own text encoder. The reported +0.9 and +0.5 top-1 gains could therefore come entirely from the distilled text encoder rather than from the image encoder. A control that fixes the text encoder (for example, using the original CLIP text encoder to score both models) is required to isolate the image-encoder contribution; without it, the claim that the image encoder improves is not supported.","section":"§4.3, Table 2"},{"comment":"The text-encoder evaluation is not fully circular, but it is strongly aligned with the training objective: the reconstruction EM measures how well the student's embeddings preserve Llama-2-like information, and the student was explicitly trained to mimic Llama 2. The jump from 35.8 to 44.4 is therefore partly a fidelity check. To establish independent text-encoder quality, the paper should include a benchmark that is not defined by similarity to the teacher, such as a compositional text-retrieval or attribute-prediction task using the text encoder alone.","section":"§4.2, Table 1"},{"comment":"The concept-learning objective rests on the assumption that k-means clusters of Llama 2 caption embeddings correspond to interpretable semantic attributes such as color, shape, and action. The only support is a qualitative visualization of 50 samples from one CUB class. If the cluster labels are arbitrary, training the image encoder with Lconc cannot transfer to attribute recognition. The paper needs a quantitative cluster-attribute correspondence measure, or an ablation showing that Lconc contributes beyond Lemb and Lcont.","section":"§3.3, §5.2"},{"comment":"No ablation isolates the three loss terms, and no sensitivity analysis is reported for the hyperparameters alpha, beta, gamma, and K. Because the contrastive loss Lcont is a continuation of the original CLIP objective on a 500k subset, the small image gains could be explained by additional fine-tuning alone. The authors should report results for each loss term removed, and for reasonable variations of beta and K, to show which components drive the observed changes.","section":"§3.5, §4.1"},{"comment":"All results are reported as single numbers without variance or statistical significance. The image-encoder differences on AWA2 and CUB are within a typical range of seed-to-seed variation, and Section 4.3 itself states that the increase is 'not strong.' The paper should include repeated runs or a paired evaluation with confidence intervals before drawing conclusions about image-encoder improvements.","section":"§4.1, §4.3"}],"minor_comments":[{"comment":"There are several typos: 'oduces' in the conclusion, 'KKLIP' in Table 2's caption, and 'concpetual' in Section 5.2.","section":"§6, Table 2, §5.2"},{"comment":"The double sum over i and j is confusingly indexed: s_i appears without a j index inside the sum over N. It should be written as s_{j,i} (or similar) with a single summation over samples.","section":"Eq. (2)"},{"comment":"The paper should specify how the 500k training and 100k evaluation records were sampled from the 15.7M-record dataset, and whether any preprocessing or deduplication was applied.","section":"§4.1"},{"comment":"The claimed 'more uniform' distribution of Llama 2 embeddings is supported only by a figure; a quantitative measure (for example, cluster-size entropy or a uniformity metric) would make the claim testable.","section":"Figure 4"},{"comment":"The figure caption should state what the colors represent and how the 50 samples were chosen; currently the visualization is difficult to interpret without additional annotation.","section":"Figure 5"},{"comment":"The paper should state explicitly that Llama 2 is frozen during training and that gradients do not flow into the teacher model.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a short workshop-style report rather than a fully developed journal submission. The core idea is not unsound, and the missing controls and ablations in the major comments are fixable within the manuscript's scope. If the authors can provide the fixed-text-control experiment, an independent text-encoder evaluation, a quantitative validation of the concept labels, and variance estimates, the central claim could become defensible. Without those additions, the evidence is too weak to support the conclusion that both encoders improve."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a short, honest paper with a new combination of tricks — Llama 2 embedding distillation, k-means concept pseudo-labels, and CLIP's contrastive loss — but the central image-encoder claim is not isolated. The text-encoder result is the most credible piece: exact match jumps from 35.8 to 44.4 on CC3M, and the loss equations are clean. The paper is also candid that the image gains are 'not strong' and provides a qualitative k-means visualization that supports the concept-label idea at least on one CUB class.\n\nThe soft spot is the AWA2/CUB evaluation. Table 2 compares stock CLIP with Knowledge-CLIP with both encoders changed. Attribute prompts are encoded with the model's own text encoder, so the +0.9/+0.5 top-1 gains could come entirely from the distilled text side. There is no fixed-text control, no ablation training CLIP with the contrastive loss alone on the 500k subset, and no error bars. The number of k-means clusters K is never stated, and no code is released, so the experiments aren't reproducible as-is. The text evaluation also has a mild circularity: the reconstruction task rewards output similar to Llama 2, so the EM gain partly measures teacher fidelity rather than independent quality.\n\nIf the authors added a control that keeps the text encoder fixed while swapping only the image encoder, and reported a few ablation values for K and the loss weights, the image claim could be rescued. As it stands, the evidence supports the text-encoder improvement but not the image-encoder one. The paper would be of interest to people working on CLIP fine-tuning or LLM-to-vision distillation, but I wouldn't build on the image result.\n\nMy recommendation: I would not send this to a serious referee in its current form; the confound is load-bearing and the missing hyperparameters and code make it hard to evaluate. It is a reasonable arXiv note, not a paper-ready-for-review yet.","headline":"The text-encoder distillation clearly helps on the reconstruction task, but the image-encoder claim is confounded by the jointly modified text encoder, and the paper doesn't supply the one control that would settle it.","tokens_in":6894,"tokens_out":2844,"would_cite":false,"duration_ms":27693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that distilling Llama 2's text embeddings into CLIP, combined with k-means concept labels and contrastive alignment, improves both the text encoder's exact-match reconstruction and the image encoder's attribute…","keywords":["knowledge distillation","CLIP","Llama 2","concept learning","k-means clustering","vision-language models","attribute recognition","contrastive learning"],"falsifier":"Run a random-label control: retrain Knowledge-CLIP with the same k-means concept labels randomly permuted across captions, keeping all other hyperparameters fixed. If AWA2 and CUB top-1 accuracy stay near 56.7 and 78.8, the concept-learning loss is not carrying the image-encoder gain and the central claim of transferred conceptual knowledge is falsified; if accuracy drops toward CLIP's 55.8 and 78.3 baselines, the labels are doing real work.","tokens_in":5970,"feed_emoji":"🧠","tokens_out":12681,"duration_ms":108639,"temperature":0.7,"pith_summary":"This paper claims that a CLIP model can be made to encode more of the conceptual content hidden in captions by distilling the text-embedding space of a large language model into it. The proposed Knowledge-CLIP couples three losses: matching CLIP's text embeddings to Llama 2 embeddings through a learned linear projection, learning soft concept labels obtained by k-means clustering Llama 2 captions, and keeping the original contrastive image-text alignment. On a 500K-sample pre-training subset, the text encoder's exact-match reconstruction on CC3M rises from 35.8 to 44.4 percent, partway between CLIP and Llama 2. On attribute recognition, image accuracy rises slightly on AWA2 (55.8 to 56.7 top-1) and CUB (78.3 to 78.8). The paper's point is that external knowledge from an LLM, transferred this way, improves both sides of the CLIP embedding.","feed_headline":"Distilling Llama 2 into CLIP lifts text exact match from 35.8 to 44.4","feed_subtitle":"Image attribute accuracy also rises on AWA2 and CUB, evidence that LLM labels can teach visual concepts.","key_machinery":"The load-bearing mechanism is the combined objective $L = \\alpha L_{\\text{emb}} + \\beta L_{\\text{conc}} + \\gamma L_{\\text{cont}}$. $L_{\\text{emb}}$ is a mean-squared-error loss that pushes a learned linear projection of CLIP's text embedding toward Llama 2's output embedding; $L_{\\text{conc}}$ is a cross-entropy loss on a frozen classifier that predicts k-means soft concept labels from the image embedding; $L_{\\text{cont}}$ is CLIP's original symmetric contrastive loss. The conceptual bridge is the k-means step: clustering Llama 2 caption embeddings offline produces a discrete label for each caption-image pair, and the classifier then lets the image encoder be trained to reproduce those labels from pixels.","core_discovery":"Knowledge-CLIP is a fine-tuning recipe for CLIP built from three simultaneous objectives: text embedding distillation, concept learning, and contrastive learning. The distillation loss trains CLIP's text encoder, after a learnable linear projector, to match Llama 2's output embeddings with mean-squared error, transferring the LLM's more uniform and attribute-rich embedding geometry. The concept-learning loss first runs offline k-means over Llama 2 caption embeddings to produce soft concept labels for each caption-image pair, trains a classifier on those labels, then freezes it and trains the image encoder to predict the labels from images. The contrastive loss continues to align text and image embeddings. With equal weights on distillation and contrastive losses and a small weight on the concept loss, the paper reports text exact match improving from 35.8 to 44.4 on CC3M and image attribute top-1 improving from 55.8 to 56.7 on AWA2 and 78.3 to 78.8 on CUB, and concludes that both CLIP encoders are improved.","pith_inferences":["Ablating each of the three losses would reveal how much of the text gain comes from distillation versus contrastive alignment; the paper only reports the combined recipe.","A cluster-quality measurement on all 200 CUB classes, not one class, would test whether the k-means labels encode attributes; the paper's visualization is suggestive but not quantitative.","If the labels are valid, the same pipeline could inject LLM-derived attribute labels into other vision-language encoders or build zero-shot attribute classifiers without human annotation.","The small image-side gains relative to the text-side gain suggest the image encoder benefits mostly indirectly, through the contrastive loss, from an improved text embedding."],"forward_implications":["A CLIP text encoder can be upgraded without changing its architecture by distilling an LLM embedding space through a learned linear projection.","The offline k-means labeling makes the concept-loss branch applicable to any caption set without manual attribute annotations.","The image encoder's attribute accuracy should improve whenever the same three-loss recipe is applied to sufficiently large paired data, with the gain concentrated on attributes the LLM embeddings already separate.","Llama 2's more uniform embedding distribution is the property that lets the distilled text encoder represent finer sentence distinctions than CLIP's original text encoder."],"supporting_citations":[{"why":"Provides the base CLIP encoders and the contrastive objective reused as $L_{\\text{cont}}$.","marker":"(Radford et al., 2021)"},{"why":"Supplies Llama 2, the teacher whose text embeddings drive both the distillation loss and the k-means concept labels.","marker":"(Touvron et al., 2023)"},{"why":"Supplies the k-means algorithm that converts Llama 2 caption embeddings into the soft concept labels.","marker":"(Hartigan and Wong, 1979)"},{"why":"Basis for the mean-squared-error embedding-distillation objective, adopted here as $L_{\\text{emb}}$.","marker":"(Jiao et al., 2019)"},{"why":"Defines the CC3M exact-match protocol and the text-encoder compositionality bottleneck the paper targets.","marker":"(Kamath et al., 2023)"},{"why":"Provides CUB, one of the two attribute datasets used to score image-encoder concept knowledge.","marker":"(Wah et al., 2011)"},{"why":"Provides AWA2, the other attribute dataset used to score image-encoder concept knowledge.","marker":"(Xian et al., 2020)"}],"fun_headline_variants":["Llama 2 lifts CLIP text exact match to 44.4","Knowledge-CLIP: Distilling Llama 2 into CLIP","CLIP text match jumps to 44.4 with LLM distillation","Improve CLIP with Llama 2: text and image gains","CLIP learns concepts from Llama 2's soft labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The concept-learning branch works only if the k-means clusters of Llama 2 caption embeddings correspond to meaningful semantic attributes such as color, shape, and action; the paper's evidence is a visualization of 50 samples from a single CUB class.","fun_headline_variants_meta":{"raw":{"variants":["Llama 2 lifts CLIP text exact match to 44.4","Knowledge-CLIP: Distilling Llama 2 into CLIP","CLIP text match jumps to 44.4 with LLM distillation","Improve CLIP with Llama 2: text and image gains","CLIP learns concepts from Llama 2's soft labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3124,"prompt_tokens":953,"completion_tokens":2171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2076}},"tokens_in":569,"tokens_out":2171,"duration_ms":16080,"temperature":1.0,"reasoning_tokens":2076,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:19:02.522458+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a random-label control: retrain Knowledge-CLIP with the same k-means concept labels randomly permuted across captions, keeping all other hyperparameters fixed. If AWA2 and CUB top-1 accuracy stay near 56.7 and 78.8, the concept-learning loss is not carrying the image-encoder gain and the central claim of transferred conceptual knowledge is falsified; if accuracy drops toward CLIP's 55.8 and 78.3 baselines, the labels are doing real work.","supporting_citations":[{"cited_title":"Belongie","cited_arxiv_id":null,"evidence_quote":"Provides CUB, one of the two attribute datasets used to score image-encoder concept knowledge."}],"review_version":1}