{"id":"af743990-5ada-42e4-9932-5e26f94e2077","arxiv_id":"2412.11663","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning CLIP's image encoder with an auxiliary loss that aligns image embeddings to per-class mean text embeddings from LMM-generated descriptions improves test accuracy by 0.6 to 1.0 points on three action and event datasets.","lead":"This paper proposes a small addition to CLIP fine-tuning: pull the image features toward text descriptions generated by a multimodal AI during training. The authors report consistent accuracy gains on three action and event classification datasets, but the improvements are under one percentage point and not tested for statistical significance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"If Eq. 1's class means include test images, the auxiliary loss leaks test-set information; the paper never specifies the split used to construct K_l, so the reported gains may be inflated.","rationale":"The reader listed 'no explicit statement that the class means are computed from the training split only' as one of several weaknesses and returned a CONDITIONAL verdict. My stress test isolates this as the single most load-bearing concern: if the class prototypes include test-set descriptions, the auxiliary loss is a form of test-set leakage and the central empirical claim is invalidated. The reader did not frame it as the dominant issue, so my agreement is partial rather than full. Because the reader already conditioned acceptance on addressing such ambiguities, the verdict does not need to change: it should remain conditional until the authors clarify the split used in Eq. 1 and, ideally, release code or rerun the experiments with training-only prototypes. If the leakage is confirmed, the paper would need rejection or major reanalysis; if the authors provide training-only results that match Table 1, the concern is resolved. The proposed concrete test directly settles this by comparing the two possible constructions of K_l.","tokens_in":5575,"tokens_out":3036,"duration_ms":31476,"concrete_test":"Ask the authors for the exact construction of K_l, or inspect the released code; then re-run the full protocol twice: (i) compute C_l using only the training split, and (ii) compute C_l using the full dataset as Eq. 1 currently appears to allow. If variant (i) reproduces the Table 1 gains within about 0.1 point, leakage is not the explanation. If variant (i) loses the reported margin over the baseline, the headline comparison is confounded by test-set information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the LMM-based auxiliary objective (Eq. 2) improves classification generalization. That claim is valid only if the class prototypes C_l in Eq. 1 are computed from the training split alone. Section 3 first describes generating semantic descriptions for 'the images of the dataset,' and Eq. 1 defines K_l as the set of text embeddings 'belonging to class l' without specifying whether test images are included. In the standard experimental setup, images are split after the MiniGPT-4 descriptions are generated, so K_l may naturally contain test-sample descriptions. If it does, each training image is pulled toward a class mean that encodes semantic content from the test images of that class. The regularization therefore injects transductive information from the test set into training, making the comparison against the no-regularization baseline unfair. The reported gains of 0.635, 1.007, and 0.765 points on UCF-101, ERA, and BAR could disappear once the leakage is removed. The paper provides neither code nor an explicit statement that only training samples were used, so the ambiguity cannot be resolved from the manuscript alone. This is the most load-bearing concern because it directly affects the validity of the headline experimental evidence, whereas the lack of error bars and alpha sensitivity are secondary reporting issues.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an auxiliary regularization loss for fine-tuning CLIP's image encoder on image classification. MiniGPT-4 is prompted to produce 10 semantic descriptions per image; the frozen CLIP text encoder embeds these descriptions, and per-class mean embeddings are computed. A fully connected classification head is attached to the CLIP image encoder, and training minimizes Eq. (3), which is the cross-entropy loss plus an α-weighted squared Euclidean distance between each image embedding and its class's mean textual embedding. Experiments on UCF-101, ERA, and BAR report test-accuracy gains of 0.635, 1.007, and 0.765 percentage points over a no-regularization finetuning baseline.","tokens_in":5835,"tokens_out":3897,"duration_ms":39245,"significance":"If the reported gains are robust, the proposal is a clean and practical contribution: the formulation in Eqs. (1)-(3) is simple, internally consistent, and it can be dropped into existing CLIP finetuning pipelines without modifying the LMM or the CLIP text encoder. The paper also compares against several relevant CLIP-based prior works, including an LMM-based concatenation baseline. The main limitations are that the empirical evidence rests on single-run results with no uncertainty estimates, no code is released, and the construction of the class-mean targets leaves a potentially load-bearing ambiguity about whether test-set information is used. These issues need to be addressed before the central claim can be considered fully supported.","major_comments":[{"comment":"The set K_l in Eq. (1) is never explicitly defined as containing only training-split descriptions. The text says the authors 'use MiniGPT-4 to extract semantic descriptions for the images of the dataset,' and Eq. (1) defines K_l as 'the set of text embeddings belonging to the class l' without specifying a split. If descriptions for test images are included in K_l, then each class mean C_l encodes semantic content from the test set, and the auxiliary loss in Eq. (2) injects transductive information into training. The comparison against the no-regularization baseline would then be unfair, and the reported gains could be inflated. This is the most load-bearing concern because it directly affects the validity of the headline results, and the manuscript includes neither an explicit training-only statement nor code to resolve the ambiguity.","section":"Section 3, Eq. (1)"},{"comment":"All experiments are reported as single runs with no error bars, no multiple seeds, and no significance tests. The reported gains are 0.635, 1.007, and 0.765 percentage points on UCF-101, ERA, and BAR, respectively; these differences are small relative to typical run-to-run variation when fine-tuning a model such as CLIP. Without repeated runs and variance reporting, the central claim that the regularization improves accuracy is not yet strongly supported. Please provide multi-seed results with mean and standard deviation, or an equivalent statistical comparison.","section":"Table 1"}],"minor_comments":[{"comment":"The notation 'min_{x_i} J_reg' is formally awkward, since x_i is not a free variable but the output of the image encoder f(I_i; θ); the minimization should be expressed over the trainable parameters θ.","section":"Section 3, Eq. (2)"},{"comment":"The statement 'we train only the output of image encoder and the fully connected layer' is ambiguous: please clarify whether only the final transformer block of the image encoder is updated, or a projection/head layer, and specify the exact set of trainable parameters.","section":"Section 3, Implementation Details"},{"comment":"No sensitivity analysis is provided for α in Eq. (3) or for the number of descriptions per image (fixed at 10). A short ablation would greatly improve confidence that the reported improvement is not specific to the single chosen configuration.","section":"Section 4.3"},{"comment":"Reproducibility would benefit from details about MiniGPT-4 inference, such as decoding temperature, maximum token length, and the exact prompt template, since the semantic descriptions are the only data-dependent input to the proposed target computation.","section":"Section 4.2"},{"comment":"The figure panels do not include a visible legend or axis labels in the provided version; please make the baseline and proposed-method curves clearly identifiable.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the train/test ambiguity in the construction of the mean semantic class descriptions. If the authors can confirm that only training-split descriptions are used and provide multi-seed results with code, the paper would be close to acceptable. The self-citations to related prior work by the same group are relevant and not gratuitous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible, simple fine-tuning regularizer for CLIP with consistent but small gains, and one unresolved data-split question that could change the headline numbers. Worth a round of revision, not a desk reject.\n\nThe actual new piece is Eq. 2: an auxiliary Euclidean loss that pulls the CLIP image encoder's output embeddings toward per-class mean text embeddings derived from MiniGPT-4 descriptions, during fine-tuning. That's distinct from the authors' prior work [14], which concatenated image and text embeddings for a linear classifier. The formulation is clean, the writing is clear, and the gains are consistent: +0.64, +1.01, +0.77 on UCF-101, ERA, BAR. The baseline is a proper same-setup fine-tune, and the comparison against [14] and other CLIP adapters is useful.\n\nThe soft spots are real. The biggest one is the construction of the class prototypes in Eq. 1. The paper says descriptions are extracted for 'the images of the dataset' and never states whether K_l is restricted to training images. If test images are included, each training embedding is pulled toward a class mean that encodes test-set content, and the comparison against the no-regularization baseline is no longer a fair generalization test. The reported gains could be transductive inflation. This is not a hypothetical nit; the manuscript text as written does not rule it out, and no code is provided to check. A single sentence plus released code would fix it.\n\nSecondary issues: Table 1 reports single runs with no error bars or significance tests, and the gains are around one point, which is within plausible run-to-run variance for fine-tuning. There is no ablation on alpha or number of descriptions per image. These are addressable.\n\nThe citation pattern is fine — self-citation to [14] and [15] is directly relevant and not excessive. The math is simple and internally consistent.\n\nWho is this for? Practitioners fine-tuning CLIP on small-to-medium action/event recognition datasets, and researchers working on LMM-guided representation learning. It's a workshop-to-conference quality idea with a real but unverified claim. I'd send it to review, asking the authors to clarify the split and share code, but I would not rely on the magnitude of the gains until that's resolved.","headline":"A sensible CLIP fine-tuning regularizer with consistent but small gains, undermined by an unresolved train/test ambiguity in how the class prototypes are built.","tokens_in":6344,"tokens_out":1816,"would_cite":false,"duration_ms":16095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that regularizing CLIP image embeddings toward LMM-generated mean semantic class descriptions during fine-tuning yields consistent accuracy gains on three classification benchmarks.","keywords":["CLIP","LMM","MiniGPT-4","regularization","image classification","semantic embeddings","vision-language models","fine-tuning"],"falsifier":"Run the same fine-tuning protocol with $\\alpha=0$ (plain cross-entropy) and with $\\alpha=10^{-2}$, on a dataset where the LMM descriptions are visibly poor or generic; if the gain disappears or becomes negative, the claim that the LMM target is beneficial fails. A sharper check: replace the LMM-generated class means with per-class mean image embeddings of the training set; if performance matches or beats the proposed method, the text-derived targets are not essential and the improvement comes from any class-mean anchor.","tokens_in":5392,"feed_emoji":"📈","tokens_out":5026,"duration_ms":41890,"temperature":0.7,"pith_summary":"This paper proposes a plug-in regularization for fine-tuning CLIP's image encoder on classification tasks. The idea is to first ask a large multimodal model, MiniGPT-4, to produce ten semantic descriptions of every training image, embed those descriptions with CLIP's frozen text encoder, and average them per class to form a mean semantic class description. During fine-tuning, the image embeddings are pulled toward their class's mean description by an added Euclidean-distance loss, alongside the standard cross-entropy classification loss. The paper reports consistent accuracy improvements over plain fine-tuning on UCF-101, ERA, and BAR, with gains between 0.6 and 1.0 percentage points. The appeal is that the regularization requires no extra computation or text input at test time.","feed_headline":"LMM text anchors lift CLIP accuracy by up to 1 point","feed_subtitle":"Averaged LMM descriptions as training targets improve CLIP fine-tuning on three benchmarks.","key_machinery":"The key objects are the mean semantic class description $C_l$ computed by Eq. (1): the average of CLIP text embeddings of the ten MiniGPT-4-generated descriptions of all training images in class $l$. The regularization loss $J_{reg}$ in Eq. (2) then measures the squared Euclidean distance between an image embedding $x_i = f(I_i)$ and its class mean $C_l$, and the total loss $J_{total} = J_{ce} + \\alpha J_{reg}$ with $\\alpha = 10^{-2}$ combines it with cross-entropy. Only the last layers of the image encoder and the new classification head are trained, so the method is cheap. The mechanism is that the class-mean text vectors act as stationary anchors that compress intra-class spread in the shared CLIP embedding space, which the classification head can then exploit.","core_discovery":"The central claim is that the auxiliary regularization objective of Eq. (2) — minimizing the Euclidean distance between the image encoder's output embedding and the LMM-generated mean semantic class description of the sample's class — produces image embeddings with enhanced discrimination ability. Because CLIP was pretrained to place images and text in a shared embedding space, the text embeddings of the LMM descriptions provide a meaningful, class-specific target that pulls same-class images closer together while keeping the classification head free to separate classes. The paper shows that on UCF-101, ERA, and BAR, adding this objective to the cross-entropy loss improves test accuracy from 91.567 to 92.202, from 85.621 to 86.628, and from 95.565 to 96.330 respectively, outperforming earlier CLIP-based approaches it compares against.","pith_inferences":["The paper does not ablate the number of descriptions (fixed at 10) or the weight $\\alpha$ (fixed at $10^{-2}$); a natural extension is a sensitivity study, and the optimal $\\alpha$ may vary with dataset size and class count.","Because the class means are computed from the training split, the method is a prototype-based regularizer; replacing LMM text with other sources of class prototypes (e.g., caption templates or word embeddings) would test whether the LMM is essential or merely a convenient generator.","The gains might grow on larger or more fine-grained datasets where intra-class variation is high and the mean text description provides a stronger prior; conversely, on datasets with ambiguous images the LMM descriptions could be noisy.","A testable extension: apply the same regularization to the text encoder or to a symmetric loss that also pulls text embeddings toward image class means, which the paper does not explore."],"forward_implications":["If the result holds, any CLIP fine-tuning pipeline can gain roughly one accuracy point by adding this single auxiliary loss, with no extra test-time cost.","The method suggests a general recipe: use a frozen text encoder to convert LMM knowledge into embedding-space targets for vision encoders.","The approach composes with other CLIP adaptations such as prompt learning or adapters, since it only modifies the training objective.","The reported state-of-the-art comparison indicates that the regularization can outperform concatenating image and text embeddings for the linear classifier, as in the prior work [14]."],"supporting_citations":[{"why":"Supplies the CLIP vision-language model with its image and text encoders, the base architecture that the paper adapts and fine-tunes.","marker":"[1]"},{"why":"MiniGPT-4 is the LMM used to generate the ten semantic descriptions per image that seed the regularization targets.","marker":"[6]"},{"why":"Prior work by the same authors that concatenates image and LMM text embeddings for classification; the proposed method is positioned as a more effective way to exploit LMM knowledge.","marker":"[14]"},{"why":"LaBo is a CLIP-plus-LLM baseline that generates class concepts with GPT-3; the paper compares against it as a state-of-the-art CLIP-based approach.","marker":"[10]"},{"why":"CLIP-A-self uses GPT-4 visual descriptions to adapt CLIP; it serves as a comparison method on UCF-101.","marker":"[12]"},{"why":"UCF-101 is one of the three evaluation datasets whose training and test splits define the reported accuracy numbers.","marker":"[16]"},{"why":"ERA is the aerial event recognition dataset used in the experimental validation.","marker":"[17]"},{"why":"BAR is the biased action recognition dataset used in the experimental validation.","marker":"[18]"}],"fun_headline_variants":["LMM text anchors lift CLIP accuracy by up to 1 point","Semantic text targets from LMM sharpen CLIP embeddings for classification","CLIP + LMM regularization boosts accuracy up to 1 point on three sets","Use LMM descriptions as regularizers to improve CLIP image classification","Text-guided CLIP fine-tuning gains up to 1 point across benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LMM-generated semantic descriptions, once embedded by CLIP's frozen text encoder and averaged per class, form training targets that genuinely improve class separation of image embeddings; if those descriptions were noisy, non-discriminative, or poorly aligned with the image embedding space, the regularization term could pull embeddings toward unhelpful locations or add noise.","fun_headline_variants_meta":{"raw":{"variants":["LMM text anchors lift CLIP accuracy by up to 1 point","Semantic text targets from LMM sharpen CLIP embeddings for classification","CLIP + LMM regularization boosts accuracy up to 1 point on three sets","Use LMM descriptions as regularizers to improve CLIP image classification","Text-guided CLIP fine-tuning gains up to 1 point across benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3761,"prompt_tokens":880,"completion_tokens":2881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2785}},"tokens_in":496,"tokens_out":2881,"duration_ms":20340,"temperature":1.0,"reasoning_tokens":2785,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:42:12.887180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning protocol with $\\alpha=0$ (plain cross-entropy) and with $\\alpha=10^{-2}$, on a dataset where the LMM descriptions are visibly poor or generic; if the gain disappears or becomes negative, the claim that the LMM target is beneficial fails. A sharper check: replace the LMM-generated class means with per-class mean image embeddings of the training set; if performance matches or beats the proposed method, the text-derived targets are not essential and the improvement comes from any class-mean anchor.","supporting_citations":[{"cited_title":"Exploiting lmm-based knowledge for image classification tasks","cited_arxiv_id":null,"evidence_quote":"Prior work by the same authors that concatenates image and LMM text embeddings for classification; the proposed method is positioned as a more effective way to exploit LMM knowledge."},{"cited_title":"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification","cited_arxiv_id":null,"evidence_quote":"LaBo is a CLIP-plus-LLM baseline that generates class concepts with GPT-3; the paper compares against it as a state-of-the-art CLIP-based approach."},{"cited_title":"En- hancing clip with gpt-4: Harnessing visual descriptions as prompts","cited_arxiv_id":null,"evidence_quote":"CLIP-A-self uses GPT-4 visual descriptions to adapt CLIP; it serves as a comparison method on UCF-101."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ERA is the aerial event recognition dataset used in the experimental validation."},{"cited_title":"Learning from failure: De-biasing classifier from biased classifier","cited_arxiv_id":null,"evidence_quote":"BAR is the biased action recognition dataset used in the experimental validation."}],"review_version":1}