{"id":"eb0f150b-18ca-44e5-80c0-10eefe28c326","arxiv_id":"2412.08176","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Local visual features cached from CLIP's image branch can refine text prompts, improving few-shot classification without external LLM descriptions.","lead":"TextRefiner is a plug-in module that improves vision-language model prompt tuning by caching fine-grained visual patterns from the image encoder and folding them into text prompts. It lifts CoOp's average accuracy on 11 image classification benchmarks from 71.66% to 76.94% while keeping inference fast and avoiding external language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline CoOp gain is confounded by a 20x training-epoch mismatch: CoOp+TextRefiner is trained for 10 epochs while the CoOp baseline uses 200, so the reported +5.3 HM may reflect early stopping rather than the cache.","rationale":"The reader's stated weakest assumption is that CLIP patch tokens carry class-discriminative fine-grained attributes that survive clustering into a shared cache. That is a plausible design risk, but the paper's own reported numbers are the primary evidence for the central claim, and those numbers are vulnerable to a more direct experimental confound: the CoOp baseline is trained for 200 epochs while CoOp+TextRefiner is trained for only 10. The reader did flag 'unequal training budgets' in the rationale, so this is a partial agreement rather than a disagreement, but the reader did not make it the central load-bearing concern. The epoch mismatch is especially damaging because the direction of the observed changes (lower base, higher novel) is exactly what one expects from reduced training in prompt-learning settings, even without any architectural change. Table 3's ablation compares against the original CoOp row, so the component analysis inherits the same confound. The proposed check—running CoOp at 10 epochs and CoOp+TextRefiner at 200 epochs—would settle whether the cache, feature aggregation, and losses contribute anything beyond early stopping. The recommendation remains CONDITIONAL acceptance, since the method may well be useful; the condition is that the headline comparison needs a matched training schedule and variance estimates before the improvement magnitude can be trusted.","tokens_in":13483,"tokens_out":6153,"duration_ms":64643,"concrete_test":"Retrain CoOp with the same 10-epoch schedule used for CoOp+TextRefiner (same batch size, learning rate, optimizer, and soft-token length) on all 11 base-to-novel benchmarks and report base, novel, and harmonic mean. Then retrain CoOp+TextRefiner for the full 200-epoch schedule. If 10-epoch CoOp already closes most of the gap, or 200-epoch CoOp+TextRefiner no longer beats 200-epoch CoOp, the headline improvement is confounded by training schedule. Ideally report three random seeds with mean±std for both schedules.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is not the patch-token assumption but the experimental protocol used for the headline comparison. In the Implementation details section, the authors state that all settings match the original papers 'except for CoOp, where the number of epochs was reduced from 200 to 10.' The strongest claimed result is 'improves CoOp from 71.66 to 76.94' (Table 1), so the comparison is between a 200-epoch CoOp baseline and a 10-epoch CoOp+TextRefiner variant. Fewer epochs is a strong implicit regularizer in few-shot prompt tuning; it typically lowers base-class accuracy while raising novel-class accuracy, which is exactly the direction of the reported shift (CoOp w/TextRefiner base drops from 82.69 to 79.74 while novel rises from 63.22 to 74.32). Table 3's ImageNet ablation also uses the full CoOp row as the empty-component baseline, so the component gains are not isolated from the schedule change. Without a CoOp trained for 10 epochs, or a CoOp+TextRefiner trained for 200 epochs, the claimed +5.28 harmonic-mean improvement cannot be attributed to TextRefiner's cache and alignment modules rather than to early stopping. This concern is concrete and directly testable; it does not require questioning the authors' good faith.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TextRefiner, a plug-and-play module that refines the text prompts of vision-language model (VLM) prompt-tuning methods using internal visual features. A local cache stores clusters of image patch tokens, a feature aggregation module fuses the cached visual descriptions into class text embeddings, and a feature alignment module (a 2-layer MLP) maps local tokens into the text embedding space. Training adds a semantic loss and a regularization loss to the standard contrastive loss. The authors evaluate on base-to-novel generalization across 11 datasets, cross-domain generalization on ImageNet variants, and inference efficiency, reporting consistent gains when TextRefiner is attached to CoOp, MaPLe, and PromptKD, with PromptKD w/TextRefiner achieving the best average harmonic mean.","tokens_in":13793,"tokens_out":3601,"duration_ms":40040,"significance":"If the reported results are reproducible, TextRefiner is a useful contribution: it improves the base-to-novel generalization of several prompt-tuning baselines without external LLM knowledge or heavy inference overhead, and it is orthogonal to the underlying prompt-tuning method. The paper ships code, reports a large-scale evaluation, and explicitly adopts the standard base-to-novel protocol, which is a genuine generalization check. The method's motivation (using internal local visual features to refine text prompts) is plausible, and the efficiency numbers are striking. However, the headline CoOp comparison is marred by an unequal training schedule, and several method details need clarification before the central claim can be accepted.","major_comments":[{"comment":"The strongest claimed result, \"improves CoOp from 71.66% to 76.94%,\" is confounded by training budget. The Implementation details section states that all settings match the original papers \"except for CoOp, where the number of epochs was reduced from 200 to 10.\" Thus Table 1 compares a 200-epoch CoOp baseline with a 10-epoch CoOp+TextRefiner variant. Since fewer epochs is itself a strong regularizer in few-shot prompt tuning, the observed shift (base accuracy dropping from 82.69 to 79.74 while novel accuracy rises from 63.22 to 74.32) is exactly what one would expect from early stopping, not necessarily from the proposed cache, aggregation, or alignment modules. The authors should report a 10-epoch CoOp baseline, or a 200-epoch CoOp+TextRefiner variant, or otherwise isolate the effect of TextRefiner from the schedule change.","section":"Implementation details / Table 1"},{"comment":"The ablation study in Table 3 uses the original CoOp row as the empty-component baseline, but the TextRefiner rows are trained for 10 epochs. Therefore the incremental gains attributed to the local cache, semantic loss, and regularization loss are not cleanly isolated from the change in training epochs. A matched-schedule baseline is needed before the component contributions can be interpreted. In addition, none of the tables report multiple seeds or error bars; given the small margins on some datasets (e.g., Caltech101 and OxfordPets rows), variance estimates are important for assessing whether the gains are significant.","section":"Table 3 / Ablations"},{"comment":"The text repeatedly states that the local cache stores fine-grained information \"in a class-wise manner,\" but the update rule in Eqs. (2)-(4) is a global cache: every local token from every class is assigned to one of M shared entries, with no class label or class-conditional term in the cosine-similarity assignment or the momentum update. The subsequent querying with class text embeddings (Eqs. (5)-(6)) does not make the stored entries class-specific. The method may work perfectly well with a shared attribute cache, but the paper should either clarify what \"class-wise\" means, or replace the description with the actual global-cache mechanism, since the current text misdescribes the algorithm.","section":"Local Cache / Eqs. (2)-(4)"}],"minor_comments":[{"comment":"There is a notation mismatch: the Implementation details mention \"momentum α and the fusion factor β,\" but the method defines the momentum coefficient as γ in Eq. (4) and the aggregation coefficient as α in Eq. (7). Please align the notation.","section":"Implementation details / Method notation"},{"comment":"The semantic loss in Eq. (10) refers to \"top-k transformed local features with attention scores,\" but no attention scores are defined in the method. Please specify how the top-k local features are selected and where the attention scores come from.","section":"Feature Alignment / Eq. (10)"},{"comment":"The paper does not state which layer of the ViT image encoder provides the local tokens V. Since the whole method depends on these tokens carrying fine-grained visual concepts, please specify the layer and, ideally, provide an analysis of how the choice of layer affects results.","section":"Method / local tokens source"},{"comment":"There is a typo: \"relesed\" should be \"released.\" Similar typos appear elsewhere (e.g., \"the transfer\" in the preliminaries), so a proofreading pass is recommended.","section":"Abstract"},{"comment":"The FPS comparison in Figure 3 is reported only for ImageNet and without a precise protocol description (batch size, input resolution, hardware details beyond \"one single A800 GPU\"). Please state the measurement protocol so the efficiency claim is reproducible.","section":"Efficiency comparison / Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The epoch mismatch for CoOp is openly stated by the authors in the Implementation details, so it is not a hidden flaw. The fix is straightforward: rerun CoOp with the same 10-epoch budget as CoOp+TextRefiner, or run TextRefiner for 200 epochs, and report both. The PromptKD w/TextRefiner results are less affected by this issue and could support the method even if the CoOp headline changes. I see no grounds for rejection, but the current presentation overstates the causal contribution of TextRefiner to the CoOp improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is solid and worth engaging with: instead of calling an external LLM to generate class descriptions, TextRefiner pulls fine-grained visual concepts from CLIP's own image-branch patch tokens, stores them in a local cache, and refines the text embeddings with an aggregation step and a small MLP alignment head. That combination is genuinely new in the cited literature, and the plug-and-play design is attractive. The paper also does the right thing by testing on the base-to-novel protocol, which is a real generalization check, and the code is released.\n\nWhere the paper gets into trouble is the headline comparison. The abstract and Table 1 claim TextRefiner improves CoOp from 71.66% to 76.94% on 11 benchmarks. But the implementation details say CoOp's training was cut from 200 epochs to 10. CoOp with fewer epochs is a stronger regularizer, and the observed shift (base accuracy drops, novel accuracy jumps) is exactly what early stopping would do. The ablation in Table 3 also uses the full 200-epoch CoOp as the empty-component baseline, so the component contributions are not isolated from the schedule change. This is a concrete, testable flaw, not a matter of authorial intent.\n\nThere are smaller soft spots too: no error bars are reported, and hyperparameters (lambda1, lambda2, alpha, M) are tuned without a stated held-out protocol. The assumption that patch tokens carry class-discriminative attributes is plausible and cited, but never directly measured on these datasets. None of these sink the method, but they do mean the claimed improvement magnitude is not trustworthy as presented.\n\nWhat does hold up? PromptKD + TextRefiner shows solid gains over PromptKD and beats LLaMP on harmonic mean, and since the authors say all other settings match the original papers, that comparison is less confounded. The mechanism is coherent and the inference overhead is genuinely low. This is a useful contribution for practitioners who want a cheap text-refinement option without LLM calls.\n\nMy recommendation: send this to peer review, but with a clear demand that the authors retrain CoOp for 10 epochs and CoOp+TextRefiner for 200, report variance across seeds, and specify how hyperparameters were chosen. With that cleanup, the paper could be a solid incremental contribution. Without it, the central claim is unreadable.","headline":"A promising plug-and-play idea for refining VLM text prompts, but the headline CoOp gain rides on an unfair epoch comparison.","tokens_in":14313,"tokens_out":1426,"would_cite":false,"duration_ms":17461,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Patch tokens refine text prompts, lifting CoOp by 5.28 points","keywords":["TextRefiner","prompt tuning","vision-language models","local cache","fine-grained visual features","CLIP","base-to-novel generalization","inference efficiency"],"falsifier":"A direct test is to take a trained TextRefiner and measure whether each cache entry's nearest patch tokens are class-consistent on a held-out split; if entries do not separate classes better than random, the mechanism is not doing what is claimed. A cheaper check is to drop the feature alignment MLP entirely and see whether the cache alone still improves CoOp.","tokens_in":13226,"feed_emoji":"🖼️","tokens_out":8338,"duration_ms":68428,"temperature":0.7,"pith_summary":"This paper claims that the fine-grained visual details needed to distinguish similar classes are already present in CLIP's image-branch patch tokens, and that a plug-and-play module can extract them to refine text prompts. If this is right, prompt-tuning methods can match or beat LLM-based description generation while keeping inference fast and needing no external knowledge. The authors show TextRefiner raising CoOp's harmonic mean of base and novel accuracy from 71.66% to 76.94% over 11 benchmarks, and PromptKD with TextRefiner reaching an 82.33 harmonic mean at 12,793 FPS.","feed_headline":"Patch tokens refine text prompts, lifting CoOp by 5.28 points","feed_subtitle":"Uses CLIP's own image patches as class descriptions, matching LLM-based methods at a fraction of the inference cost.","key_machinery":"The central object is the local cache module: a fixed-size storage $A \\in \\mathbb{R}^{M \\times d}$ whose entries are updated by cosine-similarity-weighted aggregation of patch tokens, so each entry condenses a recurring visual attribute. Feature aggregation retrieves attribute embeddings by softmax-matching each text embedding to cache entries and fusing them through a residual linear layer (Eq. 7). Feature alignment, a 2-layer MLP (Eq. 8), maps patch tokens into the text embedding space to close the modality gap, and two auxiliary losses, semantic and regularization, supervise the alignment.","core_discovery":"TextRefiner establishes that internal visual features, specifically the local patch tokens of a ViT image encoder, can serve as a source of class-level visual attributes for refining text embeddings in CLIP prompt tuning. The central claim is that this internal-knowledge route is as effective as external LLM descriptions while being nearly inference-free. During tuning, patch tokens are clustered into a local cache whose entries act as attribute priors; at inference these entries are matched to text embeddings and fused through a linear layer, with an MLP aligning local visual features into text space. The paper reports consistent gains across CoOp, MaPLe, and PromptKD, including large novel-class improvements.","pith_inferences":["A natural extension the paper does not test is whether the same local cache can be reused across datasets as a transferable visual-attribute vocabulary; if entries are class-agnostic texture or part detectors, a cache trained on one dataset should refine prompts on a held-out dataset.","The cache size $M$ is a sensitivity point: the paper's ablation shows accuracy peaks then falls as $M$ grows, suggesting that beyond a threshold the clustering fragments whole attributes; this could be turned into an automatic criterion for choosing $M$ from validation-set harmonic mean.","The feature alignment MLP may be doing much of the reported work, and the paper does not isolate whether gains come from the cache content or from the learned projection; replacing Eq. 8 with a fixed linear projection would separate these effects."],"forward_implications":["Any existing prompt-tuning method that keeps CLIP's text encoder can be upgraded by plugging in TextRefiner, since the cache operates on text output embeddings rather than requiring retraining the backbone.","Novel-class generalization improves without instance-wise conditioning: TextRefiner lifts CoOp's novel accuracy from 63.22% to 74.32%, exceeding CoCoOp's 71.69%.","Cross-domain robustness improves on ImageNet variants, with CoOp plus TextRefiner outperforming plain CoOp on ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.","The approach competes with LLM-based prompt refinement at far lower inference cost: PromptKD with TextRefiner runs at 12,793.26 FPS versus LLaMP's 1,473.46 FPS."],"supporting_citations":[{"why":"Supplies the CLIP backbone whose image and text encoders define the joint embedding space and the patch tokens used as internal visual knowledge.","marker":"Radford et al. 2021"},{"why":"CoOp is the primary baseline whose text prompts TextRefiner refines, providing the improvement from 71.66% to 76.94%.","marker":"Zhou et al. 2022b"},{"why":"CoCoOp is the instance-wise prompt-tuning baseline that TextRefiner outperforms while being faster.","marker":"Zhou et al. 2022a"},{"why":"PromptKD is the second baseline and, combined with TextRefiner, achieves the highest reported harmonic mean in the paper.","marker":"Li et al. 2024b"},{"why":"LLaMP is the external-LLM method that TextRefiner matches or beats at much higher FPS, setting the comparison target.","marker":"Zheng et al. 2024"},{"why":"Provides the evidence TextRefiner relies on that ViT intermediate local tokens capture fine-grained visual concepts like edges and textures.","marker":"Ghiasi et al. 2022"},{"why":"Supports the claim that intermediate visual features are interpretable and semantically meaningful for class descriptions.","marker":"Kim, Nam, and Ko 2022b"}],"fun_headline_variants":["Use CLIP's own patches to refine prompts, +5.28 on CoOp","Internal patch tokens refine prompts, matching LLM methods at lower cost","No LLM needed: refine text prompts using CLIP's own image patches","Patch tokens refine prompts, lifting CoOp by 5.28 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that CLIP image-branch patch tokens contain class-discriminative fine-grained attributes that can be clustered into a shared cache and mapped into text space by a shallow MLP; if that assumption fails, the cache stores noise and the refinement cannot help novel classes.","fun_headline_variants_meta":{"raw":{"variants":["Use CLIP's own patches to refine prompts, +5.28 on CoOp","Internal patch tokens refine prompts, matching LLM methods at lower cost","No LLM needed: refine text prompts using CLIP's own image patches","Patch tokens refine prompts, lifting CoOp by 5.28 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001058,"raw_usage":{"total_tokens":4436,"prompt_tokens":937,"completion_tokens":3499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3417}},"tokens_in":553,"tokens_out":3499,"duration_ms":25238,"temperature":1.0,"reasoning_tokens":3417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:07:26.345367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take a trained TextRefiner and measure whether each cache entry's nearest patch tokens are class-consistent on a held-out split; if entries do not separate classes better than random, the mechanism is not doing what is claimed. A cheaper check is to drop the feature alignment MLP entirely and see whether the cache alone still improves CoOp.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaMP is the external-LLM method that TextRefiner matches or beats at much higher FPS, setting the comparison target."}],"review_version":1}