Pith. sign in

REVIEW 4 major objections 4 minor 16 references

Enhancing Target-unspecific Tasks through a Features Matrix

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt learning for vision-language models can be rescued from overfitting by a Features Matrix of many hand-crafted prompts, a plug-in module that improves base-to-novel generalization.

desk verdict A plausible plug-in regularizer for prompt tuning, but the headline gains are not yet trustworthy because gamma and beta are selected on the same 11-dataset HM average used as the result and no seed variance is reported. read the letter →

arxiv 2505.03414 v5 pith:CY3644VO submitted 2025-05-06 cs.CV cs.CL

Fangming Cui , Yonggang Zhang , Xuan Wang , Xinmei Tian , Jun Yu This is my paper · ORCID
classification cs.CVcs.CL
keywords promptlearningvision-languagemodelsCLIPfeaturesmatrixbase-to-novelgeneralizationdomaincross-datasetcontrastiveregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets a failure mode of prompt learning: fine-tuning CLIP's prompt embeddings for few-shot base classes pushes performance up on those classes but degrades accuracy on novel, out-of-distribution, and cross-dataset targets, because the model forgets general pre-trained knowledge. To counter that, the authors propose Features Matrix (FM), a plug-in module that uses many hand-crafted prompt templates to build a matrix of pre-trained text features and adds a contrastive loss on the hardest ("unexpected") matches between those features and the tuning visual features. They claim FM is generic: attached to CoOp, CoCoOp, MaPLe, or PromptSRC it improves base-to-novel, domain, and cross-dataset generalization, and on the PromptSRC backbone it reaches state-of-the-art harmonic mean 81.32, above the easy-to-use DePT at 80.43.

What carries the argument

The Features Matrix (FM) is a fixed table of text features obtained by feeding 60 hand-crafted prompt templates (for example "a photo of a", "a picture of a", "a drawing of a") for every class through the frozen CLIP text encoder. For a given image, features of the true class are "designated", others are "non-designated"; the method ranks matches between the tuning visual feature and each matrix row, selects the low-$\beta$ designated and top-$\beta$ non-designated entries as "unexpected features", and applies a contrastive loss $L_{\mathrm{CL}} = -\log \frac{\exp(\cos(t_k, v_{\mathrm{tun}}))}{\exp(\cos(t_k, v_{\mathrm{tun}})) + \exp(\cos(t_{\hat{k}}, v_{\mathrm{tun}}))}$ with weight $\gamma$ added to the cross-entropy loss. This selection of hard matches is the component claimed to align tuning image features with overlooked pre-trained general knowledge, while the matrix itself provides the diversity of semantics that a single hand-crafted prompt lacks.

What would settle it

Run the same base-to-novel training with a control that draws the same number of features at random from the matrix, or that uses all 60 features with equal weight, and compare harmonic means; if the control matches the low-$\beta$/top-$\beta$ result, the hard-example selection is not the mechanism that carries the gain.

Watch

Extended reading notes

Core claim

The central claim is that the drop in target-unspecific performance comes from overfitting that makes the model forget general knowledge, and that a frozen Features Matrix can supply that knowledge back. The matrix is built by passing a set of 60 hand-crafted prompts, across all classes of a dataset, through the frozen CLIP text encoder. During training, for each visual feature the method scores all matrix entries, keeps the lowest-scoring designated-class features and the highest-scoring non-designated features ("unexpected features"), and optimizes a contrastive loss that aligns tuning visual features with those unexpected text features, alongside the standard cross-entropy prompt-learning loss. The authors argue that this "specifically aligning" of unexpected pre-trained features preserves general knowledge and therefore enhances target-unspecific tasks, and they report consistent gains across 11 datasets on base-to-novel, domain, and cross-dataset benchmarks, including surpassing DePT when added to PromptSRC.

Load-bearing premise

The method's improvement is attributed to selecting the lowest-scoring same-class and highest-scoring other-class text features as "unexpected features", but the experiments only vary how many features are selected and the loss weight, never comparing this selection to random or all-feature choices, so the benefit could come from the multi-prompt regularization alone.

Editorial extensions

If this is right

  • Attached to PromptSRC, FM raises the base-to-novel harmonic mean from 79.97 to 81.32 over 11 datasets, surpassing DePT (80.43).
  • The module also lifts domain generalization (PromptSRC average improves from 60.65 to 61.92 on ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R) and cross-dataset generalization (from 65.81 to 67.62).
  • FM works on textual prompting (CoOp, CoCoOp) and multi-modal prompting (MaPLe, PromptSRC) without adding learnable parameters, only extra cosine-similarity computation.
  • Ablations show a peak at $\beta = 5$ selected features and $\gamma = 0.1$ loss weight, with harmonic mean declining at larger values, consistent with the overfitting story.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If hard-feature selection is the active ingredient, FM is a form of text-side hard-example mining; a natural test would replace "unexpected features" with randomly chosen matrix features, holding the loss and count fixed.
  • The 60 hand-crafted prompts are fixed; extending the matrix with learned or class-specific templates would test whether the diversity of prompts or the selection rule drives the gain.
  • The selection rule operates only on text features; applying the same low-$\beta$/top-$\beta$ contrast to visual features from the frozen image encoder would show whether the mechanism is modality-specific.
  • The paper's own tables show that FM sometimes lowers base-class accuracy and source ImageNet accuracy and slows training, so a practitioner would likely weigh FM as a trade of target-specific accuracy for target-unspecific generalization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Features Matrix (FM), a plug-in regularization module for prompt learning in CLIP. From a frozen text encoder and a set of 60 hand-crafted prompt templates, it forms a matrix of text features; for each training image it selects the lowest-scoring designated and highest-scoring non-designated features as "unexpected" features and applies a contrastive loss to align them with the tuning visual features. This loss is added with weight gamma to the cross-entropy loss of a host method, making the module compatible with CoOp, CoCoOp, MaPLe, and PromptSRC. Experiments cover base-to-novel generalization, domain generalization, and cross-dataset generalization on 11 datasets plus ImageNet variants. The paper reports consistent average improvements over the host methods and over the DePT baseline, and claims state-of-the-art performance on target-unspecific tasks.

Significance. The core idea is simple and potentially useful: a frozen-feature regularizer that does not add learnable parameters and can be attached to several representative prompt-learning frameworks. The breadth of the evaluation, across three generalization settings and multiple backbones, is a strength, and the compute-cost table is helpful. However, the headline empirical claims are currently undermined by hyperparameter selection on the evaluation metric and by the absence of any seed-variance information; in addition, the proposed hard-example selection mechanism is not isolated experimentally. If additional controls confirm the gains, the method would be a worthwhile contribution to prompt learning for CLIP. In its present form, the evidence does not support the stated state-of-the-art claims.

major comments (4)
  1. [§4.1, §5.3, §5.4] Section 4.1 fixes gamma=0.1 and beta=5, but Sections 5.3 (Table 9) and 5.4 (Table 8) select these values by maximizing HM on the average of the same 11 datasets that form the headline results in Table 2. This is selection on the evaluation metric, so the reported gains over PromptSRC (81.32 vs 79.97) and over PromptSRC+DePT (81.32 vs 80.43) are upper-biased. Please provide a validation protocol, such as tuning on a held-out subset of datasets or reporting full sensitivity curves, and explicitly describe how the fixed hyperparameters were chosen.
  2. [Tables 2 and 3] No variance information is reported anywhere in the experimental section. All accuracy numbers appear to be from single runs, and several per-dataset differences are small relative to the usual seed-to-seed spread in this 16-shot benchmark; for example, Table 2(i) Food101 gives HM 91.45 for PromptSRC+Ours versus 91.22 for PromptSRC+DePT, and Table 2(l) DTD gives 72.07 versus 71.09. Without means and standard deviations over multiple seeds, the claimed improvements cannot be distinguished from run-to-run noise. Please report at least three seeds with standard deviations, and ideally a paired comparison test.
  3. [§3.2, Eq. (4), §5.4] The central mechanism, that selecting the lowest-scoring designated and highest-scoring non-designated features ('unexpected features') provides a better training signal, is not experimentally supported. Section 5.4 (Table 8) varies only the count beta, and Section 5.3 (Table 9) varies gamma; there is no control using randomly selected features or using all features from the matrix. Without such a control, the gains could be due entirely to the multi-prompt contrastive regularizer rather than to the proposed hard-example selection, so the stated mechanism remains unsupported.
  4. [§4.2] The statement that 'our method demonstrates significant improvements on all 11 datasets on HM' is not supported by Table 2 when the comparison includes +DePT. For example, CoOp+Ours is worse than CoOp+DePT on EuroSAT (75.13 vs 75.70), StanfordCars (75.58 vs 75.86), and Flowers102 (82.57 vs 83.08), and MaPLe+Ours is worse than MaPLe+DePT on EuroSAT (83.93 vs 84.36). Please clarify the exact comparison baseline to which 'all 11' refers and avoid the term 'significant' without a statistical test.
minor comments (4)
  1. [Table 1] The column headers 'Hand' and 'Features' are unclear; please explain whether 'Hand' refers to hand-crafted prompts and what 'Single' versus 'Matrix' means in the 'Features' column.
  2. [Eq. (4)] The notation for non-designated features (the index with a hat) is introduced informally; please define the index sets before Eq. (4) to avoid confusion with the designation used elsewhere.
  3. [References] Several references are duplicated or incomplete: Yao et al. 2023a and 2023b appear to be the same paper, and the PLOT reference cited as Chen et al. in Table 1 lacks full venue and year information. Please unify the bibliography.
  4. [§7] The limitations paragraph acknowledges slower training and lower source accuracy on ImageNet, but it does not quantify the slowdown beyond Table 11 or discuss the trade-off in the cross-dataset setting; please add concrete timing or throughput numbers and a brief discussion.

Circularity Check

0 steps flagged · score 1.0 of 10
full rationale

The derivation chain is self-contained. Equation (4) defines a contrastive loss between tuning visual features and fixed, frozen CLIP text features selected from a precomputed features matrix; Equation (6) adds this regularizer to the cross-entropy loss with a scalar weight. No term in L_CL is defined from the evaluation metric, and no fitted parameter is used to reconstruct the output. The 'unexpected features' selection (low-β designated, top-β non-designated) is a hard-example mining choice, not a quantity defined by the target result. The self-citations (Cui et al., 2024; 2025a; 2025b) appear in related-work lists only and are not load-bearing for the FM construction. The one legitimate concern is that γ and β are selected in Sections 5.3–5.4 by peaking HM on the same 11-dataset average that forms the headline result in Table 2(a), with no validation split or seed variance reported. This is a selection-on-test / statistical soundness issue rather than a circular reduction: the reported HM is not equivalent to the paper's inputs by construction, and the method's independent content—multi-prompt frozen-feature regularization—stands regardless of how the two scalars were chosen.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim that the FM improves generalization rests on the frozen CLIP feature matrix providing a meaningful regularization target and on the hard-example selection mechanism being the operative component. Two hyperparameters (beta and gamma) are tuned on the full benchmark average, and no code or seeds are provided, which limits the evidence quality.

free parameters (4)
  • beta (number of selected unexpected features) = 5
    Chosen by ablating average HM over the 11 datasets (Table 8); controls how many low-scoring designated and high-scoring non-designated features enter the contrastive loss.
  • gamma (weight of contrastive loss) = 0.1
    Chosen by ablating average HM over the 11 datasets (Table 9); scales L_CL in L_total = L_CE + gamma times L_CL.
  • embedding length of prompts = 4
    Selected by ablation (Table 7) as the peak HM; inherited from the PromptSRC-style setup.
  • learning depth (number of transformer layers tuned) = 9 (base-to-novel), 3 (domain and cross-dataset)
    Selected by ablation (Table 5); affects the degree of fine-tuning and interacts with the regularizer.
assumptions (3)
  • domain assumption Frozen CLIP text features obtained from hand-crafted prompts encode 'general knowledge' that regularizing toward preserves generalization.
    Motivation in Sections 1 and 3.1; the entire method assumes this fixed feature set is a useful target for alignment.
  • ad hoc to paper Selecting the hardest (lowest-scoring designated and highest-scoring non-designated) features provides a more useful training signal than random or all features.
    Enters at Eq. (4) in Section 3.2; no theory or control experiment (for example, random selection) supports the selection mechanism.
  • domain assumption The 11-dataset base-to-novel split and ImageNet-variant benchmarks measure 'target-unspecific generalization'.
    The experimental protocol in Section 4.1; the paper's significance claim depends on these benchmarks being meaningful measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Target-unspecific Tasks through a Features Matrix." pith.science (2026). https://pith.science/paper/CY3644VO

@misc{pith2026250503414,
  author       = {Pith},
  title        = {Pith review of: Enhancing Target-unspecific Tasks through a Features Matrix},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CY3644VO}},
  note         = {Machine review of arXiv:2505.03414}
}
read the original abstract

Recent developments in prompt learning of large Vision-Language Models (VLMs) have significantly improved performance in target-specific tasks. However, these prompting methods often struggle to tackle the target-unspecific or generalizable tasks effectively. It may be attributed to the fact that overfitting training causes the model to forget its general knowledge. The general knowledge has a strong promotion on target-unspecific tasks. To alleviate this issue, we propose a novel Features Matrix (FM) approach designed to enhance these models on target-unspecific tasks. Our method extracts and leverages general knowledge, shaping a Features Matrix (FM). Specifically, the FM captures the semantics of diverse inputs from a deep and fine perspective, preserving essential general knowledge, which mitigates the risk of overfitting. Representative evaluations demonstrate that: 1) the FM is compatible with existing frameworks as a generic and flexible module, and 2) the FM significantly showcases its effectiveness in enhancing target-unspecific tasks (base-to-novel generalization, domain generalization, and cross-dataset generalization), achieving state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2505.03414 by the authors.

Figure 1
Figure 1. Our method is orthogonal to representative architectures, such as MaPLe (M) and PromptSRC (S), surpassing the existing easy-to-use DePT (Zhang et al., 2024) by a significant margin. demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of representative textual prompting frameworks (CoOp and CoCoOp) and multi-modal prompting frameworks (MaPLe and PromptSRC). We propose a flexible and generic design, which is compatible with these representative architectures. In the figure, “snowflake pattern” represents parameter freezing, “flame pattern” represents learnable pattern, “Deep” represents learnable tokens embedded in several layers of t… view at source ↗
Figure 3
Figure 3. Illustration of our easy-to-use method. We propose a novel Features Matrix (FM) for enhancing target-unspecific tasks. Our method incorporates multiple hand-crafted prompts with classes to extract general knowledge as a pre-trained features matrix. Various generalization tasks across 11 datasets demonstrate that our method outperforms existing prompt learning methods. may be attributed to the fact that this single h… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages

  1. [5]

    An extremely simple algo- rithm for source domain reconstruction.IEEE Transac- tions on Cybernetics, 54(3):1921–1933,

    Fang, Z., Lu, J., and Zhang, G. An extremely simple algo- rithm for source domain reconstruction.IEEE Transac- tions on Cybernetics, 54(3):1921–1933,

    work page 1921

  2. [8]

    Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

    arXiv

  3. [10]

    Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models

    Shao, Z., Wang, M., Yu, Z., Pan, W., Yang, Y ., Wei, T., Zhang, H., Mao, N., Chen, W., and Yu, J. Growing a twig to accelerate large vision-language models.arXiv preprint arXiv:2503.14075,

    work page Pith review arXiv

  4. [14]

    Yao, H., Zhang, R., and Xu, C

    doi: 10.1109/TPAMI.2022.3185311. Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6757–6767, June 2023a. Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guided context optimiza...

    arXiv 2022

  5. [15]

    Tip-adapter: Training-free clip-adapter for better vision-language modeling.arXiv preprint arXiv:2111.03930,

    12 Enhancing Target-unspecific Tasks through a Features Matrix Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y ., and Li, H. Tip-adapter: Training-free clip-adapter for better vision-language modeling.arXiv preprint arXiv:2111.03930,

    arXiv

  6. [16]

    Rankadaptor: Hierarchical rank alloca- tion for efficient fine-tuning pruned llms via performance model

    Zhou, C., Han, S., Yang, L., Zhou, Y ., Cheng, X., Wang, Y ., and Li, H. Rankadaptor: Hierarchical rank alloca- tion for efficient fine-tuning pruned llms via performance model. InFindings of the Association for Computational Linguistics: NAACL 2025, pp. 5781–5795, 2025a. Zhou, C., Zhou, Y ., Zhang, S., Wang, Y ., and Liu, Z. Dy- namic operator optimizati...

    work page 2025

  7. [2004]

    V ., Møgelmose, A., and Trivedi, M

    Greer, R., Antoniussen, B., Andersen, M. V ., Møgelmose, A., and Trivedi, M. M. The why, when, and how to use active learning in large-data-driven 3d object detection for safe autonomous driving: An empirical exploration. arXiv preprint arXiv:2401.16634,

    arXiv

  8. [2008]

    Nlprompt: Noise-label prompt learning for vision-language models.arXiv preprint arXiv:2412.01256,

    Pan, B., Li, Q., Tang, X., Huang, W., Fang, Z., Liu, F., Wang, J., Yu, J., and Shi, Y . Nlprompt: Noise-label prompt learning for vision-language models.arXiv preprint arXiv:2412.01256,

    work page arXiv

Show all 16 references
  1. [2009]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    2010 arXiv

  2. [2012]

    Postedit: Posterior sampling for efficient zero-shot image editing.arXiv preprint arXiv:2410.04844,

    Tian, F., Li, Y ., Yan, Y ., Guan, S., Ge, Y ., and Yang, X. Postedit: Posterior sampling for efficient zero-shot image editing.arXiv preprint arXiv:2410.04844,

    arXiv

  3. [2019]

    A multi-target tracking algorithm for fast-moving work- pieces based on event camera

    Wang, Y ., Liu, C., Li, S., Wang, T., Lin, W., and Yu, X. A multi-target tracking algorithm for fast-moving work- pieces based on event camera. InIECON 2021–47th Annual Conference of the IEEE Industrial Electronics Society, pp. 1–5. IEEE,

    2021

  4. [2020]

    Find n’propagate: Open-vocabulary 3d object detection in urban environments.arXiv preprint arXiv:2403.13556,

    Etchegaray, D., Huang, Z., Harada, T., and Luo, Y . Find n’propagate: Open-vocabulary 3d object detection in urban environments.arXiv preprint arXiv:2403.13556,

    arXiv

  5. [2022]

    Food-101– mining discriminative components with random forests

    Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. InComputer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part VI 13, pp. 446–461. Springer,

    2014

  6. [2023]

    Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178–178. IEEE,

    2004

  7. [2024]

    A similarity paradigm through textual regularization without forget- ting

    Cui, F., Fong, J., Zeng, R., Tian, X., and Yu, J. A similarity paradigm through textual regularization without forget- ting. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 16100–16108, 2025a. Cui, F., Zhang, Y ., Wang, X., Wang, X., and Xiao, L...

    arXiv

  8. [2025]

    R., and Shah, M

    Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.