REVIEW 5 major objections 4 minor 13 references
SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation
T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read SelfPrompt claims replacing zero-shot pseudo-labels with cluster-guided pseudo-labels improves semi-supervised VLM prompt tuning by up to 6.23% on average across 13 datasets.
desk verdict Solid semi-supervised prompt tuning with plausible gains, but the abstract's headline number doesn't match the main table and the key hyperparameter p is missing from the setup. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is cluster-guided pseudo-labelling: the few labelled samples act as cluster centres in the frozen CLIP embedding space, and each cluster's p closest unlabelled images are assigned the cluster centre's label, avoiding reliance on the VLM's miscalibrated zero-shot predictions. Around it, the confidence-aware semi-supervised module partitions unlabelled data into high-confidence samples trained with cross-entropy and low-confidence samples trained with a partial-label loss, and the weakly-supervised sampling module filters out the most and least confident VLM predictions, then k-means-clusters the rest to select a diverse labelled set. The combined training objective is $L_{\text{final}} = \frac{1}{|X_L|}\sum_{(x,y)\in X_L}\ell(f(x),y) + \frac{1}{|X_+|}\sum_{(x,y)\in X_+}\ell(f(x),y) + \frac{\lambda}{|X_{\text{weak}}|}\sum_{(x,s)\in X_{\text{weak}}}\ell_w(f(x),s)$, where the last term is a partial-label cross-entropy over top-k predictions.
What would settle it
Measure the class purity of the p nearest neighbours of each labelled sample in the frozen CLIP embedding space on the 13 datasets; if a substantial fraction of neighbour sets are dominated by a different class, for example on FGVCAircraft, then the cluster-guided pseudo-labels are largely wrong and the claimed gain should vanish when those neighbour sets are replaced by random unlabelled samples. This is directly testable with the authors' code and data splits.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the quality of the pseudo-labels, not the amount of unlabelled data, is what limits semi-supervised prompt tuning of VLMs. The authors show that by clustering all labelled and unlabelled images in the frozen CLIP embedding space and assigning the label of each labelled centre to its p nearest unlabelled neighbours, they obtain pseudo-labels that are accurate from the very first training session, whereas zero-shot pseudo-labelling methods degrade as sessions accumulate. Those high-quality pseudo-labels, combined with a confidence-aware loss that trains confident pseudo-labels supervised and low-confidence samples with partial labels, produce consistent accuracy gains across 13 datasets in standard semi-supervised, active semi-supervised, and base-to-novel generalization settings.
Load-bearing premise
Cluster-guided pseudo-labelling assumes that, inside the model's frozen embedding space, the p images nearest to each labelled example are overwhelmingly from the same class as that example; if that local similarity breaks down, the pseudo-labels are wrong and training is actively misled.
Editorial extensions
If this is right
- If the central claim holds, the standard recipe for VLM semi-supervised tuning changes: start with embedding-neighbourhood pseudo-labels rather than zero-shot confidence thresholds, and treat unlabelled samples by confidence level rather than by a single pseudo-label.
- The weakly-supervised sampling module, described as integrable into any existing semi-supervised method, would be a plug-in way to improve labelling-budget use beyond SelfPrompt itself.
- The reported gain pattern, with larger improvements on datasets where zero-shot accuracy is low such as FGVCAircraft and MNIST, suggests the method's advantage grows exactly where miscalibration is worst, so it should be tested in other low-zero-shot domains.
- Base-to-novel results show gains on both seen and unseen classes, implying that adapting with unlabelled data can improve generalization rather than just fit the labelled classes.
- Training time is reported on par with CPL, so the gains do not come with a large computational penalty.
Reading between the lines
- Inference beyond the paper: if cluster-guided pseudo-labelling works because CLIP's embedding space is locally class-homogeneous, the method should transfer to other frozen encoders with similar geometry, but it will need a diagnostic of nearest-neighbour class purity before deployment on fine-grained or out-of-distribution domains.
- Inference beyond the paper: the ablation shows cluster-guided pseudo-labelling is the largest single contributor, so a natural next experiment is to combine it with stronger weak-supervision losses or uncertainty metrics.
- Inference beyond the paper: the weakly-supervised sampling's filtering discards both high- and low-confidence samples, which predicts that on very easy datasets the sampling will do less than on hard ones; this can be checked by correlating improvement with dataset difficulty.
- Inference beyond the paper: because the method selects p nearest neighbours per cluster, accuracy should degrade gracefully as p grows; the sensitivity table already shows p=50 close to p=75, suggesting an intrinsic limit on how much pseudo-labelled data helps before noise dominates.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SelfPrompt, a prompt-tuning method for vision-language models in a semi-supervised setting. SelfPrompt combines three components: cluster-guided pseudo-labelling, in which unlabeled samples near each labeled sample in the frozen CLIP embedding space are assigned the label of that labeled sample; a confidence-aware semi-supervised module that learns from high-confidence pseudo-labels with a supervised loss and from low-confidence samples with a weakly supervised partial-label loss; and a weakly supervised sampling strategy for the active semi-supervised setting that removes extreme-confidence samples and uses k-means to select a diverse labeled set. The authors evaluate on 13 datasets, reporting average improvements over prior state-of-the-art methods in standard semi-supervised tuning, active semi-supervised tuning, base-to-novel generalization, and 1-shot settings, with ablation and sensitivity studies of the main components.
Significance. If the reported gains are reliable, this is a practically useful empirical contribution to low-label VLM adaptation. The paper evaluates on substantially more datasets than prior work in this line, reports training time comparable to CPL, and includes ablations that isolate the contributions of the three proposed modules. The sensitivity analyses of clustering algorithm, quantile count q, number of cluster pseudo-labels p, and confidence fraction tau are useful. However, because the central claims are purely empirical, the reliability of the comparison depends heavily on the exact protocol, hyperparameter selection, and whether the proposed mechanism is actually responsible for the gains. The current manuscript has several reporting gaps on exactly these points, so I cannot yet endorse the headline claims as stated.
major comments (5)
- [Abstract; Section 4.1; Tables 1 and 2] The abstract and the contribution bullet state an average improvement of 6.23% in standard semi-supervised learning, but Table 1, which Section 4.2 describes as the standard 2-shot textual-prompting result, reports a 4.71% average improvement over CPL. The 6.23% value is the visual-prompting result in Table 2. The phrase 'standard semi-supervised learning' in the abstract is therefore misleading and should be qualified as the visual-prompting setup, with the textual-prompting result reported separately.
- [Section 4.1; Table 8c] The hyperparameter p introduced in Section 3.2 for cluster-guided pseudo-labelling is omitted from the main experimental setup description in Section 4.1, which lists only q = 5, tau = 0.05, and lambda = 1. Table 8c shows that p strongly affects accuracy and that p = 50 corresponds to the reported 79.33% value. The default value of p, and whether it applies to both standard and active setups, must be stated explicitly for the experiments to be reproducible.
- [Section 4.1; Table 8; Section A.1] No held-out validation protocol is described. The free parameters q, tau, lambda, and p are fixed in the main results, and Table 8 reports sensitivity averaged over the full test sets of the 13 datasets, with no indication of how these values were selected. Without a validation split or a clear statement that all values are fixed a priori, the reader cannot rule out test-set tuning. Please specify the hyperparameter selection procedure, or report per-dataset sensitivity and show that the results are stable under a validation-based selection rule.
- [Section 3.2; Table 7; Figure 1] The cluster-guided pseudo-labelling step is load-bearing: removing it in Table 7 reduces average accuracy by 4.94% (from 79.33% to 74.39%). Yet the correctness of this step is never directly measured. Section 3.2 assigns the label of each labeled sample to its p nearest neighbors in the frozen CLIP embedding space, and the only quantitative evidence is Figure 1 on FGVCAircraft, which reports end-to-end pseudo-label accuracy and therefore mixes the effects of the confidence-aware module, iterative relabelling, and the sampling module. Please report a precision@k or neighbourhood-purity measure for the cluster-guided pseudo-labels across the 13 datasets, especially the fine-grained datasets (FGVCAircraft, StanfordCars, CUB), where the assumption that frozen CLIP neighborhoods are class-homogeneous is least obviously satisfied.
- [Section 4.3, Table 4; Section 4.4, Table 6] Table 4 is introduced in the active semi-supervised section but its caption reads only 'semi-supervised tuning with textual prompting with varying numbers of shots.' Its 2-shot SelfPrompt row is 79.33%, matching the active-setting result in Table 3 and not the standard-setting result in Table 1 (76.12%). The table caption and surrounding text need to state explicitly that these are active semi-supervised results obtained with weakly supervised sampling. In addition, Table 6 compares SelfPrompt and PromptKD, which use unlabelled data, with methods such as Co-CoOp and MaPLe that do not; the average improvement in the harmonic mean may partly reflect the use of additional unlabelled data rather than the proposed components. This comparison should either be complemented by unlabelled-data-aware baselines or clearly framed with this caveat.
minor comments (4)
- [Section 4.5; Figure 4] The ablation study is introduced with the sentence 'We present an ablation study on our proposed method in Table 9,' but the table is numbered Table 7. The same paragraph twice refers to 'clutter-guided pseudo-labelling' instead of 'cluster-guided pseudo-labelling,' and the Figure 4 caption contains the typo 'fist and cat' instead of 'fish and cat.'
- [Table 4] The text in Section 4.3 states that the improvement over CPL is 7.92% with 2 shots and 11.78% with 1 shot, but the table caption does not state whether the labeled set was selected by weakly supervised sampling in all of these rows. Making this explicit would prevent readers from conflating the standard and active protocols.
- [Section 4.4, Table 6] Table 6 uses a ViT-B/16 backbone while the main tables use CLIP-B/32; the switch is mentioned only in the table title. A sentence in Section 4.4 explaining that base-to-novel experiments follow the PromptKD protocol with ViT-B/16 would clarify this.
- [Appendix A.1] The appendix repeats the hyperparameter statement from Section 4.1 but still omits p. Once p is added to Section 4.1, the appendix should be updated to match.
Circularity Check
No significant circularity: the paper's claims are empirical, evaluated on held-out test sets, and its component contributions are supported by ablations rather than defined into existence.
full rationale
SelfPrompt makes no formal derivation that reduces a predicted quantity to a fitted parameter. The cluster-guided pseudo-labelling step (Section 3.2) uses labelled samples as cluster centers and assigns labels to nearby unlabelled samples, but this is a proposed algorithmic heuristic whose quality is then measured on held-out test accuracy; the outcome is not defined in terms of the selection rule. The confidence-aware module uses the model's own confidence to choose pseudo-labels, which is standard self-training rather than definitional circularity. The weakly-supervised sampling module is likewise evaluated against alternative sampling strategies (Table 9) and through ablations (Table 7), which is exactly the kind of independent empirical support that rules out a fitted-input-called-prediction pattern. The only self-citation identified is to the authors' prior work (Roy & Etemad, 2024) in Appendix A.3, used to motivate excluding overconfident samples; this citation is not load-bearing because the paper also provides its own ablation evidence for the design choice, and no uniqueness claim or forbidden-alternative argument rests on it. All central claims are benchmark comparisons against external and prior methods with reported standard deviations, so the derivation chain is self-contained in the relevant sense: the reported improvements are empirical findings, not consequences of how the method's components are defined.
Assumptions & free parameters
free parameters (4)
- q (confidence quantile count) =
5 (main); 3, 10, 20 studied in Table 8b
- tau (strong pseudo-label fraction) =
0.05 (main); 0.10, 0.20 studied in Table 8d
- lambda (weak loss weight) =
1
- p (pseudo-labels per cluster) =
not stated in main text; best 50 in Table 8c
assumptions (4)
- domain assumption CLIP embedding space is locally class-homogeneous
- domain assumption Unlabeled set is drawn from the same distribution as the labelled set and test set
- domain assumption VLM confidence is a meaningful signal for sample informativeness
- domain assumption Partial-label loss with top-k predictions handles noisy pseudo-labels
Cite this review
Pith. "Pith review of SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation." pith.science (2026). https://pith.science/paper/INQ5UEGE
@misc{pith2026250114148,
author = {Pith},
title = {Pith review of: SelfPrompt: Confidence-Aware Semi-Supervised Tuning for Robust Vision-Language Model Adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/INQ5UEGE}},
note = {Machine review of arXiv:2501.14148}
}
read the original abstract
We present SelfPrompt, a novel prompt-tuning approach for vision-language models (VLMs) in a semi-supervised learning setup. Existing methods for tuning VLMs in semi-supervised setups struggle with the negative impact of the miscalibrated VLMs on pseudo-labelling, and the accumulation of noisy pseudo-labels. SelfPrompt addresses these challenges by introducing a cluster-guided pseudo-labelling method that improves pseudo-label accuracy, and a confidence-aware semi-supervised learning module that maximizes the utilization of unlabelled data by combining supervised learning and weakly-supervised learning. Additionally, we investigate our method in an active semi-supervised learning setup, where the labelled set is strategically selected to ensure the best utilization of a limited labelling budget. To this end, we propose a weakly-supervised sampling technique that selects a diverse and representative labelled set, which can be seamlessly integrated into existing methods to enhance their performance. We conduct extensive evaluations across 13 datasets, significantly surpassing state-of-the-art performances with average improvements of 6.23% in standard semi-supervised learning, 6.25% in active semi-supervised learning, and 4.9% in base-to-novel generalization, using a 2-shot setup. Furthermore, SelfPrompt shows excellent generalization in single-shot settings, achieving an average improvement of 11.78%.
Figures
Reference graph
Works this paper leans on
-
[1]
Training is performed on a single Nvidia V100 GPU
Results are reported as the average accuracy and the standard deviation over three runs with random seeds. Training is performed on a single Nvidia V100 GPU. A.2. Related Works Vision-language models. Vision-language models (VLMs), pre-trained on vast web-scale datasets of image-text pairs, have demonstrated remarkable generalization across a wide range o...
work page 2022
-
[6]
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,
-
[9]
and (Menghini et al., 2023). Specifically, we utilize a few samples from the training set of a dataset as the labelled set and the remaining samples as the unlabelled set, followed by the assessment of the trained model on the test set. Implementation details. Following (Zhang et al.,
work page 2023
-
[11]
later introduced a multi-modal approach that trains both textual and visual prompts simultaneously, leveraging their synergy to facilitate multi-modal representation learning while avoiding an overemphasis on unimodal features. A similar multi-modal approach was proposed in PromptSRC (Khattak et al., 2023), which, unlike MaPLe, focused on learning task-ag...
work page 2023
-
[12]
enhances vision-language models by augmenting inputs with diverse perspectives and enriched descriptions, dynamically weighting inputs by pre- diction uncertainty, and transporting semantic correlations into a shared embedding space. While these methods have demonstrated improved performance on downstream tasks compared to pre-trained VLMs, they still fac...
work page 2020
-
[13]
proposes to generate refined candidate pseudo-labels through intra- and inter-instance label selection, using a confidence score matrix to improve label accuracy and class balance during fine-tuning. Both GRIP and CPL adopt an iterative process, where the model is used to continuously refine and select additional samples from the unlabelled set. Although ...
work page 2017
-
[2012]
The caltech-ucsd birds-200-2011 dataset
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. California Institute of Technology,
work page 2011
-
[2013]
LeVine, W., Pikus, B., Raja, P., and Gil, F. A. Enabling cali- bration in the zero-shot inference of large vision-language models. arXiv preprint arXiv:2303.12748,
Show all 13 references
-
[2019]
Unsupervised prompt learning for vision-language models
Huang, T., Chu, J., and Wei, F. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649,
-
[2020]
U., Rasheed, H., Maaz, M., Khan, S., and Khan, F
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning.arXiv preprint arXiv:2210.03117,
-
[2022]
Exploring visual prompts for adapting large-scale models
Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022a. Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models....
-
[2023]
we use Flower102 (Nilsback & Zisser- man, 2008), Resisc-45 (Cheng et al., 2017), DTD (Cim- poi et al., 2014), CUB-200 (Wah et al., 2011), EuroSAT (Helber et al., 2019), FGVCAircraft (Maji et al., 2013), and MNIST (Deng, 2012). Additionally, we use the following 6 datasets, bri...
2008
-
[2024]
Fine-grained visual classification of aircraft
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.