{"id":"f01bd9d7-2255-481a-9e8d-d7293dbaf6a4","arxiv_id":"2501.14148","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SelfPrompt combines cluster-based pseudo-labelling with confidence-aware learning to improve semi-supervised prompt tuning of CLIP-style vision-language models.","lead":"A new method for adapting vision-language models with very few labelled examples uses clustering and confidence scores to make better use of unlabeled images. It reports average accuracy gains over previous semi-supervised prompt-tuning methods on image benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cluster-guided pseudo-labelling's core assumption—that CLIP's frozen embedding space is locally class-homogeneous—is never measured; the main gains depend on it.","rationale":"The central claim is that SelfPrompt significantly surpasses SOTA. The novelty (and the ablations) place the burden on the cluster-guided pseudo-labelling module. The paper's own Figure 1 is the only pseudo-label accuracy evidence and it is a single dataset; no precision@k is reported for the other 12. Since the frozen CLIP encoder is never adapted before pseudo-labelling, the local class-homogeneity assumption is exactly the risk the reader identified. This is more load-bearing than secondary concerns (test-set hyperparameter selection, the ambiguous t=τM definition) because a failure here would mean the proposed mechanism is not what drives the observed gains. The proposed test is cheap and settles the question: compute precision@p on all datasets. If the concern lands, the verdict should remain conditional, with the preprint needing additional analysis or a modified claim. If precision is high, the concern is resolved.","tokens_in":18032,"tokens_out":11437,"duration_ms":102598,"concrete_test":"Using the frozen CLIP ViT-B/32 encoder, reproduce the 2-shot labelled-set protocol (same seeds/random splits as Table 1) and for each of the 13 datasets compute the mean precision@50 of the nearest-neighbour pseudo-labels generated by Section 3.2. Tabulate precision per dataset and correlate with the per-dataset gains in Table 1 (Δ column). If fine-grained datasets show precision below ~50% while still showing large gains, the cluster-guided mechanism is not the source of improvement; if precision is high (e.g., >80%), the assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 constructs pseudo-labels by taking, for each labelled sample, the p nearest unlabeled samples in the frozen CLIP image encoder's embedding space and assigning the labelled sample's class. The method's reported gains (e.g., +4.71% over CPL in Table 1, +10.54% on FGVCAircraft) depend on these nearest neighbours being predominantly same-class. The only quantitative evidence is Figure 1, which plots pseudo-label accuracy on FGVCAircraft (one dataset) and mixes the effect of the confidence-aware module and iterative re-labelling. The ablation (Table 7) shows removing cluster-guided pseudo-labelling costs 4.94% average accuracy, so CGP is a primary driver of the headline improvement. For fine-grained datasets (FGVCAircraft zero-shot 17.6%, CUB 51.8%, StanfordCars 60.3%), the frozen CLIP space may not separate classes at the required granularity; if precision@50 is low, the gains must come from elsewhere (e.g., the confidence-aware module masking noisy labels) and the paper's mechanism is not established. No precision@k for the CGP step is reported across the 13 datasets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SelfPrompt, a prompt-tuning method for vision-language models in a semi-supervised setting. SelfPrompt combines three components: cluster-guided pseudo-labelling, in which unlabeled samples near each labeled sample in the frozen CLIP embedding space are assigned the label of that labeled sample; a confidence-aware semi-supervised module that learns from high-confidence pseudo-labels with a supervised loss and from low-confidence samples with a weakly supervised partial-label loss; and a weakly supervised sampling strategy for the active semi-supervised setting that removes extreme-confidence samples and uses k-means to select a diverse labeled set. The authors evaluate on 13 datasets, reporting average improvements over prior state-of-the-art methods in standard semi-supervised tuning, active semi-supervised tuning, base-to-novel generalization, and 1-shot settings, with ablation and sensitivity studies of the main components.","tokens_in":18260,"tokens_out":5039,"duration_ms":46667,"significance":"If the reported gains are reliable, this is a practically useful empirical contribution to low-label VLM adaptation. The paper evaluates on substantially more datasets than prior work in this line, reports training time comparable to CPL, and includes ablations that isolate the contributions of the three proposed modules. The sensitivity analyses of clustering algorithm, quantile count q, number of cluster pseudo-labels p, and confidence fraction tau are useful. However, because the central claims are purely empirical, the reliability of the comparison depends heavily on the exact protocol, hyperparameter selection, and whether the proposed mechanism is actually responsible for the gains. The current manuscript has several reporting gaps on exactly these points, so I cannot yet endorse the headline claims as stated.","major_comments":[{"comment":"The abstract and the contribution bullet state an average improvement of 6.23% in standard semi-supervised learning, but Table 1, which Section 4.2 describes as the standard 2-shot textual-prompting result, reports a 4.71% average improvement over CPL. The 6.23% value is the visual-prompting result in Table 2. The phrase 'standard semi-supervised learning' in the abstract is therefore misleading and should be qualified as the visual-prompting setup, with the textual-prompting result reported separately.","section":"Abstract; Section 4.1; Tables 1 and 2"},{"comment":"The hyperparameter p introduced in Section 3.2 for cluster-guided pseudo-labelling is omitted from the main experimental setup description in Section 4.1, which lists only q = 5, tau = 0.05, and lambda = 1. Table 8c shows that p strongly affects accuracy and that p = 50 corresponds to the reported 79.33% value. The default value of p, and whether it applies to both standard and active setups, must be stated explicitly for the experiments to be reproducible.","section":"Section 4.1; Table 8c"},{"comment":"No held-out validation protocol is described. The free parameters q, tau, lambda, and p are fixed in the main results, and Table 8 reports sensitivity averaged over the full test sets of the 13 datasets, with no indication of how these values were selected. Without a validation split or a clear statement that all values are fixed a priori, the reader cannot rule out test-set tuning. Please specify the hyperparameter selection procedure, or report per-dataset sensitivity and show that the results are stable under a validation-based selection rule.","section":"Section 4.1; Table 8; Section A.1"},{"comment":"The cluster-guided pseudo-labelling step is load-bearing: removing it in Table 7 reduces average accuracy by 4.94% (from 79.33% to 74.39%). Yet the correctness of this step is never directly measured. Section 3.2 assigns the label of each labeled sample to its p nearest neighbors in the frozen CLIP embedding space, and the only quantitative evidence is Figure 1 on FGVCAircraft, which reports end-to-end pseudo-label accuracy and therefore mixes the effects of the confidence-aware module, iterative relabelling, and the sampling module. Please report a precision@k or neighbourhood-purity measure for the cluster-guided pseudo-labels across the 13 datasets, especially the fine-grained datasets (FGVCAircraft, StanfordCars, CUB), where the assumption that frozen CLIP neighborhoods are class-homogeneous is least obviously satisfied.","section":"Section 3.2; Table 7; Figure 1"},{"comment":"Table 4 is introduced in the active semi-supervised section but its caption reads only 'semi-supervised tuning with textual prompting with varying numbers of shots.' Its 2-shot SelfPrompt row is 79.33%, matching the active-setting result in Table 3 and not the standard-setting result in Table 1 (76.12%). The table caption and surrounding text need to state explicitly that these are active semi-supervised results obtained with weakly supervised sampling. In addition, Table 6 compares SelfPrompt and PromptKD, which use unlabelled data, with methods such as Co-CoOp and MaPLe that do not; the average improvement in the harmonic mean may partly reflect the use of additional unlabelled data rather than the proposed components. This comparison should either be complemented by unlabelled-data-aware baselines or clearly framed with this caveat.","section":"Section 4.3, Table 4; Section 4.4, Table 6"}],"minor_comments":[{"comment":"The ablation study is introduced with the sentence 'We present an ablation study on our proposed method in Table 9,' but the table is numbered Table 7. The same paragraph twice refers to 'clutter-guided pseudo-labelling' instead of 'cluster-guided pseudo-labelling,' and the Figure 4 caption contains the typo 'fist and cat' instead of 'fish and cat.'","section":"Section 4.5; Figure 4"},{"comment":"The text in Section 4.3 states that the improvement over CPL is 7.92% with 2 shots and 11.78% with 1 shot, but the table caption does not state whether the labeled set was selected by weakly supervised sampling in all of these rows. Making this explicit would prevent readers from conflating the standard and active protocols.","section":"Table 4"},{"comment":"Table 6 uses a ViT-B/16 backbone while the main tables use CLIP-B/32; the switch is mentioned only in the table title. A sentence in Section 4.4 explaining that base-to-novel experiments follow the PromptKD protocol with ViT-B/16 would clarify this.","section":"Section 4.4, Table 6"},{"comment":"The appendix repeats the hyperparameter statement from Section 4.1 but still omits p. Once p is added to Section 4.1, the appendix should be updated to match.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical contribution and the overall direction is reasonable, but the referee report focuses on three issues that matter for the final verdict: the abstract's headline number is not the standard-protocol number, the default value of p is missing from the setup, and there is no validation protocol that would rule out test-set tuning of q, tau, lambda, and p. The cluster-guided pseudo-labelling mechanism, which is the most novel component, also lacks a direct evaluation of its precision. These are all fixable with additional experiments and reporting, so I recommend major revision rather than rejection. I would also ask the editor to ensure that the authors address the comparison fairness in Table 6, since comparing against methods that do not use unlabelled data weakens the claim of a 4.9% average improvement in base-to-novel generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SelfPrompt is a solid, useful semi-supervised prompt tuning paper that likely deserves a serious referee, but it needs some cleanup on reporting before I'd trust the headline numbers.\n\nWhat's new: The combination of cluster-guided pseudo-labelling (CGP) with confidence-aware SSL and active sampling is genuinely different from GRIP and CPL. Using frozen CLIP embeddings to find nearest neighbours to labelled samples, rather than zero-shot prediction, is a plausible fix for miscalibration. The paper does the work: 13 datasets, both textual and visual prompts, ablations, sensitivity analysis, different backbones, and base-to-novel generalization. The gains over CPL are consistent and often large, especially on fine-grained datasets like FGVCAircraft and EuroSAT. The ablation shows CGP is the main driver (~4.9% drop when removed), which is believable given the mechanism.\n\nSoft spots, in order of annoyance:\n\n1. The abstract reports 6.23% as the standard SSL improvement, but that's the visual prompting number from Table 2. The textual prompting table shows 4.71%. This is a reporting inconsistency that should be fixed before I'd call the headline claim accurate.\n\n2. The hyperparameter p (pseudo-labels per cluster) is never specified in the setup section. The sensitivity table shows p=50 works best, but it's not stated which value was used for the main results. That's an easy fix but essential for reproducibility.\n\n3. The core assumption of CGP—that CLIP's embedding space is locally class-homogeneous—is asserted but never directly measured. The precision of those nearest-neighbour pseudo-labels is only shown for one dataset via pseudo-label accuracy curves that mix in the confidence module. Given how much of the gain rests on this step, a precision@k table across all 13 datasets would make the mechanism much more convincing. That said, the empirical consistency of the gains gives me some confidence the assumption holds reasonably well.\n\n4. No validation protocol for hyperparameter selection is described, which leaves the usual cherry-picking worry. The sensitivity ranges are narrow, though, so I'm not deeply concerned.\n\nThe 4-shot regression the reader mentioned isn't really there: Table 4 shows SelfPrompt's 4-shot average is slightly *higher* than 2-shot. CPL, on the other hand, degrades at 4-shot, which the paper correctly notes.\n\nThis paper is for anyone working on CLIP adaptation in label-scarce settings. It doesn't open new capabilities, but it gives a practical drop in labelling cost. I'd send it to peer review—it deserves a careful referee.\n\nRecommendation: Engage with it. Conditional accept is the right range.","headline":"Solid semi-supervised prompt tuning with plausible gains, but the abstract's headline number doesn't match the main table and the key hyperparameter p is missing from the setup.","tokens_in":18789,"tokens_out":2890,"would_cite":true,"duration_ms":25290,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SelfPrompt claims replacing zero-shot pseudo-labels with cluster-guided pseudo-labels improves semi-supervised VLM prompt tuning by up to 6.23% on average across 13 datasets.","keywords":["SelfPrompt","semi-supervised learning","prompt tuning","vision-language models","pseudo-labelling","active learning","CLIP","base-to-novel generalization"],"falsifier":"Measure the class purity of the p nearest neighbours of each labelled sample in the frozen CLIP embedding space on the 13 datasets; if a substantial fraction of neighbour sets are dominated by a different class, for example on FGVCAircraft, then the cluster-guided pseudo-labels are largely wrong and the claimed gain should vanish when those neighbour sets are replaced by random unlabelled samples. This is directly testable with the authors' code and data splits.","tokens_in":1717,"feed_emoji":"🏷️","tokens_out":1742,"duration_ms":63018,"temperature":0.7,"pith_summary":"The paper tries to establish that a semi-supervised prompt-tuning method called SelfPrompt lets a frozen vision-language model adapt to a downstream image task using only a handful of labelled examples plus a pool of unlabelled images. Its central claim is that the standard practice of using the model's own zero-shot predictions as pseudo-labels is a bottleneck, because those predictions are miscalibrated and the resulting noisy labels accumulate over iterative training. SelfPrompt instead draws pseudo-labels from neighbourhoods around labelled samples in the model's embedding space, and then splits the unlabelled data into high-confidence samples trained with full supervision and low-confidence samples trained with weak partial-label losses. The paper reports average improvements of 4.71% over the prior CPL method under the 2-shot textual-prompt protocol (6.23% over previous methods overall), with larger gains in active and single-shot settings. A sympathetic reader would care because this is a parameter-efficient route to squeezing usable supervision out of unlabelled data in label-scarce regimes.","feed_headline":"Cluster pseudo-labels lift semi-supervised VLM tuning by 6.23%","feed_subtitle":"With only two labels per class, a cluster-and-confidence recipe beats previous prompt-tuning methods across 13 image benchmarks.","key_machinery":"The load-bearing mechanism is cluster-guided pseudo-labelling: the few labelled samples act as cluster centres in the frozen CLIP embedding space, and each cluster's p closest unlabelled images are assigned the cluster centre's label, avoiding reliance on the VLM's miscalibrated zero-shot predictions. Around it, the confidence-aware semi-supervised module partitions unlabelled data into high-confidence samples trained with cross-entropy and low-confidence samples trained with a partial-label loss, and the weakly-supervised sampling module filters out the most and least confident VLM predictions, then k-means-clusters the rest to select a diverse labelled set. The combined training objective is $L_{\\text{final}} = \\frac{1}{|X_L|}\\sum_{(x,y)\\in X_L}\\ell(f(x),y) + \\frac{1}{|X_+|}\\sum_{(x,y)\\in X_+}\\ell(f(x),y) + \\frac{\\lambda}{|X_{\\text{weak}}|}\\sum_{(x,s)\\in X_{\\text{weak}}}\\ell_w(f(x),s)$, where the last term is a partial-label cross-entropy over top-k predictions.","core_discovery":"On the paper's own terms, the discovery is that the quality of the pseudo-labels, not the amount of unlabelled data, is what limits semi-supervised prompt tuning of VLMs. The authors show that by clustering all labelled and unlabelled images in the frozen CLIP embedding space and assigning the label of each labelled centre to its p nearest unlabelled neighbours, they obtain pseudo-labels that are accurate from the very first training session, whereas zero-shot pseudo-labelling methods degrade as sessions accumulate. Those high-quality pseudo-labels, combined with a confidence-aware loss that trains confident pseudo-labels supervised and low-confidence samples with partial labels, produce consistent accuracy gains across 13 datasets in standard semi-supervised, active semi-supervised, and base-to-novel generalization settings.","pith_inferences":["Inference beyond the paper: if cluster-guided pseudo-labelling works because CLIP's embedding space is locally class-homogeneous, the method should transfer to other frozen encoders with similar geometry, but it will need a diagnostic of nearest-neighbour class purity before deployment on fine-grained or out-of-distribution domains.","Inference beyond the paper: the ablation shows cluster-guided pseudo-labelling is the largest single contributor, so a natural next experiment is to combine it with stronger weak-supervision losses or uncertainty metrics.","Inference beyond the paper: the weakly-supervised sampling's filtering discards both high- and low-confidence samples, which predicts that on very easy datasets the sampling will do less than on hard ones; this can be checked by correlating improvement with dataset difficulty.","Inference beyond the paper: because the method selects p nearest neighbours per cluster, accuracy should degrade gracefully as p grows; the sensitivity table already shows p=50 close to p=75, suggesting an intrinsic limit on how much pseudo-labelled data helps before noise dominates."],"forward_implications":["If the central claim holds, the standard recipe for VLM semi-supervised tuning changes: start with embedding-neighbourhood pseudo-labels rather than zero-shot confidence thresholds, and treat unlabelled samples by confidence level rather than by a single pseudo-label.","The weakly-supervised sampling module, described as integrable into any existing semi-supervised method, would be a plug-in way to improve labelling-budget use beyond SelfPrompt itself.","The reported gain pattern, with larger improvements on datasets where zero-shot accuracy is low such as FGVCAircraft and MNIST, suggests the method's advantage grows exactly where miscalibration is worst, so it should be tested in other low-zero-shot domains.","Base-to-novel results show gains on both seen and unseen classes, implying that adapting with unlabelled data can improve generalization rather than just fit the labelled classes.","Training time is reported on par with CPL, so the gains do not come with a large computational penalty."],"supporting_citations":[{"why":"Supplies the CLIP vision-language backbone and the zero-shot prediction formulation that SelfPrompt builds on and attempts to adapt.","marker":"(Radford et al., 2021)"},{"why":"Defines textual prompt tuning and the base-to-novel generalization protocol used for evaluation.","marker":"(Zhou et al., 2022a)"},{"why":"GRIP, the prior semi-supervised prompt-tuning method that SelfPrompt compares against and whose zero-shot pseudo-labelling limitation motivates the cluster-guided approach.","marker":"(Menghini et al., 2023)"},{"why":"CPL, the prior state-of-the-art baseline, whose candidate pseudo-label loss is reused as the weak partial-label loss in the confidence-aware module.","marker":"(Zhang et al., 2024)"},{"why":"MaPLe, a multimodal prompt-tuning baseline used in the base-to-novel generalization comparisons.","marker":"(Khattak et al., 2022)"},{"why":"PromptKD, which supplies the unlabelled-data protocol for the base-to-novel generalization experiments.","marker":"(Li et al., 2024)"},{"why":"Introduces visual prompt tuning, used for the visual-prompt variant of SelfPrompt.","marker":"(Bahng et al., 2022b)"}],"fun_headline_variants":["Cluster pseudo-labels sharpen semi-supervised VLM tuning","Confidence-aware semi-supervised tuning lifts VLM gains","SelfPrompt: cluster and confidence for robust VLM adaptation","Better pseudo-labels, not more data, boost VLM tuning","Cluster-guided pseudo-labels beat zero-shot for VLM SSL"],"cache_read_input_tokens":20992,"weakest_assumption_plain":"Cluster-guided pseudo-labelling assumes that, inside the model's frozen embedding space, the p images nearest to each labelled example are overwhelmingly from the same class as that example; if that local similarity breaks down, the pseudo-labels are wrong and training is actively misled.","fun_headline_variants_meta":{"raw":{"variants":["Cluster pseudo-labels sharpen semi-supervised VLM tuning","Confidence-aware semi-supervised tuning lifts VLM gains","SelfPrompt: cluster and confidence for robust VLM adaptation","Better pseudo-labels, not more data, boost VLM tuning","Cluster-guided pseudo-labels beat zero-shot for VLM SSL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2675,"prompt_tokens":945,"completion_tokens":1730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1646}},"tokens_in":561,"tokens_out":1730,"duration_ms":11133,"temperature":1.0,"reasoning_tokens":1646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:20:18.384929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the class purity of the p nearest neighbours of each labelled sample in the frozen CLIP embedding space on the 13 datasets; if a substantial fraction of neighbour sets are dominated by a different class, for example on FGVCAircraft, then the cluster-guided pseudo-labels are largely wrong and the claimed gain should vanish when those neighbour sets are replaced by random unlabelled samples. This is directly testable with the authors' code and data splits.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GRIP, the prior semi-supervised prompt-tuning method that SelfPrompt compares against and whose zero-shot pseudo-labelling limitation motivates the cluster-guided approach."}],"review_version":1}