Pith. sign in

REVIEW 4 major objections 4 minor 13 references

Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bi-CoG: filtering pseudo-labels by both inter-model voting and intra-model augmentation consistency reliably improves semi-supervised fine-tuning of vision-language models.

desk verdict Useful pseudo-labeling recipe with broad empirical gains, but the theoretical centerpiece is an assumed power-law — send to review with a request to fix or reframe it. read the letter →

arxiv 2510.20477 v3 pith:AQZLYLHO submitted 2025-10-23 cs.LG

Rui Zhu , Song-Lin Lv , Zi-Kang Wang , Lan-Zhe Guo This is my paper · ORCID
classification cs.LG
keywords semi-supervisedlearningvision-languagemodelspseudo-labelingself-trainingprompttuningconsistencyregularizationerror-awarefilteringopen-worldgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that semi-supervised fine-tuning of pre-trained vision-language models improves when pseudo-labels are filtered in three coupled stages: majority voting across models, per-model agreement between original and weakly augmented views with disagreement under strong augmentation, and a dynamically sized pseudo-label budget that grows as estimated model error falls. If true, Bi-CoG is a plug-and-play module that lifts existing prompt-tuning baselines in both open-world base/novel generalization and standard semi-supervised settings, with reported gains up to 5.69 percent across 14 datasets. The paper provides a theoretical derivation for the budget from noisy self-training bounds, along with a power-law correction connecting estimated and true error ratios, and reports ablations showing each stage contributes.

What carries the argument

The load-bearing object is a three-stage pseudo-label selection pipeline. Stage one builds candidate sets D^j_Inter by leave-one-out majority voting: for the target model f_j, only samples on which a majority of the other K-1 models agree are kept. Stage two intersects with D^j_Intra, the samples where each other model's prediction is unchanged under weak augmentation but changes under strong augmentation, which the paper treats as a proxy for reliable yet bias-sensitive samples. Stage three subsamples the intersection to size (e_hat_{t-1}/e_hat_t)^{alpha t} L_{t-1}, where e_hat_t is the error of the voting ensemble on labeled data; this budget is justified by Lemma 1's condition 0 < e_t/e_{

What would settle it

Compute the true pseudo-label error on a held-out subset of the unlabeled pool at each self-training iteration and compare e_t/e_{t-1} with (e_hat_t/e_hat_{t-1})^{alpha t} across several datasets with a fixed alpha. A systematic deviation (for example, a log-log slope that is not alpha t, or alpha needing per-dataset re-fitting) would falsify Theorem 1 and imply the gains come from the consistency filters rather than the error-aware budget. A simpler ablation: replace the error-aware budget with a fixed budget equal to the average selected size; if performance does not drop, the budget mechani

Watch

Extended reading notes

Core claim

The central claim is that pseudo-label quality for self-training vision-language models improves when selection uses both inter-model consistency (leave-one-out majority voting among the other K-1 models) and intra-model consistency (requiring a model to agree on the original and weakly augmented views while disagreeing under strong augmentation), and when the number of pseudo-labels added per iteration is capped by an error-aware budget L_t = (e_hat_{t-1}/e_hat_t)^{alpha t} L_{t-1} - 1 derived from a lemma about noisy self-training. The paper reports that this three-stage pipeline, wrapped around existing prompt-tuning methods, improves harmonic-mean base/novel accuracy and overall accuracy

Load-bearing premise

The load-bearing premise is that the true pseudo-label error-rate ratio across iterations follows the power law e_t/e_{t-1} ≈ (e_hat_t/e_hat_{t-1})^{alpha t}, where e_hat_t is the error measured on labeled data; this relation is introduced as an assumption in the supplementary proof, so if labeled-set error does not track unlabeled pseudo-label error in that way, the adaptive budget is miscalibrated and the error-aware filtering component's claimed benefit loses its foundatio

Editorial extensions

If this is right

  • Existing prompt-tuning pipelines for vision-language models can be improved by adding Bi-CoG as a plug-and-play pseudo-labeling layer, with no change to the base method's hyperparameters.
  • The reported gains hold in both open-world base/novel generalization and standard 4-shot and 25-shot semi-supervised settings, including roughly 8-point harmonic-mean gains on CIFAR-10 and CIFAR-100.
  • The dynamic budget removes the need to preset pseudo-label counts or confidence thresholds, and the self-training loop stops automatically when no model improves.
  • Compared with prior pseudo-labeling approaches on Flowers102, Bi-CoG improves accuracy by up to 5.9 points in the semi-supervised setting and 0.66 points in the open-world setting under the same CoOp backbone.
  • If the error-aware bound is valid, adding unlabeled data is helpful exactly when the error-rate ratio and the pseudo-label count ratio satisfy 0 < e_t/e_{t-1} < L_{t-1}/L_t < 1, giving a principled stopping rule for self-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the power-law correction is assumed rather than derived, one could treat it as a testable empirical law: fitting alpha on a few datasets and checking whether a single alpha transfers would determine whether the budget is a general mechanism or a per-dataset calibration artifact.
  • The intra-model criterion (stable under weak augmentation, changed under strong augmentation) is a cheap uncertainty proxy; replacing it with an explicit confidence measure and comparing the resulting gains would isolate whether augmentation consistency itself or mere filtering strength drives the improvement.
  • The leave-one-out voting design suggests that increasing K beyond 3 might improve pseudo-label reliability but also shrink the candidate pool; a scaling study on K would reveal whether the reported gains saturate or reverse.
  • A natural extension is to apply Bi-CoG to other parameter-efficient fine-tuning targets, such as adapters or low-rank layers, since the pseudo-label selection does not depend on the prompt parameterization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Bi-CoG, a plug-and-play pseudo-labeling module for fine-tuning vision-language models (VLMs) under label scarcity. The method uses K ≥ 3 pre-trained VLMs and selects pseudo-labels through three stages: inter-model consistency via majority voting, intra-model consistency using weak/strong augmentation agreement, and an error-aware dynamic budget that limits the number of pseudo-labels per iteration based on an estimated error-rate ratio. The authors provide a theoretical analysis (Lemma 1 and Theorem 1) and evaluate Bi-CoG on 14 datasets under open-world generalization and semi-supervised settings, integrating it with CoOp, MaPLe, and PromptSRC, and comparing against GRIP and CPL on Flowers102.

Significance. If the reported gains are reliable, Bi-CoG would be a broadly applicable, simple enhancement for prompt-tuning methods, with improvements of up to 5.69% in harmonic mean across 14 datasets and strong results on SSL benchmarks. The paper is notable for addressing pseudo-label bias and dynamic thresholding in VLM fine-tuning, and the ablation study shows each component contributes to the final performance. However, the theoretical centerpiece is currently circular, and the empirical evaluation has unresolved fairness and reproducibility questions. The method's practical utility is plausible, but the current manuscript does not yet substantiate the claim that the error-aware filtering is theoretically grounded.

major comments (4)
  1. [Methodology, Theorem 1 and Supplementary A.2, Eq. (8)/(20)] The theoretical justification for the error-aware pseudo-label budget is circular. Eq. (8) in the main text is stated as a theorem, but the proof in Supplementary A.2 begins by assuming exactly this relation (Eq. 20): a power-law approximation between the true and estimated error-rate ratios. No independent derivation from Lemma 1, PAC bounds, or other learning theory is provided. Consequently, the budget constraints in Eqs. (9)–(10) reduce to an assumed ansatz with a hand-set hyperparameter α. The only empirical support is Figure 4 on StanfordCars with α=1, which is insufficient to establish the general relation across 14 datasets and training regimes. The claim that 'theoretical analysis demonstrates effectiveness' is therefore not supported; the error-aware component should be presented as a heuristic or derived from first principles.
  2. [Experiments, RQ3, Table 3] The comparison against GRIP and CPL may be unfair. Bi-CoG uses K=3 models and fine-tunes all of them via leave-one-out, whereas GRIP and CPL—as described in the paper—appear to use a single CoOp model. If so, the gains in Table 3 could arise primarily from model ensembling rather than the proposed bi-consistency or error-aware mechanism. The manuscript does not state whether GRIP/CPL were also run with three models, nor does it ablate the number of models K. This needs clarification and, if the comparison is asymmetric, a corrected experiment with matched model counts.
  3. [Experimental Settings, Table 7] The paper claims that Bi-CoG 'eliminates the need for manually setting hyperparameters' and is 'adaptive,' but Table 7 shows per-dataset, per-base-method hand-tuned warmup epochs ζ and self-training epochs κ (e.g., EuroSAT and StanfordCars use κ=20 for PromptSRC while most others use κ=1). This contradicts the adaptability claim and raises concerns about overfitting to test sets. The main text does not discuss how these values were chosen or whether results are sensitive to them.
  4. [Table 1 and Table 2] Several reported improvements are within one standard deviation of the baseline and thus not statistically significant. Examples include: MaPLe+Bi-CoG on OxfordPets (96.93±0.06 vs 96.60±0.36), PromptSRC+Bi-CoG on Caltech101 (96.45±0.16 vs 96.11±0.08), Food101 (91.25±0.01 vs 90.91±0.12), and ImageNet-100 (91.07±0.02 vs 90.49±0.22); also Table 2 CIFAR-100 4-shot PromptSRC+Bi-CoG (60.56±0.79 vs 60.33±0.17). With only three runs and no significance testing, the claim of 'consistently and significantly improves' is overstated. The paper should either report paired significance tests, increase repetitions, or temper the claim.
minor comments (4)
  1. [General] Typos: 'contrains' should be 'contains' in the Error-aware Filtering section; 'Bi-CoG'' has a stray apostrophe in the Methodology summary; 'open-world generation' should be 'open-world generalization'; 'ImagetNet-100' in Table 5 should be 'ImageNet-100'. Also, Figure 4's caption appears truncated ('Error rate ratio' repeated).
  2. [Reproducibility] No code or data release is mentioned, and no random seed details are provided beyond three runs. Given the hand-tuned epochs in Table 7, a public implementation would be important for verification.
  3. [Introduction and Related Work] The citation to 'Gupte et al. 2024' for V-PET appears mismatched: the reference list entry describes an active learning paper, not a VLM ensembling method. Please verify.
  4. [Methodology, Eq. (5)] The definition of D^j_Intra uses f_k(x_i) as the pseudo-label from other models, but the criterion 'f_k(x_i) = f_k(\tilde{x}^w_i)' and 'f_k(x_i) \neq f_k(\tilde{x}^s_i)' is applied to each of the K−1 models. The notation is slightly ambiguous about whether the label used is the model's own prediction on the original input or the majority label; clarifying would help.

Circularity Check

1 steps flagged · score 8.0 of 10

Theorem 1 is not proven: Eq. 8 in the main text is identical to Eq. 20 in the supplementary proof, where it is simply assumed; the pseudo-label budget (Eqs. 9–10) therefore reduces to an unverified power-law ansatz.

  1. other [Main text: 'Error-aware Filtering', Theorem 1 / Eq. (8); Supplementary A.2, Eq. (20); Algorithm 1 line 14]
    "Theorem 1. As the training progresses, the true error rate ratio can be approximately equal to the t-power of the estimated error rate ratio... e_t/e_{t-1} ≈ (ehat_t/ehat_{t-1})^{αt} (8)... [Supplementary A.2] We begin by assuming that, during the course of training, the ratio of the true error rates between two consecutive iterations can be closely approximated by a power-law function of the corresponding ratio of the estimated error rates, given by: e_t/e_{t-1} ≈ (ehat_t/ehat_{t-1})^{αt} (20)"

    The proof of Theorem 1 starts by assuming Eq. (20), which is identical to the theorem statement Eq. (8). It then substitutes this assumed relation into Lemma 1 to obtain L*_t (Eq. 22) and the constraint (Eq. 24), restated in the main text as Eq. (9) and implemented as Eq. (10) / Algorithm 1 line 14. Therefore the error-aware pseudo-label budget is not derived from Lemma 1 or learning theory; it is the assumed power-law relation itself, with α a hand-set scaling factor. The theoretical justification reduces to an ansatz; the only empirical support offered is Figure 4 on StanfordCars.

full rationale

The derivation chain is circular at its theoretical centerpiece: Theorem 1 is asserted in the main text, and the supplementary proof assumes exactly the asserted relation. All subsequent budget formulas (Eq. 9, Eq. 10, Algorithm 1 line 14) are algebraic consequences of that assumption, so they do not constitute a proof. The empirical benchmarking (Tables 1–3) and ablations are independent and not circular; however, the paper's stated contribution that 'theoretical analysis ... demonstrate[s] the effectiveness' is undermined because the error-aware component's justification is an assumed power-law with a hand-set α. This is a score of 8 rather than 6 because the central theoretical result itself is forced by the assumption, not merely one component.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on four unproven or assumed inputs: a PAC-style noise bound, a belief that labeled-data error estimates track unlabeled pseudo-label error, the power-law relation that is actually Theorem 1, and the weak-stable/strong-flip heuristic. The free parameters are mostly hand-set, including α, K, initial error, per-dataset epochs, augmentation strengths, and an ImageNet subsample. No new physical or model-level entity is invented.

free parameters (6)
  • Scaling factor α = 1 (fixed in main experiments)
    Controls how quickly the estimated error ratio is discounted in Eq. 8/9. No independent evidence for its value; treated as a hand-set hyperparameter.
  • Number of models K = 3
    Ensemble size for inter-model consistency. Results may depend on K; no sensitivity analysis is provided.
  • Initial error estimate e'_j = 0.5
    Arbitrary starting value in Algorithm 1 that determines when a model becomes eligible for update.
  • Per-dataset warmup and self-training epochs (ζ, κ) = e.g., CoOp CIFAR-100: ζ=15, κ=1; see Table 7
    Hand-selected for each dataset and each base method. This weakens the 'plug-and-play' and 'no manual tuning' claims.
  • Weak/strong augmentation hyperparameters = See Table 6
    Flip/crop probabilities, jitter magnitudes, grayscale probability, blur kernel and probability. The intra-model consistency filter depends critically on these; no sensitivity analysis is given.
  • ImageNet unlabeled subsample = 5% of unlabeled set
    Compute constraint; may affect the ImageNet result and comparability with other datasets.
assumptions (4)
  • standard math PAC-style relation between error rate, sample size, and noise ratio (Eq. 11-14).
    Invoked in the proof of Lemma 1 to justify the condition e_t/e_{t-1} < L_{t-1}/L_t.
  • domain assumption The error rate estimated on the labeled set approximately equals the true error rate of pseudo-labels on unlabeled data.
    Used to replace e_t with ehat_t in Theorem 1 and in the budget formula; no empirical or theoretical justification is given.
  • ad hoc to paper The true error-rate ratio follows a power-law of the estimated error-rate ratio: e_t/e_{t-1} ≈ (ehat_t/ehat_{t-1})^{αt}.
    This is Theorem 1's content, but the proof in Supplementary A.2 assumes it as its starting point. The entire error-aware filtering mechanism depends on it.
  • domain assumption A sample whose prediction is stable under weak augmentation but flips under strong augmentation is reliable and less biased.
    Used in Eq. 5 to define the intra-model consistency set. No theoretical or measured evidence supports this specific criterion; it is a heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models." pith.science (2026). https://pith.science/paper/AQZLYLHO

@misc{pith2026251020477,
  author       = {Pith},
  title        = {Pith review of: Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQZLYLHO}},
  note         = {Machine review of arXiv:2510.20477}
}
abstract

Exploiting unlabeled data through semi-supervised learning (SSL) or leveraging pre-trained models via fine-tuning are two prevailing paradigms for addressing label-scarce scenarios. Recently, growing attention has been given to combining fine-tuning of pre-trained vision-language models (VLMs) with SSL, forming the emerging paradigm of semi-supervised fine-tuning. However, existing methods often suffer from model bias and hyperparameter sensitivity, due to reliance on prediction consistency or pre-defined confidence thresholds. To address these limitations, we propose a simple yet effective plug-and-play methodology named $\underline{\textbf{Bi-Co}}$nsistency-$\underline{\textbf{G}}$uided Self-Training (Bi-CoG), which assigns high-quality and low-bias pseudo-labels, by simultaneously exploiting inter-model and intra-model consistency, along with an error-aware dynamic pseudo-label assignment strategy. Both theoretical analysis and extensive experiments over 14 datasets demonstrate the effectiveness of Bi-CoG, which consistently and significantly improves the performance of existing methods.

Figures

Figures reproduced from arXiv: 2510.20477 by the authors.

Figure 1
Figure 1. Comparison of existing pseudo-labeling paradigms and our proposed Bi-CoG. (a) Majority voting from multiple VLMs; (b) Top-k selection based on predicted logits; (c) Bi-CoG dynamically generate accurate and unbiased pseudo-labels based on bi-consistency and model error. thresholds. For instance, GRIP (Menghini, Delworth, and Bach 2023) and UPL (Huang, Chu, and Wei 2022) en￾force class balance by selecting equal sampl… view at source ↗
Figure 2
Figure 2. The overall framework of Bi-CoG’s pseudo-label selection. During the self-training phase, Bi-CoG employs inter [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Pseudo-label accuracy over training iterations [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison between the estimated and true er [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

  1. [9]

    arXiv:arXiv:1212.0402

    UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:arXiv:1212.0402. Wang, A.; and Russakovsky, O

    arXiv

  2. [11]

    InCom- puter Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, 493–510

    Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification. InCom- puter Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, 493–510. Berlin, Heidelberg: Springer-Verlag. ISBN 978-3- 031-19832-8. Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022a. Condi- tional prompt learning for vision...

    2022

  3. [12]

    InInterna- tional Conference on Learning Representations

    The Rich Get Richer: Disparate Impact of Semi-Supervised Learning. InInterna- tional Conference on Learning Representations. Supplementary Material A. Detailed Theoretical Analysis A.1. Proof of Lemma 1 Proof.When the aggregated training data contains noisy samples, letFdenote the hypothesis space,ηthe noise ratio in the training set (η <0.5),ϵ >0the targ...

    1988

  4. [224]

    padding 28 Strong Augmentations ColorJitterRandomly changes the brightness, contrast, saturation, and hue. brightness 0.4 contrast 0.4 saturation 0.1 hue 0.4 RandomGrayscaleConverts the image to grayscale with a probability ofp.p0.2 GaussianBlurApplies Gaussian blur to the image with a probability ofp. kernel size 21 p0.5 Table 6: Details of Weak and Stro...

    2009

  5. [2004]

    Inthe 2024 IEEE Conference on Computer Vision and Pattern Recognition Workshops,

    Learning Gen- erative Visual Models from Few Training Examples: An In- cremental Bayesian Approach Tested on 101 Object Cate- gories. Inthe 2024 IEEE Conference on Computer Vision and Pattern Recognition Workshops,

    2024

  6. [2009]

    InProceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255

    ImageNet: a large-scale hierarchical image database. InProceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255. Fei-Fei, L.; Fergus, R.; and Perona, P

    2009

  7. [2010]

    InProceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, 3485–3492

    Sun database: Large-scale scene recognition from abbey to zoo. InProceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, 3485–3492. Yang, X.; Song, Z.; King, I.; and Xu, Z

    2010

  8. [2012]

    InProceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3498–3505

    Cats and dogs. InProceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3498–3505. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    2012

Show all 13 references
  1. [2013]

    Inthe 2013 IEEE International Conference on Computer Vision Workshops, 554–561

    3D ob- ject representations for fine-grained categorization. Inthe 2013 IEEE International Conference on Computer Vision Workshops, 554–561. Krizhevsky, A.; Hinton, G.; et al

    2013

  2. [2014]

    InPro- ceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 3606–3613

    Describing Textures in the Wild. InPro- ceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 3606–3613. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei- Fei, L

    2014

  3. [2022]

    Jia, C.; Yang, Y .; Xia, Y .; Chen, Y .-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y .-H.; Li, Z.; and Duerig, T

    Unsupervised prompt learning for vision-language models.arXiv preprint arXiv:2204.03649. Jia, C.; Yang, Y .; Xia, Y .; Chen, Y .-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y .-H.; Li, Z.; and Duerig, T

    arXiv

  4. [2023]

    arXiv:arXiv:1306.5151

    Fine-grained visual classification of aircraft. arXiv:arXiv:1306.5151. Menghini, C.; Delworth, A.; and Bach, S

    arXiv

  5. [2024]

    In Al-Onaizan, Y .; Bansal, M.; and Chen, Y .-N., eds.,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 14394–14410

    Vision-Language Model Fine-Tuning via Sim- ple Parameter-Efficient Modification. In Al-Onaizan, Y .; Bansal, M.; and Chen, Y .-N., eds.,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 14394–14410. Miami, Florida, USA: Associa- tion for C...

    2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.