REVIEW 3 major objections 5 minor 2 cited by
Online Gaussian Test-Time Adaptation of Vision-Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A Gaussian likelihood model with one fixed hyper-parameter adapts a vision-language model to a test-time stream, beating two cache-based rivals on 9 of 11 datasets.
desk verdict Solid empirical OTTA paper with a genuinely useful evaluation protocol, but the headline comparison against competitors uses fixed ImageNet-tuned hyperparameters, so the win rate is narrower than it appears. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a class-conditional Gaussian model of normalized visual features with a shared covariance, combined with zero-shot soft labels as priors in a MAP prediction rule. The rule is $p(c_i=k|f_i) \propto p^\nu_{i,k}\, y_{i,k}$, where $\nu$, fixed at 0.05, flattens the Gaussian likelihood so that early, poorly estimated Gaussians do not overwhelm the zero-shot prior. The Gaussian parameters come from a per-class cache of at most eight samples chosen by lowest zero-shot entropy, and the precision matrix uses a regularized estimator when fewer than $4d$ samples are available, switching to the inverse covariance otherwise.
What would settle it
Construct a dataset (or permute the class names in the text encoder) so that the zero-shot model is confident but wrong on a large share of the low-entropy samples, then run OGA: if its mean accuracy falls below the zero-shot baseline, the entropy-selection pseudo-label assumption has failed. The paper's own Aircraft result, where OGA's Expected Tail Accuracy drops to 24.6 against a 24.9 zero-shot score, shows this failure mode can already appear in practice.
Extended reading notes
Core claim
OGA's central claim is that a class-conditional Gaussian model estimated from a small cache of low-entropy samples can improve zero-shot CLIP predictions online, with no dataset-specific tuning. The method selects the samples with the smallest zero-shot entropy, stores up to eight per class, estimates per-class means and a shared covariance (using a regularized precision estimator while the cache holds fewer than $4d$ samples), and forms the posterior $p(c_i=k|f_i) \propto p^\nu_{i,k}\, y_{i,k}$, where $y_{i,k}$ is the zero-shot soft label and $\nu = 0.05$ is fixed. With this rule the paper reports an average accuracy of 68.5% across 11 datasets versus 67.7% for TDA and 67.5% for DMN, and first place on 9 of 11 datasets. The same rule applied on top of CoOp or TaskRes few-shot models improves those models in most settings, and the Expected Tail Accuracy of OGA exceeds the competitors' average accuracy on four datasets.
Load-bearing premise
The method trusts that the samples with the lowest zero-shot entropy are labeled correctly enough to estimate the Gaussian class distributions, so on datasets where the zero-shot model is too weak those pseudo-labels contaminate the estimate and can erase the adaptation gain.
Editorial extensions
If this is right
- If the fixed $\nu = 0.05$ transfers as claimed, online adaptation on a new dataset needs no labeled validation set, removing the main obstacle to cache-based adapters.
- Because OGA's Expected Tail Accuracy beats the average accuracy of TDA and DMN on ImageNet, SUN397, StanfordCars, and Pets, even its worst runs are competitive with the rivals' typical runs.
- Applying OGA on top of CoOp and TaskRes improves the few-shot model in most of the tested 1-shot and 4-shot settings, so offline few-shot learning and online Gaussian adaptation are complementary.
- Keeping only eight samples per class and avoiding augmentations makes OGA a computationally cheap black-box option for streamed data.
- Reporting 100 runs and Expected Tail Accuracy would make future OTTA comparisons harder to win by chance, since measured accuracy varies substantially across runs for every method tested.
Reading between the lines
- The entropy-based cache selection is the weakest link: on datasets where zero-shot accuracy is low, the most confident predictions are often confidently wrong, so a selection rule that mixes entropy with feature diversity could extend OGA to the regimes where it currently loses.
- The paper's own ablation suggests that an adaptive $\nu$ depending on cache state could improve both early-stream behavior and final accuracy, a direction the authors leave for future work.
- Expected Tail Accuracy is independent of the Gaussian machinery and could serve as a general reporting standard for any stochastic evaluation that averages over random runs.
- Because OGA operates only on normalized embeddings, the same MAP rule should transfer to other contrastive vision-language backbones, provided their text and image embeddings live on the same unit sphere; the paper's cross-architecture results are consistent with this expectation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Online Gaussian Adaptation (OGA) for black-box online test-time adaptation of CLIP. OGA maintains a cache of low-entropy samples, models class-conditional feature likelihoods as multivariate Gaussians with a shared covariance, and combines the resulting likelihoods with zero-shot soft labels through a power-weighted posterior rule. The method uses a single fixed hyperparameter ν=0.05 across datasets. The paper reports experiments on 11 datasets with 100 runs per dataset, across multiple backbones and prompt settings, and shows that OGA outperforms TDA and DMN on most datasets. It also introduces Expected Tail Accuracy (ETA), the average accuracy in the worst 10% of runs, and advocates for more runs in OTTA evaluation. Additionally, it demonstrates that OTTA can be applied on top of few-shot methods (CoOp, TaskRes) to yield further gains.
Significance. If confirmed, the paper makes three contributions: a simple and efficient OTTA method with a fixed hyperparameter, a broader evaluation protocol (100 runs, ETA), and a demonstration that OTTA stacks with few-shot adapters. The empirical study is unusually extensive: 11 datasets, 100 runs each, multiple backbones, prompt ensembles, and few-shot settings, with code released. However, the headline claim of outperforming state-of-the-art methods depends on a comparison protocol in which competitors are run with fixed ImageNet-tuned hyperparameters rather than their published per-dataset configurations; the average margin over TDA is only +0.8 points. The choice of ν=0.05 as a global hyperparameter is also presented as a compromise across datasets, not a fully tuning-free derivation.
major comments (3)
- [Section 5 (Competitors) and Supplementary C] The competitors TDA and DMN are run with hyperparameters fixed to their ImageNet-tuned values (Supplementary C), whereas the original papers tune these hyperparameters per dataset. This evaluates constrained variants rather than the published methods. In Table 1a, the average gain of OGA over TDA is only +0.8 points and OGA is second on two datasets. If TDA and DMN were given their per-dataset tuned hyperparameters, the win rate could reverse. The claim that 'OGA outperforms state-of-the-art methods' (Abstract, Section 6) therefore requires either a comparison against the published configurations or a sensitivity analysis demonstrating that OGA retains its advantage across the range of competitor hyperparameters.
- [Section 7, Figure 3] The abstract and Section 1 emphasize fixed hyper-parameters with 'no need for hyper-parameters tuning.' However, Figure 3 shows that the optimal ν varies across datasets (for example, ν=0.010 appears best for Flower102 and DTD, ν=0.025 for StanfordCars, and ν=0.002 for ImageNet), and ν=0.05 is a compromise chosen after observing the benchmark suite. This is selection on the evaluation data, which can inflate reported performance. The paper should either present a validation-based procedure for selecting ν, demonstrate robustness to ν across a wide range, or more carefully phrase the claim as 'a single global hyperparameter set after seeing the benchmarks' rather than 'no need for tuning.'
- [Section 6, Table 1b] On Aircraft, the ETA of OGA (24.6) is below the zero-shot accuracy (24.9), and the same is true for TDA and DMN. This demonstrates that pseudo-label contamination can eliminate the adaptation gain. The paper acknowledges this but does not discuss any safeguard. Since the central claim is consistent improvement, the authors should either propose a simple mechanism to avoid degradation below zero-shot (for example, early stopping when cache quality is poor) or clearly delimit the conditions under which OGA should be applied.
minor comments (5)
- [Section 4, Eq. (8)] The rule is labeled MAP, but ν is applied to the likelihood only, so it is a tempered likelihood posterior rather than a strict MAP estimate. Consider renaming to 'pseudo-Bayesian' consistently to avoid overstating the theoretical grounding.
- [Section 1, Figure 1] The caption states results are averaged over 100 runs, but the figure does not show error bars; please include confidence intervals or refer to Table 1 for the spread.
- [Section 5, Data stream generation] The text says methods are compared on the same 100 runs, but does not specify what varies across runs (for example, random order of test images). Please clarify the source of variability.
- [Section 7, Table 4] The cache size ablation reports only the mean accuracy; adding standard deviations would help assess whether differences across cache sizes are significant.
- [Eq. (11)] The definition of ETA uses an indicator on acc(r) ≤ acc0.1; with ties at the threshold, the tail may not be exactly 10% of runs. Please clarify the handling of ties.
Circularity Check
OGA's derivation is self-contained; the self-citations and the fixed-ν choice are not circular, though the competitor protocol is an evaluation concern.
full rationale
The central derivation is Eqs. (5)-(8): class-conditional Gaussian likelihoods estimated from a cache of low-entropy pseudo-labeled samples are combined with zero-shot soft labels as priors in a MAP rule. The cache-selection rule and the prediction rule do not identify with each other: a sample's zero-shot entropy decides whether it enters the cache, while its posterior is recomputed from the estimated Gaussian means/covariance and the zero-shot prior via Eq. (8). The output is therefore not equal by construction to its inputs; it is a genuine self-training adaptation rule whose failure mode on Aircraft (ETA 24.6 vs zero-shot 24.9) is acknowledged in the paper. The method's mathematical claims do not import a uniqueness theorem or a fitted parameter renamed as a prediction: ν=0.05 is a single global hyperparameter, and its ablation (Figure 3) is reported as a trade-off rather than a per-dataset fit. The self-citations (e.g., [27], [28]) concern related Gaussian-modeling and MTA work and are not used to justify the correctness of OGA's predictions. The skeptic's concern that TDA and DMN are evaluated with fixed ImageNet hyperparameters and without augmentations is an experimental-protocol validity issue, not a circularity in the derivation chain, so it does not raise the circularity score.
Assumptions & free parameters
free parameters (4)
- nu (likelihood exponent) =
0.05
- cache size per class =
8
- Bayes-Ridge threshold =
4d samples
- batch size =
32
assumptions (5)
- domain assumption Class-conditional visual features follow multivariate Gaussian distributions with a shared covariance.
- domain assumption Zero-shot softmax probabilities are valid class priors in Bayes' rule.
- domain assumption Low zero-shot entropy samples have reliable pseudo-labels.
- standard math Kubokawa-Srivastava Bayes-Ridge estimator is a valid precision estimator in high dimensions.
- ad hoc to paper The power-weighting nu in Eq. 8 preserves the ranking intended by MAP.
invented entities (1)
-
Expected Tail Accuracy (ETA)
Cite this review
Pith. "Pith review of Online Gaussian Test-Time Adaptation of Vision-Language Models." pith.science (2026). https://pith.science/paper/JLO6F6CE
@misc{pith2026250104352,
author = {Pith},
title = {Pith review of: Online Gaussian Test-Time Adaptation of Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/JLO6F6CE}},
note = {Machine review of arXiv:2501.04352}
}
read the original abstract
Online test-time adaptation (OTTA) of vision-language models (VLMs) has recently garnered increased attention to take advantage of data observed along a stream to improve future predictions. Unfortunately, existing methods rely on dataset-specific hyperparameters, significantly limiting their adaptability to unseen tasks. In response, we propose Online Gaussian Adaptation (OGA), a novel method that models the likelihoods of visual features using Gaussian distributions and incorporates zero-shot priors into an interpretable Maximum A Posteriori (MAP) estimation framework with fixed hyper-parameters across all datasets. We demonstrate that OGA outperforms state-of-the-art methods on most datasets and runs. Additionally, we show that combining OTTA with popular few-shot techniques (a practical yet overlooked setting in prior research) is highly beneficial. Furthermore, our experimental study reveals that common OTTA evaluation protocols, which average performance over at most three runs per dataset, are inadequate due to the substantial variability observed across runs for all OTTA methods. Therefore, we advocate for more rigorous evaluation practices, including increasing the number of runs and considering additional quantitative metrics, such as our proposed Expected Tail Accuracy (ETA), calculated as the average accuracy in the worst 10% of runs. We hope these contributions will encourage more rigorous and diverse evaluation practices in the OTTA community. Code is available at https://github.com/cfuchs2023/OGA .
Figures
Forward citations
Cited by 2 Pith papers
-
Clustering-based aggregate value regression
The paper proposes AVR-C, a clustered aggregate-value regression method, and claims a bias-variance trade-off theory under model misspecification where the number of clusters controls forecast error.
-
Adapting Vision-Language Models Without Labels: A Comprehensive Survey
A survey that organizes unsupervised vision-language model adaptation by unlabeled-data availability into four paradigms: data-free transfer, domain transfer, episodic test-time, and online test-time adaptation.
Reference graph
Works this paper leans on
-
[1]
Food-101–mining discriminative components with random forests
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014. 5
2014
-
[2]
Describing textures in the wild
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 5
2014
-
[3]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 1, 5
work page 2009
-
[4]
Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004. 5
2004
-
[5]
Clip-adapter: Better vision-language models with feature adapters
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581–595, 2024. 2
work page 2024
-
[6]
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5
2019
-
[7]
Unsupervised prompt learning for vision-language models
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. 2
arXiv 2022
-
[8]
Label propagation for zero-shot classification with vision-language models
Yannis Kalantidis, Giorgos Tolias, et al. Label propagation for zero-shot classification with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23209–23218, 2024. 2, 3
work page 2024
Show all 33 references
-
[9]
Efficient test-time adaptation of vision-language models
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14162–14171, 2024. 2, 3, 4, 5
2024
-
[10]
Enhancing remote sensing vision-language models for zero-shot scene classifi- cation
Karim El Khoury, Maxime Zanella, Beno ˆıt G ´erin, Tiffanie Godelaine, Beno ˆıt Macq, Sa ¨ıd Mahmoudi, Christophe De Vleeschouwer, and Ismail Ben Ayed. Enhancing remote sensing vision-language models for zero-shot scene classifi- cation. arXiv preprint arXiv:2409.00698, 2024. 3
2024 arXiv
-
[11]
3d object representations for fine-grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 5
2013
-
[12]
Estimation of the precision matrix of a singular wishart distribution and its ap- plication in high-dimensional data
Tatsuya Kubokawa and Muni S Srivastava. Estimation of the precision matrix of a singular wishart distribution and its ap- plication in high-dimensional data. Journal of Multivariate Analysis, 99(9):1906–1928, 2008. 4
1906
-
[13]
Swapprompt: Test-time prompt adaptation for vision- language models
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision- language models. Advances in Neural Information Process- ing Systems, 36, 2024. 2, 3
2024
-
[14]
Fine-grained visual classi- fication of aircraft
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 5
2013 arXiv
-
[15]
Transductive zero-shot and few-shot clip
S ´egol`ene Martin, Yunshi Huang, Fereshteh Shakeri, Jean- Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot clip. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 28816–28826, 2024. 2, 3
2024
-
[16]
Automated flower classification over a large number of classes
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 5
2008
-
[17]
Black box few-shot adaptation for vision- language models
Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision- language models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15534–15546,
-
[18]
Cats and dogs
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5
2012
-
[19]
Learning transferable visual models from natural language supervi- sion
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...
2021
-
[20]
Test- time prompt tuning for zero-shot generalization in vision- language models
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022. 2, 3
2022
-
[21]
A closer look at the few-shot adaptation of large vision-language models
Julio Silva-Rodr ´ıguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23681–23690, 2024. 2, 4
2024
-
[22]
Ucf101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 5 9
2012 arXiv
-
[23]
A hard-to-beat baseline for training- free clip-based adaptation
Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training- free clip-based adaptation. In The Twelfth International Con- ference on Learning Representations, 2024. 2, 4
2024
-
[24]
Sun database: Large-scale scene recognition from abbey to zoo
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 5
2010
-
[25]
Task residual for tuning vision-language models
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10899–10909, 2023. 2, 3, 5, 7
2023
-
[26]
Low-rank few-shot adaptation of vision-language models
Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1593–1603, 2024. 2
2024
-
[27]
Maxime Zanella and Ismail Ben Ayed. On the test-time zero- shot generalization of vision-language models: Do we really need prompt learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23783–23793, 2024. 2, 3, 5, 6
2024
-
[28]
Boost- ing vision-language models with transduction
Maxime Zanella, Beno ˆıt G´erin, and Ismail Ben Ayed. Boost- ing vision-language models with transduction. Neural Infor- mation Processing Systems (NeurIPS), 2024. 2, 3, 4, 6
2024
-
[29]
Boosting vision-language mod- els for histopathology classification: Predict all at once
Maxime Zanella, Fereshteh Shakeri, Yunshi Huang, Houda Bahig, and Ismail Ben Ayed. Boosting vision-language mod- els for histopathology classification: Predict all at once. In International Workshop on Foundation Models for General Medical AI, pages 153–162. Springer, 2024. 3
2024
-
[30]
Tip- adapter: Training-free adaption of clip for few-shot classi- fication
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classi- fication. In European conference on computer vision, pages 493–510. Springer, 2022. 2, 3
-
[31]
Dual memory networks: A versa- tile adaptation approach for vision-language models
Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versa- tile adaptation approach for vision-language models. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 28718–28728, 2024. 2, 3, 5
2024
-
[32]
Learning to prompt for vision-language models
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,
-
[2022]
Prompts We show the handcrafted prompts used in the relevant experiments in Table 7
2, 3, 5, 7 10 Online Gaussian Test-Time Adaptation of Vision-Language Models Supplementary Material A. Prompts We show the handcrafted prompts used in the relevant experiments in Table 7. Table 7. Prompt templates for each dataset. (a) Prompt templates used in the experiments ...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.