{"id":"e6731b6f-7b27-4ca8-8c77-25f63c640623","arxiv_id":"2501.04352","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OGA, a Gaussian-based online test-time adaptation method for vision-language models, edges out prior methods on most benchmarks with one fixed hyperparameter, and advocates more rigorous multi-run evaluation.","lead":"Online Gaussian Adaptation (OGA) improves vision-language model predictions on streaming test data by modeling visual features with Gaussian distributions and combining them with zero-shot priors. It reports small accuracy gains over prior online test-time adaptation methods on most of 11 benchmarks, and proposes a 100-run evaluation protocol with a tail-risk metric, Expected Tail Accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OGA's claimed advantage may hinge on comparing against competitors stripped of their per-dataset hyperparameters and augmentations; under native configurations the win rate could reverse.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper's empirical case is strong but needs qualification. The reader's weakest_assumption focuses on pseudo-label contamination in low-accuracy regimes. I find this concern real but less load-bearing than the comparison protocol: the paper explicitly admits the Aircraft tail failure, and all cache-based competitors also rely on pseudo-labels, so this issue does not single out OGA or overturn its comparative win. The more load-bearing assumption is that TDA and DMN are fairly represented. Section 5 states competitors are run without augmentations for equal compute, and Supplementary C reveals their hyperparameters are fixed to ImageNet-tuned values rather than the per-dataset values used in their original papers. Since the central claim is 'outperforms state-of-the-art,' the appropriate comparison includes the competitors' native configurations. The fact that Figure 3 shows dataset-dependent optimal nu also means the 'fixed hyperparameter' claim is not as clean as stated: 0.05 may be a benchmark-level selection rather than a principled default. These issues do not warrant rejection because OGA's wins are consistent across backbones, prompts, and few-shot settings, and the requested experiments are readily runnable from released code. If the native-configuration comparison still shows OGA ahead on most datasets, the central claim survives; if not, it needs substantial qualification. Thus the conditional verdict should stand unchanged.","tokens_in":17681,"tokens_out":12894,"duration_ms":135331,"concrete_test":"Run TDA and DMN on the same 100 streams with their original per-dataset hyperparameters and any augmentations their released code uses, and compare against OGA with nu=0.05. If OGA no longer achieves higher average accuracy on at least 9 of 11 datasets, or if its ETA advantages on ImageNet, SUN397, StanfordCars, and Pets disappear, the headline claim must be qualified to apply only against fixed-hyperparameter, no-augmentation competitors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 is the load-bearing evidence for the central claim that OGA outperforms state-of-the-art OTTA methods. However, Section 5 and Supplementary C state that TDA and DMN are run with the hyperparameters tuned for ImageNet, fixed across all datasets, and without the augmentations some of them use. The original TDA/DMN papers tune hyperparameters per dataset, so this protocol evaluates constrained variants of the competitors, not the published methods. OGA's single nu=0.05 is also not shown to be truly tuning-free: Figure 3 indicates different nu values are optimal on different datasets, so 0.05 appears to be a compromise chosen after observing the benchmark suite. Because OGA's average gain over TDA is only +0.8 points and OGA is second on two datasets, a modest shift in competitor configuration could erase the 9/11 win rate. The pseudo-label contamination flagged by the reader is real but less decisive: it is acknowledged by the authors (Aircraft ETA falls below zero-shot) and affects all cache-based methods; OGA still wins on Aircraft. The comparison protocol is the assumption whose failure would most directly invalidate the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Online Gaussian Adaptation (OGA) for black-box online test-time adaptation of CLIP. OGA maintains a cache of low-entropy samples, models class-conditional feature likelihoods as multivariate Gaussians with a shared covariance, and combines the resulting likelihoods with zero-shot soft labels through a power-weighted posterior rule. The method uses a single fixed hyperparameter ν=0.05 across datasets. The paper reports experiments on 11 datasets with 100 runs per dataset, across multiple backbones and prompt settings, and shows that OGA outperforms TDA and DMN on most datasets. It also introduces Expected Tail Accuracy (ETA), the average accuracy in the worst 10% of runs, and advocates for more runs in OTTA evaluation. Additionally, it demonstrates that OTTA can be applied on top of few-shot methods (CoOp, TaskRes) to yield further gains.","tokens_in":17921,"tokens_out":5949,"duration_ms":53332,"significance":"If confirmed, the paper makes three contributions: a simple and efficient OTTA method with a fixed hyperparameter, a broader evaluation protocol (100 runs, ETA), and a demonstration that OTTA stacks with few-shot adapters. The empirical study is unusually extensive: 11 datasets, 100 runs each, multiple backbones, prompt ensembles, and few-shot settings, with code released. However, the headline claim of outperforming state-of-the-art methods depends on a comparison protocol in which competitors are run with fixed ImageNet-tuned hyperparameters rather than their published per-dataset configurations; the average margin over TDA is only +0.8 points. The choice of ν=0.05 as a global hyperparameter is also presented as a compromise across datasets, not a fully tuning-free derivation.","major_comments":[{"comment":"The competitors TDA and DMN are run with hyperparameters fixed to their ImageNet-tuned values (Supplementary C), whereas the original papers tune these hyperparameters per dataset. This evaluates constrained variants rather than the published methods. In Table 1a, the average gain of OGA over TDA is only +0.8 points and OGA is second on two datasets. If TDA and DMN were given their per-dataset tuned hyperparameters, the win rate could reverse. The claim that 'OGA outperforms state-of-the-art methods' (Abstract, Section 6) therefore requires either a comparison against the published configurations or a sensitivity analysis demonstrating that OGA retains its advantage across the range of competitor hyperparameters.","section":"Section 5 (Competitors) and Supplementary C"},{"comment":"The abstract and Section 1 emphasize fixed hyper-parameters with 'no need for hyper-parameters tuning.' However, Figure 3 shows that the optimal ν varies across datasets (for example, ν=0.010 appears best for Flower102 and DTD, ν=0.025 for StanfordCars, and ν=0.002 for ImageNet), and ν=0.05 is a compromise chosen after observing the benchmark suite. This is selection on the evaluation data, which can inflate reported performance. The paper should either present a validation-based procedure for selecting ν, demonstrate robustness to ν across a wide range, or more carefully phrase the claim as 'a single global hyperparameter set after seeing the benchmarks' rather than 'no need for tuning.'","section":"Section 7, Figure 3"},{"comment":"On Aircraft, the ETA of OGA (24.6) is below the zero-shot accuracy (24.9), and the same is true for TDA and DMN. This demonstrates that pseudo-label contamination can eliminate the adaptation gain. The paper acknowledges this but does not discuss any safeguard. Since the central claim is consistent improvement, the authors should either propose a simple mechanism to avoid degradation below zero-shot (for example, early stopping when cache quality is poor) or clearly delimit the conditions under which OGA should be applied.","section":"Section 6, Table 1b"}],"minor_comments":[{"comment":"The rule is labeled MAP, but ν is applied to the likelihood only, so it is a tempered likelihood posterior rather than a strict MAP estimate. Consider renaming to 'pseudo-Bayesian' consistently to avoid overstating the theoretical grounding.","section":"Section 4, Eq. (8)"},{"comment":"The caption states results are averaged over 100 runs, but the figure does not show error bars; please include confidence intervals or refer to Table 1 for the spread.","section":"Section 1, Figure 1"},{"comment":"The text says methods are compared on the same 100 runs, but does not specify what varies across runs (for example, random order of test images). Please clarify the source of variability.","section":"Section 5, Data stream generation"},{"comment":"The cache size ablation reports only the mean accuracy; adding standard deviations would help assess whether differences across cache sizes are significant.","section":"Section 7, Table 4"},{"comment":"The definition of ETA uses an indicator on acc(r) ≤ acc0.1; with ties at the threshold, the tail may not be exactly 10% of runs. Please clarify the handling of ties.","section":"Eq. (11)"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the empirical study is substantial. My main concern is the comparison protocol, which I have raised as a major comment. The ETA metric is a useful addition, though similar to lower-tail conditional expectations used in finance. The code release is a plus. I recommend major revision to address the competitor configuration issue and the ν selection concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-run empirical paper that makes a modest but real contribution to VLM online test-time adaptation, and its evaluation protocol is worth copying even if you never use OGA. The method itself is a clean combination of Gaussian likelihood modeling from the few-shot/transductive literature with zero-shot priors in a MAP rule, and it ships one scalar hyperparameter, nu=0.05, fixed across all datasets. The extra experiments on top of CoOp and TaskRes are a practical setting that previous OTTA work ignored, and the results there are more uniformly positive than in the zero-shot case.\n\nThe strongest part is the evaluation discipline: 100 runs per dataset, same streams for all methods, multiple backbones and prompts, and a new metric, Expected Tail Accuracy, that captures worst-case outcomes. That is a real contribution to methodology—the standard of 3 runs is clearly inadequate given the variance they show.\n\nNow the soft spots. The headline claim that OGA beats TDA and DMN on 9 of 11 datasets relies on running those competitors with fixed ImageNet-tuned hyperparameters, not the per-dataset values in their original papers. The authors are transparent about this in Supplementary C, and there is a defensible logic: in a true TTA scenario you don't have labels to tune per dataset. But it means the comparison is against constrained variants, and the average margin is only +0.8 points. Under the competitors' native per-dataset configuration, the win rate could shrink or reverse. This is the load-bearing weak point, and the referee should push for a supplementary table with per-dataset tuned competitors. If that reverses the ranking, the claim should soften; if not, the robustness story is much stronger.\n\nThe fixed nu=0.05 is also a compromise after looking at the benchmark suite—Figure 3 shows different values are optimal on different datasets. That's not fatal, but the paper should say 'a single value chosen on the full suite' rather than implying it came from a principle. The pseudo-label contamination on Aircraft is acknowledged and affects all cache-based methods, so I don't treat it as a differentiator.\n\nWho this is for: anyone working on black-box VLM adaptation, OTTA, or evaluation metrics for stochastic test-time procedures. The ETA metric and the 100-run protocol are worth adopting on their own. Serious referees should engage with this paper; the competitor-configuration question is a revision issue, not a rejection issue.","headline":"Solid empirical OTTA paper with a genuinely useful evaluation protocol, but the headline comparison against competitors uses fixed ImageNet-tuned hyperparameters, so the win rate is narrower than it appears.","tokens_in":21,"tokens_out":2637,"would_cite":true,"duration_ms":67938,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Gaussian likelihood model with one fixed hyper-parameter adapts a vision-language model to a test-time stream, beating two cache-based rivals on 9 of 11 datasets.","keywords":["online test-time adaptation","vision-language models","Gaussian likelihood modeling","maximum a posteriori estimation","cache-based adaptation","Expected Tail Accuracy","CLIP","few-shot adaptation"],"falsifier":"Construct a dataset (or permute the class names in the text encoder) so that the zero-shot model is confident but wrong on a large share of the low-entropy samples, then run OGA: if its mean accuracy falls below the zero-shot baseline, the entropy-selection pseudo-label assumption has failed. The paper's own Aircraft result, where OGA's Expected Tail Accuracy drops to 24.6 against a 24.9 zero-shot score, shows this failure mode can already appear in practice.","tokens_in":17488,"feed_emoji":"📈","tokens_out":12522,"duration_ms":103951,"temperature":0.7,"pith_summary":"The paper proposes Online Gaussian Adaptation (OGA), a method for adapting a vision-language model such as CLIP to an unlabeled data stream without per-dataset tuning. OGA models the visual-feature distribution of each class as a multivariate Gaussian, combines those likelihoods with the model's own zero-shot predictions treated as priors in a Maximum A Posteriori rule, and keeps a single fixed weighting hyper-parameter, $\\nu = 0.05$, across all datasets. On 11 standard benchmarks the paper reports that OGA outperforms two recent online adapters, TDA and DMN, on 9 datasets in average accuracy, and that the average accuracy of its worst 10% of runs, a quantity the paper calls Expected Tail Accuracy, exceeds the competitors' average accuracy on ImageNet, SUN397, StanfordCars, and Pets. The paper also argues that current three-run evaluation protocols are too noisy and that applying online adaptation on top of few-shot prompt or adapter methods is broadly beneficial.","feed_headline":"One fixed knob: Gaussian adapter wins 9 of 11 datasets","feed_subtitle":"Online test-time adaptation without per-dataset tuning, plus a worst-case metric that exposes run-to-run risk.","key_machinery":"The machinery is a class-conditional Gaussian model of normalized visual features with a shared covariance, combined with zero-shot soft labels as priors in a MAP prediction rule. The rule is $p(c_i=k|f_i) \\propto p^\\nu_{i,k}\\, y_{i,k}$, where $\\nu$, fixed at 0.05, flattens the Gaussian likelihood so that early, poorly estimated Gaussians do not overwhelm the zero-shot prior. The Gaussian parameters come from a per-class cache of at most eight samples chosen by lowest zero-shot entropy, and the precision matrix uses a regularized estimator when fewer than $4d$ samples are available, switching to the inverse covariance otherwise.","core_discovery":"OGA's central claim is that a class-conditional Gaussian model estimated from a small cache of low-entropy samples can improve zero-shot CLIP predictions online, with no dataset-specific tuning. The method selects the samples with the smallest zero-shot entropy, stores up to eight per class, estimates per-class means and a shared covariance (using a regularized precision estimator while the cache holds fewer than $4d$ samples), and forms the posterior $p(c_i=k|f_i) \\propto p^\\nu_{i,k}\\, y_{i,k}$, where $y_{i,k}$ is the zero-shot soft label and $\\nu = 0.05$ is fixed. With this rule the paper reports an average accuracy of 68.5% across 11 datasets versus 67.7% for TDA and 67.5% for DMN, and first place on 9 of 11 datasets. The same rule applied on top of CoOp or TaskRes few-shot models improves those models in most settings, and the Expected Tail Accuracy of OGA exceeds the competitors' average accuracy on four datasets.","pith_inferences":["The entropy-based cache selection is the weakest link: on datasets where zero-shot accuracy is low, the most confident predictions are often confidently wrong, so a selection rule that mixes entropy with feature diversity could extend OGA to the regimes where it currently loses.","The paper's own ablation suggests that an adaptive $\\nu$ depending on cache state could improve both early-stream behavior and final accuracy, a direction the authors leave for future work.","Expected Tail Accuracy is independent of the Gaussian machinery and could serve as a general reporting standard for any stochastic evaluation that averages over random runs.","Because OGA operates only on normalized embeddings, the same MAP rule should transfer to other contrastive vision-language backbones, provided their text and image embeddings live on the same unit sphere; the paper's cross-architecture results are consistent with this expectation."],"forward_implications":["If the fixed $\\nu = 0.05$ transfers as claimed, online adaptation on a new dataset needs no labeled validation set, removing the main obstacle to cache-based adapters.","Because OGA's Expected Tail Accuracy beats the average accuracy of TDA and DMN on ImageNet, SUN397, StanfordCars, and Pets, even its worst runs are competitive with the rivals' typical runs.","Applying OGA on top of CoOp and TaskRes improves the few-shot model in most of the tested 1-shot and 4-shot settings, so offline few-shot learning and online Gaussian adaptation are complementary.","Keeping only eight samples per class and avoiding augmentations makes OGA a computationally cheap black-box option for streamed data.","Reporting 100 runs and Expected Tail Accuracy would make future OTTA comparisons harder to win by chance, since measured accuracy varies substantially across runs for every method tested."],"supporting_citations":[{"why":"TDA is the main cache-based online baseline that OGA is compared against and whose entropy-filtering strategy OGA shares.","marker":"[9]"},{"why":"DMN is the second online cache-based competitor, providing the dual-memory baseline.","marker":"[31]"},{"why":"Tip-Adapter supplies the cache-model prediction rule that TDA and DMN build on and that OGA replaces with a MAP rule.","marker":"[30]"},{"why":"This work motivates the Gaussian modeling of CLIP features in training-free adaptation.","marker":"[23]"},{"why":"Transductive Gaussian modeling of visual features is the other source of the balanced-mixture likelihood used by OGA.","marker":"[28]"},{"why":"This reference supplies the regularized estimator of the precision matrix used when the cache is too small for a stable inverse.","marker":"[12]"},{"why":"CLIP provides the zero-shot model, the embedding space, and the soft-label priors that OGA combines with Gaussian likelihoods.","marker":"[19]"},{"why":"This study documents the per-dataset hyper-parameter sensitivity of cache-based few-shot adapters, motivating OGA's fixed-parameter design.","marker":"[21]"},{"why":"CoOp is the few-shot prompt-tuning method used to test OGA on top of adapted models.","marker":"[32]"},{"why":"TaskRes is the few-shot adapter method used to test OGA on top of adapted models.","marker":"[25]"}],"fun_headline_variants":["Online Gaussian Adaptation: zero tuning, 9/11 wins","No per-dataset knobs: OGA tops 9 of 11 datasets","Test-time adaptation with fixed hyperparams, better accuracy","Gaussian cache adapts CLIP on the fly without tuning","Expected Tail Accuracy exposes run variability in OTTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method trusts that the samples with the lowest zero-shot entropy are labeled correctly enough to estimate the Gaussian class distributions, so on datasets where the zero-shot model is too weak those pseudo-labels contaminate the estimate and can erase the adaptation gain.","fun_headline_variants_meta":{"raw":{"variants":["Online Gaussian Adaptation: zero tuning, 9/11 wins","No per-dataset knobs: OGA tops 9 of 11 datasets","Test-time adaptation with fixed hyperparams, better accuracy","Gaussian cache adapts CLIP on the fly without tuning","Expected Tail Accuracy exposes run variability in OTTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1408,"prompt_tokens":1008,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":624,"tokens_out":400,"duration_ms":4559,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:34:52.050447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a dataset (or permute the class names in the text encoder) so that the zero-shot model is confident but wrong on a large share of the low-entropy samples, then run OGA: if its mean accuracy falls below the zero-shot baseline, the entropy-selection pseudo-label assumption has failed. The paper's own Aircraft result, where OGA's Expected Tail Accuracy drops to 24.6 against a 24.9 zero-shot score, shows this failure mode can already appear in practice.","supporting_citations":[{"cited_title":"Efficient test-time adaptation of vision-language models","cited_arxiv_id":null,"evidence_quote":"TDA is the main cache-based online baseline that OGA is compared against and whose entropy-filtering strategy OGA shares."},{"cited_title":"Dual memory networks: A versa- tile adaptation approach for vision-language models","cited_arxiv_id":null,"evidence_quote":"DMN is the second online cache-based competitor, providing the dual-memory baseline."},{"cited_title":"Tip- adapter: Training-free adaption of clip for few-shot classi- fication","cited_arxiv_id":null,"evidence_quote":"Tip-Adapter supplies the cache-model prediction rule that TDA and DMN build on and that OGA replaces with a MAP rule."},{"cited_title":"A hard-to-beat baseline for training- free clip-based adaptation","cited_arxiv_id":null,"evidence_quote":"This work motivates the Gaussian modeling of CLIP features in training-free adaptation."},{"cited_title":"Boost- ing vision-language models with transduction","cited_arxiv_id":null,"evidence_quote":"Transductive Gaussian modeling of visual features is the other source of the balanced-mixture likelihood used by OGA."},{"cited_title":"Estimation of the precision matrix of a singular wishart distribution and its ap- plication in high-dimensional data","cited_arxiv_id":null,"evidence_quote":"This reference supplies the regularized estimator of the precision matrix used when the cache is too small for a stable inverse."},{"cited_title":"A closer look at the few-shot adaptation of large vision-language models","cited_arxiv_id":null,"evidence_quote":"This study documents the per-dataset hyper-parameter sensitivity of cache-based few-shot adapters, motivating OGA's fixed-parameter design."},{"cited_title":"Task residual for tuning vision-language models","cited_arxiv_id":null,"evidence_quote":"TaskRes is the few-shot adapter method used to test OGA on top of adapted models."}],"review_version":1}