Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors
Reviewed by Pithpith:7WCTNWHIopen to challenge →
read the original abstract
Test-time adaptation (TTA) fine-tunes pre-trained deep neural networks for unseen test data. The primary challenge of TTA is limited access to the entire test dataset during online updates, causing error accumulation. To mitigate it, TTA methods have utilized the model output's entropy as a confidence metric that aims to determine which samples have a lower likelihood of causing error. Through experimental studies, however, we observed the unreliability of entropy as a confidence metric for TTA under biased scenarios and theoretically revealed that it stems from the neglect of the influence of latent disentangled factors of data on predictions. Building upon these findings, we introduce a novel TTA method named Destroy Your Object (DeYO), which leverages a newly proposed confidence metric named Pseudo-Label Probability Difference (PLPD). PLPD quantifies the influence of the shape of an object on prediction by measuring the difference between predictions before and after applying an object-destructive transformation. DeYO consists of sample selection and sample weighting, which employ entropy and PLPD concurrently. For robust adaptation, DeYO prioritizes samples that dominantly incorporate shape information when making predictions. Our extensive experiments demonstrate the consistent superiority of DeYO over baseline methods across various scenarios, including biased and wild. Project page is publicly available at https://whitesnowdrop.github.io/DeYO/.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
Private and Stable Test-Time Adaptation with Differential Privacy
Differential privacy versions of TTA methods achieve privacy on ImageNet-C with small accuracy cost and can improve stability via clipping in continual settings.
-
Neural Collapse in Test-Time Adaptation
Sample-wise neural collapse reveals that feature-classifier misalignment drives TTA degradation under shifts, which NCTTA corrects via hybrid geometric-predictive targets.
-
Test-Time Distillation for Continual Model Adaptation
CoDiRe blends VLM and target model predictions via MSP-based weighting and Optimal Transport rectification to enable stable continual test-time adaptation, outperforming CoTTA by 10.55% on ImageNet-C at 48% of the com...
-
Multi-Hypothesis Test-Time Adaptation to Mitigate Underspecification
A multi-level diversification wrapper for test-time adaptation that treats entropy minimization as multi-hypothesis inference to reduce underspecification and improve robustness by 1-4%.
-
DOME: Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation
DOME learns sample-specific domain variables from sparse supervision via vision-language models and a sparse domain bank to improve test-time adaptation performance.
-
Sample-wise Targeted Adversarial Attacks on Test-time Adaptation
Proposes meta-learning attack with priority-aware gradient alignment for sample-wise targeted attacks on TTA that maintain label distribution consistency with no-attack baseline.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.