REVIEW 3 major objections 5 minor 29 references
Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A single adaptive entropy update can separate known from unknown classes during multimodal test-time adaptation.
desk verdict Useful first benchmark and method for multimodal open-set TTA, with a solid empirical story and one real soft spot: the fixed entropy threshold that the whole sign-flipping mechanism leans on. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the adaptive sample weight $W_{\text{ada}} = \tanh(\beta (H(\hat{p})-\alpha))$, where $H(\hat{p})$ is prediction entropy normalized to $[0,1]$ and $\alpha,\beta$ are fixed hyperparameters. Applied as a multiplier to the entropy loss (UAE) and to a cross-modal prediction-discrepancy loss (AMP), this one expression turns the update into an automatic sign switch: entropy is minimized when $H(\hat{p})<\alpha$ and maximized when $H(\hat{p})>\alpha$, with ambiguous samples near the threshold receiving near-zero weight. A negative-entropy diversity term is added to prevent collapsed predictions. This mechanism is what the paper claims converts the observed entropy-difference correlation into an actionable online update rule.
What would settle it
One decisive test would be to construct a target domain where unknown samples are initially very confident (entropy below $\alpha=0.8$): AEO's Tanh sign would then treat them as known and minimize their entropy, so AUROC and FPR95 should worsen relative to the source model in one adaptation pass.
Extended reading notes
Core claim
The paper's central claim is that MM-OSTTA performance strongly correlates with the difference between average prediction entropy of unknown and known samples in the target domain, and that the right test-time objective is to amplify this difference rather than minimize entropy globally. AEO does this with an adaptive weight based on normalized entropy: samples with entropy above a fixed threshold are treated as unknown and pushed toward higher entropy; samples below the threshold are treated as known and pushed toward lower entropy; samples near the threshold are down-weighted because their status is ambiguous. A second term adapts the prediction discrepancy across modalities, encouraging diverse predictions for unknown samples and consistent predictions for known ones. The paper reports that this combination separates the score distributions of known and unknown classes during online adaptation, leading to large gains in unknown-class detection, and that the gains persist in long-term and continual settings where other TTA baselines degrade.
Load-bearing premise
The method assumes that a fixed entropy cutoff (chosen before seeing target data) separates unknown from known samples in every target domain and every modality; if a domain's unknowns have entropy below that cutoff, the sign of the update reverses and the separation collapses.
Editorial extensions
If this is right
- If the entropy-gap claim is correct, any future multimodal TTA method should be evaluated not only on accuracy but on whether it widens the known-unknown entropy gap, since that gap is the paper's proposed driver of unknown-class detection.
- AEO's reported robustness across video, audio, optical flow, LiDAR and camera suggests the same update rule transfers across tasks as different as action recognition and 3D semantic segmentation.
- In long-term and continual settings, where repeated adaptation usually accumulates error, the paper reports that AEO continues to improve (for example, its H-score increases after ten rounds on HAC), which directly addresses deployment scenarios where domains drift indefinitely.
- The paper's result implies that open-set handling does not require a separate OOD detector trained with auxiliary outliers; an entropy-based adaptive update at test time is claimed to be sufficient.
- Baselines that filter low-confidence samples or minimize entropy globally are claimed to be insufficient for MM-OSTTA, because they leave the entropy gap unchanged or shrink it.
Reading between the lines
- The fixed threshold $\alpha=0.8$ is the part most likely to need per-domain tuning; a natural extension the paper does not test is estimating the threshold from the running statistics of the target batch.
- Because the paper shows a correlation between entropy gap and FPR95, one cheap diagnostic for deployed systems would be to monitor the entropy gap online as a proxy for open-set detection health, without labels.
- The M-modal loss formulas in the appendix suggest AEO extends beyond three modalities, but the paper only reports experiments with up to three modality streams; testing four or more fused sensors is an open step.
- If the entropy gap is indeed causal, then a complementary test would be to fix the gap artificially (for example, by reweighting the loss to a target gap value) and check that FPR95 tracks it; the paper does not run this intervention.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Adaptive Entropy-aware Optimization (AEO) for multimodal open-set test-time adaptation (MM-OSTTA), a new task setting in which a source-trained multimodal model must adapt online to a target stream that contains both domain-shifted known-class samples and novel (unknown) classes. AEO is motivated by an observed correlation between the average entropy difference of known vs. unknown samples and unknown-detection performance (FPR95). It consists of two losses: Unknown-aware Adaptive Entropy Optimization (UAE), which weights each sample by Tanh(β(H(p)−α)) and maximizes/minimizes entropy accordingly, and Adaptive Modality Prediction Discrepancy Optimization (AMP), which adaptively enforces cross-modal consistency for known samples and discrepancy for unknowns. The paper builds a benchmark from EPIC-Kitchens, HAC, Kinetics-100-C, and nuScenes, covering action recognition and 3D semantic segmentation across five modalities, and reports consistent improvements over TTA and open-set TTA baselines, including long-term and continual adaptation settings.
Significance. If the claimed mechanism holds, AEO is a practically valuable contribution: it is the first explicit MM-OSTTA framework, uses a simple online update rule, and shows consistent gains in known-class accuracy and unknown-class detection across multiple tasks and modalities. The paper ships code, includes ablations for each loss term, sensitivity analyses for hyperparameters, and tests across architectures and pre-training strategies. The main reservations concern the fixed threshold α that governs the sign of the update, the oracle-balanced batching protocol, and the unquantified correlation claim; these limit the currently supported strength of the central claim but are addressable with additional experiments and analysis.
major comments (3)
- [Sec. 3.2, Eqs. (3)-(4), Appendix B.2, Fig. 5] The sign of Wada = Tanh(β(H(p) − α)) flips at the fixed threshold α = 0.8; the intended entropy amplification for unknown samples requires H(p_unknown) > α during adaptation. The paper fixes α before seeing any target data and provides sensitivity analysis only on HAC (Fig. 5). Several Source rows in Tables 1–3 have FPR95 above 90 (e.g., D1→D2 in Table 1, Gaussian(v)+Gaussian(a) in Table 3), and Appendix C.13.4 acknowledges that initial score distributions of known and unknown samples are often closely aligned. The paper does not report, for any target domain or modality combination, the fraction of unknown samples whose normalized entropy lies above α before or during adaptation. If this fraction is not close to 1, the UAE update minimizes rather than maximizes entropy for a substantial part of the unknown stream, which is the opposite of the claimed mechanism. The central claim that one fixed update rule amplifies the known/unknown entropy gap across all benchmark conditions is therefore not established; the authors should either calibrate α per domain/modality or provide per-domain evidence that the unknown entropy distribution stays above α online.
- [Appendix B.2 and Table 10] The evaluation protocol constructs mini-batches with equal numbers of known and unknown samples. This requires oracle knowledge of the known/unknown status of each incoming sample, which contradicts the paper's stated setting of an unlabeled target stream (Sec. 2). Table 10 varies the unknown ratio from 20% to 80%, but it still assumes the ratio is known so that the batches can be constructed. In a truly unlabeled online stream, the model cannot balance batches in this way, and the natural ratio may be highly skewed or change over time. The reported gains are thus demonstrated under a favorable batching protocol. The authors should explicitly describe this protocol in the main text, justify it as a standard evaluation choice, and additionally evaluate AEO on a natural (unbalanced) stream where batches are drawn without using any known/unknown labels.
- [Sec. 3.1 and Fig. 2] The central motivating observation—that the entropy difference between known and unknown samples 'strongly correlates' with MM-OSTTA performance—is supported only by a visual scatter plot with no correlation coefficient, number of points, or statistical test. Since this correlation is the design principle of the AEO framework, the paper should report a quantitative correlation analysis (e.g., Pearson or Spearman with confidence intervals) over the domain-shift scenarios shown. In addition, because the entropy used in the analysis and the MSP score used to compute FPR95 are both functions of the same softmax distribution, the authors should clarify how the reported correlation is not a near-tautological consequence of using two closely related quantities.
minor comments (5)
- [Sec. 4.1, Eq. (9)] The H-score uses (1 − FPR95) in a harmonic mean with accuracy and AUROC; a brief explanation of why this particular aggregate is chosen, and how it behaves when FPR95 is near 1, would improve readability.
- [Appendix B.2] The hyperparameters α = 0.8 and β = 4.0 are used throughout the main experiments but appear only in the appendix; consider stating them in the main text or in a hyperparameter table.
- [Tables 1–3 and Appendix C.9] The main results are single-run numbers; three-seed statistics are provided for only one HAC setting in the appendix. Reporting mean and standard deviation in the main tables, or at least stating that single runs are shown, would improve reproducibility and make the magnitude of the gains easier to interpret.
- [Throughout] There are several typographical errors and awkward phrasings (e.g., 'uncertainy' in Sec. 3.3, 'adaptaion' in the Table 11 caption, 'enhence' in Sec. 3.1); a careful proofreading pass is recommended.
- [Sec. 4.2] Phrases such as 'improving the Source baseline H-score by 22.07%' mix absolute percentage-point gains with relative improvements; please specify which quantity is meant (e.g., 'by 22.07 points' or 'by 64.7% relatively').
Circularity Check
No significant circularity: the entropy-difference motivation is empirical, the optimized loss is not identical to the evaluation metric, and the self-citations are not load-bearing.
full rationale
The paper's core motivating observation (Sec. 3.1) is an empirical correlation between the entropy difference of known and unknown samples and FPR95, computed with oracle labels on a pre-trained model. This is not a circular definition: MM-OSTTA performance is measured by FPR95/AUROC/Acc/H-score, which depend on the MSP score in Eq. (2), not on the entropy-gap quantity itself. The proposed losses in Eqs. (3)-(8) deliberately optimize entropy separation, but the reported evaluation metrics are external to the training objective; no evaluation term is a component of LAEO. The detection rule uses MSP, not entropy, so the method's output is not identical to its training signal by construction. The fixed-threshold alpha robustness concern (sign reversal when unknown-sample entropy falls below alpha) is a legitimate correctness risk, but it concerns whether the heuristic works, not whether the derivation is circular. Self-citations (e.g., the HAC dataset and SimMMDG pretraining from Dong et al. 2023) are used as experimental assets or ablations, not as load-bearing justification of the method's principle, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The paper contains no step where a fitted parameter is renamed as a prediction, and no claim reduces equationally to its own inputs. Therefore the analysis is self-contained against the stated benchmark and merits a circularity score of 0.
Assumptions & free parameters
free parameters (4)
- alpha =
0.8
- beta =
4.0
- gamma1 =
0.1
- gamma2 =
0.1
assumptions (5)
- domain assumption Unknown test samples have higher prediction entropy than known samples after adaptation.
- ad hoc to paper A single global threshold alpha=0.8 separates known from unknown samples across all target domains and modalities.
- domain assumption Target mini-batches contain both known and unknown samples in a balanced 50/50 proportion during evaluation.
- domain assumption The entropy-difference versus FPR95 correlation observed on EPIC-Kitchens transfers to HAC, Kinetics-100-C, and nuScenes.
- domain assumption Updating only the last layer of each modality encoder plus the classifier is sufficient for MM-OSTTA.
Cite this review
Pith. "Pith review of Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization." pith.science (2026). https://pith.science/paper/3O2GWOJU
@misc{pith2026250113924,
author = {Pith},
title = {Pith review of: Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/3O2GWOJU}},
note = {Machine review of arXiv:2501.13924}
}
read the original abstract
Test-time adaptation (TTA) has demonstrated significant potential in addressing distribution shifts between training and testing data. Open-set test-time adaptation (OSTTA) aims to adapt a source pre-trained model online to an unlabeled target domain that contains unknown classes. This task becomes more challenging when multiple modalities are involved. Existing methods have primarily focused on unimodal OSTTA, often filtering out low-confidence samples without addressing the complexities of multimodal data. In this work, we present Adaptive Entropy-aware Optimization (AEO), a novel framework specifically designed to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA) for the first time. Our analysis shows that the entropy difference between known and unknown samples in the target domain strongly correlates with MM-OSTTA performance. To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP). These components enhance the ability of model to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. To thoroughly evaluate our proposed methods in the MM-OSTTA setting, we establish a new benchmark derived from existing datasets. This benchmark includes two downstream tasks and incorporates five modalities. Extensive experiments across various domain shift situations demonstrate the efficacy and versatility of the AEO framework. Additionally, we highlight the strong performance of AEO in long-term and continual MM-OSTTA settings, both of which are challenging and highly relevant to real-world applications. Our source code is available at https://github.com/donghao51/AEO.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[2]
Sensitivity to overlapping score distributions. In the challenging open-set setting, the initial score distributions of known and unknown samples are often closely aligned and difficult to distinguish, as 25 Published as a conference paper at ICLR 2025 shown in Fig. 6 (a). In such cases, SAR and EATA could potentially minimize the entropy of unknown sampl...
work page 2025
-
[5]
18 Published as a conference paper at ICLR 2025 Source Tent SAR OSTTA UniEnt READ AEO (ours) H-score 37.30 16.90 16.11 28.14 45.38 59.1160.56 Table 13: Average H-score on all 36 possible combinations of corruptions. Kinetics-100-C Acc↑ FPR95↓ AUROC↑ H-score↑ Source 61.06 73.09 68.82 43.51 UniEnt 66.87 59.55 80.31 56.37 READ 68.65 52.52 82.31 62.44 AEO (Ou...
work page 2025
-
[7]
Dpu: Dynamic prototype updating for multimodal out-of-distribution detection
Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang, Zhengzhong Tu, and Yue Zhao. Dpu: Dynamic prototype updating for multimodal out-of-distribution detection. arXiv preprint arXiv:2411.08227,
-
[8]
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,
-
[10]
Entropic open-set active learning
12 Published as a conference paper at ICLR 2025 Bardia Safaei, VS Vibashan, Celso M de Melo, and Vishal M Patel. Entropic open-set active learning. In AAAI,
work page 2025
-
[13]
Robust TTA methods (Niu et al., 2022; Zhou et al.,
update specific model parameters using incoming test samples based on unsupervised objectives such as entropy minimization and pseudo-labels. Robust TTA methods (Niu et al., 2022; Zhou et al.,
work page 2022
-
[14]
and action recognition (Yang et al., 2024), often involving multiple modalities. A.2 O PEN -SET TEST-TIME ADAPTATION Open-set test-time adaptation (OSTTA) addresses situations where the target domain includes classes absent in the source domain, presenting greater challenges due to the risk of incorrect adaptation to unknown class samples, which can cause...
work page 2024
-
[15]
proposes a self-supervised alignment method combined with adversarial alignment for multimodal DA. Similarly, Kim et al. (2021) employ cross-modal contrastive learning to align representations across both modalities and domains. Besides, Zhang et al. (2022) introduce an audio- adaptive encoder and an audio-infused recognizer to mitigate domain shifts. RNA...
work page 2021
Show all 29 references
-
[16]
presents a universal framework for multimodal domain generalization by separating features within each modal- ity into modality-specific and modality-shared components, while applying constraints to encourage meaningful representation learning. Building on SimMMDG, MOOSA (Dong...
2022
-
[17]
Additionally, Du et al
uses external OOD samples from other datasets during training to improve discrimination between ID and OOD samples. Additionally, Du et al. (2022) propose synthesizing virtual outliers for training-time regularization. While most existing OOD methods are designed for unimodal ...
2022
-
[18]
During open-set TTA, we construct mini-batches with equal numbers of known and unknown samples
and SimMMDG (Dong et al., 2023), to evaluate TTA performance on different models. During open-set TTA, we construct mini-batches with equal numbers of known and unknown samples. We use a batch size of 64 and the Adam optimizer with a learning rate of 2e-5 for all experiments. ...
2023
-
[19]
for the LiDAR stream. We adopt the fusion framework proposed in PMF (Zhuang et al., 2021), modifying it by adding an additional segmentation head to the combined features from the camera and LiDAR streams. For optimization, we use SGD with Nesterov (Loshchilov & Hutter,
2021
-
[20]
EPIC-Kitchens is a large-scale egocentric dataset collected from 32 participants in their native kitchen environments
EPIC-Kitchens (Damen et al., 2018). EPIC-Kitchens is a large-scale egocentric dataset collected from 32 participants in their native kitchen environments. The participants recorded all their daily kitchen activities, with annotated start and end times for each action. We use a...
2018
-
[21]
following the ideas in Hendrycks & Dietterich (2019) and Yang et al. (2024). We apply six different types of corruptions (Gaussian, Defocus, Frost, Brightness, Pixelate, and JPEG) on videos and six others (Gaussian, Wind, Traffic, Thunder, Rain, and Crowd) on audios. We random...
2019
-
[23]
Our AEO achieves the best performance on 31 out of 36 combinations and obtains the highest average H-score, demonstrating its robustness under diverse corruption scenarios. C.3 R OBUSTNESS UNDER MIXED CORRUPTIONS FOR OPEN -SET DATA By default, the corruptions applied to the op...
2017
-
[24]
C.5 I NFLUENCE OF LDiv TO THE PERFORMANCES We investigate the impact ofLDiv in Eq
Our AEO demonstrates low sensitivity to the choice of score functions, achieving comparable performance across different metrics. C.5 I NFLUENCE OF LDiv TO THE PERFORMANCES We investigate the impact ofLDiv in Eq. (7) to the performances, a negative entropy loss term widely use...
2023
-
[25]
As shown in Tab
to promote diversity in predictions. As shown in Tab. 16, removingLDiv results in performance on the EPIC-Kitchens dataset remaining comparable to the original, while significantly reducing performance on the HAC and Kinetics-100-C datasets. This demonstrates the critical role...
2025
-
[26]
14 (b), (e), and (h))
minimizes the entropy of all samples, further narrowing the gap between known and unknown samples (Fig. 14 (b), (e), and (h)). In contrast, the entropy values generated by 20 Published as a conference paper at ICLR 2025 (g) D2 → D3 (No TTA) (h) D2 → D3 (Tent) (i) D2 → D3 (Ours...
2025
-
[27]
perform well and exceed the Source baseline, other TTA methods fail to achieve robust perfor- mance, underscoring the complexities of multimodal open-set adaptation. In contrast, our method demonstrates strong robustness across all modality combinations, significantly improvin...
2020
-
[28]
(2024) and Gao et al
C.13.3 C OMPARISON WITH OPEN -SET METHODS Safaei et al. (2024) and Gao et al. (2024) also utilized the entropy difference between known and unknown samples in open-set settings. However, our work goes beyond merely leveraging this difference. In Sec. 3.1, we provide a deeper a...
2024
-
[2016]
Benchmarking neural network robustness to common corruptions and perturbations
11 Published as a conference paper at ICLR 2025 Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261,
2025 arXiv
-
[2017]
A short note about kinetics-600
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340,
-
[2018]
Extremely simple activation shaping for out-of-distribution detection
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. arXiv preprint arXiv:2209.09858,
-
[2019]
Sharpness-aware minimization for efficiently improving generalization
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412,
2010 arXiv
-
[2020]
Recall and refine: A simple but effective source-free open-set domain adaptation framework
Ismail Nejjar, Hao Dong, and Olga Fink. Recall and refine: A simple but effective source-free open-set domain adaptation framework. arXiv preprint arXiv:2411.12558,
-
[2021]
13 Published as a conference paper at ICLR 2025 A R ELATED WORK A.1 T EST-TIME ADAPTATION Test-time Adaptation (TTA) seeks to adapt a pre-trained model on the source domain online, address- ing distribution shifts without requiring access to either source data or target labels...
2025
-
[2022]
Superfusion: Multilevel lidar-camera fusion for long-range hd map generation
Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xieyuanli Chen. Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211.15656,
-
[2023]
Unified entropy optimization for open-set test-time adaptation
Zhengqing Gao, Xu-Yao Zhang, and Cheng-Lin Liu. Unified entropy optimization for open-set test-time adaptation. arXiv preprint arXiv:2404.06065,
-
[2024]
Stamp: Outlier-aware test-time adaptation with stable memory replay
Yongcan Yu, Lijun Sheng, Ran He, and Jian Liang. Stamp: Outlier-aware test-time adaptation with stable memory replay. arXiv preprint arXiv:2407.15773,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.