pith. machine review for the scientific record. sign in

arxiv: 2604.24602 · v2 · submitted 2026-04-27 · 💻 cs.CV

Recognition: unknown

Majorization-Guided Test-Time Adaptation for Vision-Language Models under Modality-Specific Shift

Lixian Chen , Yanhui Chen , Junyi Lin
Authors on Pith no claims yet

Pith reviewed 2026-05-08 04:17 UTC · model grok-4.3

classification 💻 cs.CV
keywords test-time adaptationvision-language modelsmodality shiftmajorizationentropy minimizationmultimodal fusionreliability-aware gatezero-shot transfer
0
0 comments X

The pith

Majorization view shows that entropy-based test-time adaptation for vision-language models must control modality reliability to prevent error increases under asymmetric shifts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models often face asymmetric distribution shifts where the visual and textual branches change independently, allowing an unreliable modality to dominate the fused prediction. Standard entropy minimization on the fused output can then sharpen incorrect rankings rather than correct ones. The paper analyzes this failure through majorization of multimodal posteriors and reformulates the adaptation task as a constrained de-mixing problem. It proposes MG-MTTA, which freezes the backbone and updates only a lightweight gate or adapter by minimizing fused entropy while respecting a reliability-aware prior built from anchor-based modality consistency and cross-modal conflict. Results on ImageNet-based benchmarks demonstrate accuracy gains specifically under textual and joint shifts, establishing that multimodal test-time adaptation succeeds when it manages modality trust explicitly.

Core claim

The central claim is that entropy minimization on the fused posterior increases error under modality-specific shifts because an unreliable modality can dominate fusion. Through a majorization perspective, adaptation is cast as a constrained de-mixing problem on the fused prediction. MG-MTTA solves this by updating only a lightweight gate with an objective that combines fused-posterior entropy minimization and a reliability-aware gate prior derived from anchor-based modality consistency and cross-modal conflict. The analysis supplies conditions under which entropy reduction preserves correct ranking and a threshold that marks modality-dominance failure. On ImageNet-based benchmarks this lifts

What carries the argument

The reliability-aware gate prior, constructed from anchor-based modality consistency and cross-modal conflict, which augments the fused-posterior entropy minimization objective to prevent an unreliable modality from dominating the adapted prediction.

If this is right

  • Entropy reduction preserves the correct ranking only under the conditions identified by the majorization analysis.
  • A threshold characterizes when modality-dominance failure occurs.
  • Top-1 accuracy rises from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift.
  • The method stays competitive on visual-only shift benchmarks by updating only a lightweight gate or adapter while the backbone remains frozen.
  • Multimodal test-time adaptation must control modality reliability rather than minimize prediction entropy alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reliability-prior construction could be tested on other multimodal pairs such as audio-visual models facing independent modality shifts.
  • The majorization de-mixing formulation may extend to non-entropy objectives in other fusion architectures without changing the backbone.
  • Deploying such lightweight gates could support continual adaptation in streaming applications where labeled validation data are unavailable.
  • If the consistency anchors prove stable across datasets, the approach might reduce reliance on full supervised fine-tuning for handling distribution shifts.

Load-bearing premise

The reliability-aware gate prior constructed from anchor-based modality consistency and cross-modal conflict accurately reflects the true underlying reliability of each modality without requiring additional labeled data or supervision.

What would settle it

Run the method on a benchmark where the constructed gate prior is deliberately inverted to favor the less reliable modality and check whether accuracy gains disappear or reverse compared to plain entropy minimization.

Figures

Figures reproduced from arXiv: 2604.24602 by Junyi Lin, Lixian Chen, Yanhui Chen.

Figure 1
Figure 1. Figure 1: Motivation. Under modality-specific shift, biased fusion can produce a sharper but less reliable posterior. Entropy-only view at source ↗
Figure 2
Figure 2. Figure 2: Overview of MG-MTTA. A frozen vision-language backbone produces modality-level predictions from shifted visual view at source ↗
Figure 4
Figure 4. Figure 4: Reliability and conflict diagnostics under token view at source ↗
Figure 5
Figure 5. Figure 5: Case-wise analysis under the L5 strongest probe. The examples show recovery under severe textual stress, a lower view at source ↗
read the original abstract

Vision-language models transfer well in zero-shot settings, but at deployment the visual and textual branches often shift asymmetrically. Under this condition, entropy-based test-time adaptation can sharpen the fused posterior while increasing error, because an unreliable modality may still dominate fusion. We study this failure mode through a majorization view of multimodal posteriors and cast adaptation as a constrained de-mixing problem on the fused prediction. Based on this view, we propose MG-MTTA, which keeps the backbone frozen and updates only a lightweight gate or adapter. The objective combines fused-posterior entropy minimization with a reliability-aware gate prior built from anchor-based modality consistency and cross-modal conflict. Our analysis gives conditions under which entropy reduction preserves the correct ranking and a threshold that characterizes modality-dominance failure. On the ImageNet-based benchmark, MG-MTTA improves top-1 accuracy from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift, while remaining competitive in the visual-only benchmark. These results show that multimodal test-time adaptation should control modality reliability, not just prediction entropy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes MG-MTTA for vision-language models under asymmetric modality shifts. It views multimodal posteriors through majorization, casts adaptation as constrained de-mixing of the fused prediction, and updates only a lightweight gate or adapter. The objective minimizes fused-posterior entropy while incorporating a reliability-aware gate prior derived from anchor-based modality consistency and cross-modal conflict. Analysis supplies conditions under which entropy reduction preserves correct ranking and a threshold for modality-dominance failure. On ImageNet-based benchmarks, the method raises top-1 accuracy from 57.97 to 66.51 under semantics-preserving textual shift and from 21.68 to 26.27 under joint visual-textual shift while remaining competitive on visual-only shifts.

Significance. If the central claims hold, the work supplies a principled mechanism for handling modality-specific shifts at test time without backbone updates and demonstrates that controlling per-modality reliability can outperform pure entropy minimization. The majorization framing and explicit threshold characterization are positive contributions that could guide future multimodal adaptation research.

major comments (2)
  1. Abstract: the reported gains (57.97 to 66.51 and 21.68 to 26.27) are presented without error bars, standard deviations, or ablation isolating the reliability-aware gate prior from plain entropy minimization; without these, it is impossible to confirm that improvements stem from controlled de-mixing rather than incidental regularization.
  2. Abstract (analysis paragraph): the stated conditions under which entropy reduction preserves correct ranking and the threshold characterizing modality-dominance failure do not address the self-referential construction of the anchor-based modality consistency prior; because anchors are built from the initial shifted model outputs without labels, asymmetric shifts can bias the prior toward the unreliable modality and undermine the de-mixing guarantee.
minor comments (2)
  1. Abstract: the terms 'semantics-preserving textual shift' and 'joint visual-textual shift' are used without a concise definition or reference to the exact benchmark construction protocol.
  2. Abstract: no mention is made of the number of runs, random seeds, or statistical testing used to obtain the quoted accuracy figures.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for highlighting both the potential of the majorization framing and the need for clearer statistical and analytical support in the abstract. We respond to each major comment below.

read point-by-point responses

  1. Referee: [—] Abstract: the reported gains (57.97 to 66.51 and 21.68 to 26.27) are presented without error bars, standard deviations, or ablation isolating the reliability-aware gate prior from plain entropy minimization; without these, it is impossible to confirm that improvements stem from controlled de-mixing rather than incidental regularization.

    Authors: We agree that the abstract would be strengthened by including statistical details and an explicit reference to the isolating ablation. The full manuscript already reports mean performance over five random seeds together with standard deviations in the main results table and provides a dedicated ablation in Section 5.2 that directly compares MG-MTTA to a pure entropy-minimization baseline. We will revise the abstract to report the gains with standard deviations and to note that the ablation isolates the contribution of the reliability-aware gate prior beyond entropy minimization. revision: yes

  2. Referee: [—] Abstract (analysis paragraph): the stated conditions under which entropy reduction preserves correct ranking and the threshold characterizing modality-dominance failure do not address the self-referential construction of the anchor-based modality consistency prior; because anchors are built from the initial shifted model outputs without labels, asymmetric shifts can bias the prior toward the unreliable modality and undermine the de-mixing guarantee.

    Authors: The analysis in Section 3 derives the ranking-preservation conditions and the modality-dominance threshold from the majorization properties of the fused posterior, treating the reliability prior as given. The anchor construction is indeed self-referential because it uses the initial model outputs. However, the prior also incorporates an explicit cross-modal conflict term whose purpose is to detect and attenuate dominance by the less reliable modality. We will add a short discussion paragraph to the analysis section that acknowledges the self-referential nature of the anchors, explains how the conflict term mitigates bias, and notes that the derived threshold can serve as a diagnostic for cases where the prior may be compromised. revision: partial

Circularity Check

0 steps flagged

No circularity: derivation self-contained via independent majorization analysis and empirical benchmarks

full rationale

The abstract and described approach frame adaptation as constrained de-mixing of fused posteriors using a majorization view, with an objective that adds entropy minimization to a reliability-aware gate prior derived from anchor consistency and cross-modal conflict. No equations, self-citations, or fitted parameters are quoted that reduce the claimed predictions (e.g., accuracy gains or ranking-preservation conditions) to inputs by construction. The threshold characterization of modality-dominance failure and conditions for entropy reduction appear as independent analysis rather than tautological redefinition. Reported results are benchmark comparisons, not statistically forced outputs of the prior itself. The method is therefore self-contained against external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the reliability-aware gate prior and majorization conditions are described at high level but their grounding cannot be audited without the full text.

pith-pipeline@v0.9.0 · 5507 in / 1121 out tokens · 70628 ms · 2026-05-08T04:17:31.919960+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages · 3 internal anchors

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    work page internal anchor Pith review doi:10.48550/arxiv.2204.14198 2022

  2. [2]

    Marsden, Tobias Raichle, and Bin Yang

    Mario Döbler, Robert A. Marsden, Tobias Raichle, and Bin Yang. 2024. A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test- Time Adaptation for Vision-Language Models. doi:10.48550/ARXIV.2405.14977

    work page doi:10.48550/arxiv.2405.14977 2024

  3. [3]

    Jindong Gu, Ahmad Beirami, Xuezhi Wang, Alex Beutel, Philip Torr, and Yao Qin. 2023. Towards Robust Prompts on Vision-Language Models. doi:10.48550/ ARXIV.2304.08479

    work page arXiv 2023

  4. [5]

    Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. 2024. Efficient Test-Time Adaptation of Vision-Language Models. doi:10. 48550/ARXIV.2403.18293

    work page arXiv 2024

  5. [6]

    Songtao Li and Hao Tang. 2024. Multimodal Alignment and Fusion: A Survey. doi:10.48550/ARXIV.2411.17040

    work page doi:10.48550/arxiv.2411.17040 2024

  6. [7]

    Jian Liang, Ran He, and Tieniu Tan. 2024. A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts.International Journal of Computer Vision 133, 1 (July 2024), 31–64. doi:10.1007/s11263-024-02181-w

    work page doi:10.1007/s11263-024-02181-w 2024

  7. [8]

    J. Lin. 1991. Divergence measures based on the Shannon entropy.IEEE Transac- tions on Information Theory37, 1 (1991), 145–151. doi:10.1109/18.61115

    work page doi:10.1109/18.61115 1991

  8. [9]

    Sarthak Kumar Maharana, Baoming Zhang, Leonid Karlinsky, Rogerio Feris, and Yunhui Guo. 2024. BATCLIP: Bimodal Online Test-Time Adaptation for CLIP. doi:10.48550/ARXIV.2412.02837

    work page doi:10.48550/arxiv.2412.02837 2024

  9. [10]

    Marshall, Ingram Olkin, and Barry C

    Albert W. Marshall, Ingram Olkin, and Barry C. Arnold. 2011.Inequalities: Theory of Majorization and Its Applications. Springer New York. doi:10.1007/978-0-387- 68276-1

    work page doi:10.1007/978-0-387- 2011

  10. [11]

    Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. 2022. Efficient Test-Time Model Adaptation without Forgetting. doi:10.48550/ARXIV.2204.02610

    work page doi:10.48550/arxiv.2204.02610 2022

  11. [12]

    Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. 2023. Towards Stable Test-Time Adaptation in Dynamic Wild World. doi:10.48550/ARXIV.2302.12400

    work page doi:10.48550/arxiv.2302.12400 2023

  12. [13]

    Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyungwoo Song. 2024. Towards Calibrated Robust Fine-Tuning of Vision-Language Models. InNeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models. https://openreview.net/forum?id=S9h0eLl71q

    2024

  13. [14]

    Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. 2023. Towards Calibrated Robust Fine-Tuning of Vision-Language Models. doi:10.48550/ARXIV. 2311.01723

    work page internal anchor Pith review doi:10.48550/arxiv 2023

  14. [15]

    George Papandreou, Athanassios Katsamanis, Vassilis Pitsikalis, and Petros Maragos. 2009. Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition.IEEE Transactions on Audio, Speech, and Language Processing17, 3 (2009), 423–435. doi:10.1109/TASL.2008. 2011515

    work page doi:10.1109/tasl.2008 2009

  15. [16]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. doi:10.48550/ARXIV.2103.00020

    work page internal anchor Pith review doi:10.48550/arxiv.2103.00020 2021

  16. [17]

    2022.Entropy, Divergence, and Majorization in Classical and Quantum Thermodynamics

    Takahiro Sagawa. 2022.Entropy, Divergence, and Majorization in Classical and Quantum Thermodynamics. Springer Singapore. doi:10.1007/978-981-16-6644-5

    work page doi:10.1007/978-981-16-6644-5 2022

  17. [19]

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anand- kumar, and Chaowei Xiao. 2022. Test-Time Prompt Tuning for Zero-Shot Gener- alization in Vision-Language Models. doi:10.48550/ARXIV.2209.07511

    work page doi:10.48550/arxiv.2209.07511 2022

  18. [20]

    Sahil Sidheekh, Pranuthi Tenali, Saurabh Mathur, Erik Blasch, and Sriraam Natarajan. 2024. On the Robustness and Reliability of Late Multi-Modal Fusion using Probabilistic Circuits. In2024 27th International Conference on Information Fusion (FUSION). 1–8. doi:10.23919/FUSION59988.2024.10706372

    work page doi:10.23919/fusion59988.2024.10706372 2024

  19. [21]

    Jingchen Sun, Rohan Sharma, Vishnu Suresh Lokhande, and Changyou Chen

  20. [22]

    In: 16 D

    Cross-Modal Feature Alignment and MMD Improve Robustness of Prompt Tuning. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV). 4714–4724. doi:10.1109/WACV61041.2025.00462

    work page doi:10.1109/wacv61041.2025.00462 2025

  21. [23]

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2020. Tent: Fully Test-time Adaptation by Entropy Minimization. doi:10. 48550/ARXIV.2006.10726

    work page arXiv 2020

  22. [24]

    Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. 2022. Continual Test-Time Domain Adaptation. doi:10.48550/ARXIV.2203.13591

    work page doi:10.48550/arxiv.2203.13591 2022

  23. [25]

    Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees G. M. Snoek. 2024. Any-Shift Prompting for Generalization over Distributions. doi:10.48550/ARXIV.2402.10099

    work page doi:10.48550/arxiv.2402.10099 2024

  24. [26]

    Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. 2024. Test- time Adaptation against Multi-modal Reliability Bias. InThe Twelfth Interna- tional Conference on Learning Representations. https://openreview.net/forum? id=TPZRq4FALB

    2024

  25. [27]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. doi:10.48550/ARXIV.2303.15343

    work page doi:10.48550/arxiv.2303.15343 2023

  26. [28]

    Duoyi Zhang, Md Abul Bashar, and Richi Nayak. 2025. A novel multi-modal fusion method based on uncertainty-guided meta-learning.Pattern Recognition 158 (2025), 110993. doi:10.1016/j.patcog.2024.110993

    work page doi:10.1016/j.patcog.2024.110993 2025

  27. [29]

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. 2023. Provable Dynamic Fusion for Low-Quality Multimodal Data. doi:10.48550/ARXIV.2306.02050

    work page doi:10.48550/arxiv.2306.02050 2023

  28. [30]

    Yonggang Zhang and Xinmei Tian. 2025. Consistent prompt learning for vision- language models.Knowledge-Based Systems310 (Feb. 2025), 112974. doi:10.1016/ j.knosys.2025.112974 Lixian Chen, Yanhui Chen, and Junyi Lin A Additional Assumptions and Definitions This appendix collects auxiliary definitions that are useful for proofs and implementation. The major...

    work page arXiv 2025