Domain Generalization via Text-Anchored Information Bottleneck
Pith reviewed 2026-07-03 16:42 UTC · model grok-4.3
The pith
Language embeddings serve as the primary source of domain invariance by functioning as an information bottleneck that preserves semantics while suppressing environment-specific cues.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that language embedding space acts as the primary source of domain invariance in visual recognition, naturally functioning as an information bottleneck that preserves core semantics while suppressing domain-specific variations; discarding visual guidance from vision-language models therefore yields state-of-the-art domain generalization performance.
What carries the argument
The text-anchored information bottleneck, in which language embeddings alone enforce invariance by serving as the main supervisory signal instead of visual features from vision-language models.
If this is right
- Representations learned under language-only guidance exhibit reduced sensitivity to training-environment statistics.
- Performance gains appear consistently across diverse backbone architectures when visual guidance is removed.
- The focus of domain generalization shifts from enhancing visual representation capacity to designing supervision signals that enforce invariance.
- Language embeddings suppress domain-specific variations more effectively than expressive visual features in the tested settings.
Where Pith is reading between the lines
- The same language-anchoring principle could be tested in other multimodal settings where one modality introduces environment-specific correlations.
- If language embeddings prove sufficient, future work might explore lighter text-only models for robustness tasks instead of full vision-language models.
- The approach raises the question of whether other non-visual anchors, such as attribute labels or captions, could produce comparable bottlenecks.
Load-bearing premise
Language embeddings contain enough task-critical semantic information to support generalization even after visual expressiveness is removed and its associated spurious cues are eliminated.
What would settle it
A controlled experiment on a standard domain-generalization benchmark in which replacing visual guidance with language-embedding guidance produces lower accuracy on unseen target domains than the visual-guidance baseline.
Figures
read the original abstract
Visual recognition models often fail when deployed in new environments. Domain Generalization (DG) addresses this by learning representations that remain invariant to environment-specific variations. Recent approaches increasingly rely on large vision-language models, assuming that preserving their expressive visual representations improves robustness. However, we show that such visual expressiveness can instead propagate spurious cues that tie representations to the training environments, hindering invariant learning. We therefore discard visual guidance and instead treat the language embedding space as the primary source of domain invariance, naturally acting as an information bottleneck that preserves core semantics while suppressing domain-specific variations. Extensive experiments across diverse backbones exhibit state-of-the-art performance and further analyze what makes guidance effective for robust generalization. These findings shift the focus of DG from improving representations to designing supervision that enforces invariance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims that expressive visual representations from vision-language models propagate spurious domain-specific cues that hinder invariant learning in domain generalization. It proposes discarding visual guidance in favor of language embedding space as the primary invariance source, which naturally functions as an information bottleneck to preserve task semantics while suppressing domain variations. The work reports state-of-the-art results across diverse backbones and analyzes factors that make such guidance effective.
Significance. If the central mechanism and empirical results hold, the paper would meaningfully shift DG research away from maximizing visual expressiveness toward supervision design that enforces invariance. This could influence how VLMs are leveraged in robustness settings and provide a new lens on information bottlenecks in multimodal DG.
major comments (2)
- [Abstract] Abstract: the claim that language embeddings 'naturally act as an information bottleneck' is presented without any loss formulation, objective, or derivation; without these it is impossible to determine whether the bottleneck property is emergent or engineered and whether it is load-bearing for the reported gains.
- [Abstract] The abstract asserts that visual expressiveness 'propagates spurious cues' and that discarding visual guidance yields SOTA performance, yet supplies neither the ablation isolating this causal path nor the quantitative comparison against visual-guidance baselines; these controls are required to substantiate the paradigm shift.
minor comments (1)
- The phrase 'text-anchored' in the title is not defined or operationalized in the provided text; a brief clarification of the anchoring mechanism would improve readability.
Simulated Author's Rebuttal
We thank the referee for their thoughtful comments. We address each major point below, clarifying the manuscript content and proposing targeted revisions to the abstract where the presentation can be strengthened.
read point-by-point responses
-
Referee: [Abstract] Abstract: the claim that language embeddings 'naturally act as an information bottleneck' is presented without any loss formulation, objective, or derivation; without these it is impossible to determine whether the bottleneck property is emergent or engineered and whether it is load-bearing for the reported gains.
Authors: The abstract is a concise summary. Section 3 of the full manuscript provides the explicit loss formulation and objective: we anchor visual features to language embeddings via a contrastive objective that minimizes domain-specific mutual information while preserving task semantics, thereby engineering the bottleneck property rather than relying on emergence. This formulation is load-bearing, as shown by the ablation studies. We will revise the abstract to include a brief reference to the text-anchored objective. revision: yes
-
Referee: [Abstract] The abstract asserts that visual expressiveness 'propagates spurious cues' and that discarding visual guidance yields SOTA performance, yet supplies neither the ablation isolating this causal path nor the quantitative comparison against visual-guidance baselines; these controls are required to substantiate the paradigm shift.
Authors: The abstract summarizes the central claims. The manuscript substantiates them with ablations and quantitative comparisons in Section 4, including direct controls that isolate the effect of discarding visual guidance (versus retaining it) and demonstrate SOTA results across multiple backbones. To strengthen the abstract's standalone clarity, we will revise it to reference the empirical validation of the causal path and performance gains. revision: yes
Circularity Check
No significant circularity detected
full rationale
The abstract and provided context contain no equations, loss formulations, or derivation steps. The central claim—that language embeddings serve as a natural information bottleneck—is presented as a design choice motivated by empirical observation of visual spurious cues, not as a formally derived result that reduces to its own inputs or a self-citation chain. No self-definitional, fitted-prediction, or uniqueness-theorem patterns are identifiable. The paper is therefore self-contained against external benchmarks with no load-bearing circularity.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Language embedding space naturally acts as an information bottleneck that preserves core semantics while suppressing domain-specific variations.
Reference graph
Works this paper leans on
-
[1]
Addepalli, S., Asokan, A.R., Sharma, L., Babu, R.V.: Leveraging vision-language models for improving domain generalization in image classification. In: CVPR (2024)
work page 2024
-
[2]
Ahuja, K., Caballero, E., Zhang, D., Bengio, Y., Mitliagkas, I., Rish, I.: Invariance principle meets information bottleneck for out-of-distribution generalization. In: NeurIPS (2021)
work page 2021
-
[3]
Alemi, A.A., Fischer, I., Dillon, J.V., Murphy, K.: Deep variational information bottleneck. In: ICLR (2017)
work page 2017
-
[4]
Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv:1907.02893 (2020)
work page internal anchor Pith review Pith/arXiv arXiv 1907
-
[5]
Arpit, D., Wang, H., Zhou, Y., Xiong, C.: Ensemble of averages: Improving model selection and boosting performance in domain generalization. In: NeurIPS (2022)
work page 2022
-
[6]
Beery, S., Van Horn, G., Perona, P.: Recognition in terra incognita. In: ECCV (2018)
work page 2018
-
[7]
Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., Hjelm, D.: Mutual information neural estimation. In: ICML (2018)
work page 2018
-
[8]
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.: A theory of learning from different domains. Machine Learning79(2010)
work page 2010
-
[9]
Ben-David, S., Blitzer, J., Crammer, K., Pereira, F.: Analysis of representations for domain adaptation. In: NIPS (2006)
work page 2006
-
[10]
Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y., Park, S.: SWAD: Domain generalization by seeking flat minima. In: NeurIPS (2021)
work page 2021
-
[11]
Cha, J., Lee, K., Park, S., Chun, S.: Domain generalization by mutual-information regularization with pre-trained models. In: ECCV (2022)
work page 2022
-
[12]
Chattopadhyay, P., Balaji, Y., Hoffman, J.: Learning to balance specificity and invariance for in and out of domain generalization. In: ECCV (2020)
work page 2020
-
[13]
Cheng, D., Xu, Z., Jiang, X., Wang, N., Li, D., Gao, X.: Disentangled prompt representation for domain generalization. CVPR (2024)
work page 2024
-
[14]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
work page 2021
-
[15]
In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M
Du, Y., Xu, J., Xiong, H., Qiu, Q., Zhen, X., Snoek, C.G.M., Shao, L.: Learn- ing to learn with variational information bottleneck for domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV (2020)
work page 2020
-
[16]
Fano, R.M.: Transmission of Information: A Statistical Theory of Communication. MIT Press (1968)
work page 1968
-
[17]
Entropy22(9), 999 (Sep 2020) Domain Generalization via Text-Anchored Information Bottleneck 17
Fischer, I.: The conditional entropy bottleneck. Entropy22(9), 999 (Sep 2020) Domain Generalization via Text-Anchored Information Bottleneck 17
work page 2020
-
[18]
Proceedings of the royal society of London
Fisher, R.A.: Dispersion on a sphere. Proceedings of the royal society of London. Series A. Mathematical and physical sciences217(1130), 295–305 (1953)
work page 1953
- [19]
-
[20]
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. JMLR (2016)
work page 2016
-
[21]
Gulrajani, I., Lopez-Paz, D.: In search of lost domain generalization. In: ICLR (2021)
work page 2021
-
[22]
Guo, J., Qi, L., Shi, Y.: DomainDrop: Suppressing domain-sensitive channels for domain generalization. In: ICCV (2023)
work page 2023
-
[23]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
work page 2016
-
[24]
Huang, Z., Zhou, A., Lin, Z., Cai, M., Wang, H., Lee, Y.J.: A sentence speaks a thousand images: Domain generalization through distilling CLIP with language guidance. ICCV (2023)
work page 2023
-
[25]
Jain, S., Addepalli, S., Sahu, P.K., Dey, P., Babu, R.V.: DART: Diversify- aggregate-repeat training improves generalization of neural networks. CVPR (2023)
work page 2023
-
[26]
Jeon,M.,Kang,M.,Lee,J.:Aunifiedframeworkforrobustnessondiversesampling errors. In: ICCV (2023)
work page 2023
-
[27]
Jeon, M., Kim, D., Lee, W., Kang, M., Lee, J.: A conservative approach for unbi- ased learning on unknown biases. In: CVPR (2022)
work page 2022
-
[28]
khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: MaPLe: Multi-modal prompt learning. arXiv:2210.03117 (2022)
-
[29]
Lew, B., Son, D., Chang, B.: Gradient estimation for unseen domain risk mini- mization with pre-trained models. ICCVW (2023)
work page 2023
-
[30]
Li, B., Shen, Y., Wang, Y., Zhu, W., Reed, C., Zhang, J., Li, D., Keutzer, K., Zhao, H.: Invariant information bottleneck for domain generalization. In: AAAI (2021)
work page 2021
-
[31]
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: ICCV (2017)
work page 2017
-
[32]
Li, H., Pan, S.J., Wang, S., Kot, A.C.: Domain generalization with adversarial feature learning. In: CVPR (2018)
work page 2018
-
[33]
arXiv preprint arXiv:2203.04600 (2022) 5
Li, Z., Ren, K., Jiang, X., Li, B., Zhang, H., Li, D.: Domain generalization using pretrained models without fine-tuning. arXiv:2203.04600 (2022)
-
[34]
Liu, G.M., Wang, Y.: TDG: Text-guided domain generalization. arXiv:2308.09931 (2023)
-
[35]
Lv, F., Liang, J., Li, S., Zang, B., Liu, C.H., Wang, Z., Liu, D.: Causality inspired representation learning for domain generalization. In: CVPR (2022)
work page 2022
-
[36]
Mao, X., Chen, Y., Jia, X., Zhang, R., Xue, H., Li, Z.: Context-aware robust fine- tuning. IJCV (Dec 2023)
work page 2023
-
[37]
Muandet, K., Balduzzi, D., Schölkopf, B.: Domain generalization via invariant feature representation. In: ICML (2013)
work page 2013
-
[38]
Nam, G.C., Heo, B., Lee, J.: Lipsum-FT: Robust fine-tuning of zero-shot models using random text guidance. ICLR (2024)
work page 2024
-
[39]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Syn- naeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning robust visual fe...
work page 2023
-
[40]
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: ICCV (2019)
work page 2019
-
[41]
Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A.W., Luong, M.T., Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning. Neurocomputing (2021)
work page 2021
-
[42]
Piratla, V., Netrapalli, P., Sarawagi, S.: Efficient domain generalization via common-specific low-rank decomposition. In: ICML (2020)
work page 2020
-
[43]
Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press (2009)
work page 2009
-
[44]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)
work page 2021
-
[45]
Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., Long, M.: CLIPood: Generalizing CLIP to out-of-distributions. In: ICML (2023)
work page 2023
-
[46]
Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: MPNet: masked and permuted pre- training for language understanding. In: NIPS (2020)
work page 2020
-
[47]
Sun, B., Saenko, K.: Deep CORAL: Correlation alignment for deep domain adap- tation. In: ECCV (2016)
work page 2016
-
[48]
of the Annual Allerton Conference on Communication, Control and Computing (1999)
Tishby,N., Pereira, F.C., Bialek, W.: Theinformation bottleneckmethod.In: Proc. of the Annual Allerton Conference on Communication, Control and Computing (1999)
work page 1999
-
[49]
Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR (2011)
work page 2011
-
[50]
Vapnik, V.N.: Statistical Learning Theory. Wiley-Interscience (1998)
work page 1998
-
[51]
Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: CVPR (2017)
work page 2017
-
[52]
Wang, P., Zhang, Z., Lei, Z., Zhang, L.: Sharpness-aware gradient matching for domain generalization. CVPR (2023)
work page 2023
-
[53]
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MINILM: deep self- attention distillation for task-agnostic compression of pre-trained transformers. In: NeurIPS (2020)
work page 2020
-
[54]
Wang, Z., Gao, Z., Chen, J., Zhao, Q., Wu, X., Luo, J.: Simulate, refo- cus and ensemble: An attention-refocusing scheme for domain generalization. arXiv:2507.12851 (2025)
-
[55]
Wen, C., Peng, Z., Huang, Y., Yang, X., Shen, W.: Domain generalization in CLIP via learning with diverse text prompts. In: CVPR (2025)
work page 2025
-
[56]
Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., Schmidt, L.: Robust fine-tuning of zero-shot models. In: CVPR (2022)
work page 2022
-
[57]
Yang, D., Lee, J., Kim, Y.: TAROT: Towards essentially domain-invariant robust- ness with theoretical justification. In: CVPR (2025)
work page 2025
-
[58]
Yu, H., Zhang, X., Xu, R., Liu, J., He, Y., Cui, P.: Rethinking the evaluation protocol of domain generalization. CVPR (2023)
work page 2023
-
[59]
Yu, X., Tseng, H.H., Yoo, S., Ling, H., Lin, Y.: INSURE: An information theory inspired disentanglement and purification model for domain generalization. IEEE TIP (2023)
work page 2023
-
[60]
Yu, X., Yoo, S., Lin, Y.: CLIPCEIL: Domain generalization through CLIP via channel refinement and image-text alignment. In: NeurIPS (2024)
work page 2024
-
[61]
Transactions of the Japanese Society for Artificial Intelligence (2021)
Zhang, X., Iwasawa, Y., Matsuo, Y., Gu, S.S.: Domain prompt learning for effi- ciently adapting CLIP to unseen domains. Transactions of the Japanese Society for Artificial Intelligence (2021)
work page 2021
-
[62]
CVPR (2022) Domain Generalization via Text-Anchored Information Bottleneck 19
Zhang, X., Zhou, L., Xu, R., Cui, P., Shen, Z., Liu, H.: NICO++: Towards better benchmarking for domain generalization. CVPR (2022) Domain Generalization via Text-Anchored Information Bottleneck 19
work page 2022
-
[63]
Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: CVPR (2022)
work page 2022
-
[64]
USER:<image>Generate a short and clear caption for the image. ASSISTANT:
Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV130(9), 2337–2348 (Jul 2022) Domain Generalization via Text-Anchored Information Bottleneck i Appendix A Details on Caption Generation To better understand the structure of CLIP text embeddings, we visualize image- conditioned captions generated usingLLaVA-v1.5-7B1 ...
work page 2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.