Pith. sign in

REVIEW 1 major objections 1 minor 62 references

Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learning

T0 review · 1 major / 1 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Latte claims that each federated client can adapt CLIP at test time by combining local memory with prototypes from similar clients, and proves the error falls as in-distribution clients grow and is unaffected by out-of-distribution clients.

desk verdict Solid empirical contribution on collaborative TTA for VLMs in federated settings; the headline multi-client theory does not go through as written and needs to be fixed or weakened before the paper is cited for that claim. read the letter →

arxiv 2507.21494 v1 pith:NUFNKOI5 submitted 2025-07-29 cs.LG

Wenxuan Bao , Ruxi Deng , Ruizhong Qiu , Tianxin Wei , Hanghang Tong , Jingrui He This is my paper · ORCID
classification cs.LG
keywords test-timeadaptationvision-languagemodelsfederatedlearningmemory-basedCLIPdistributionshiftpersonalizationprototyperetrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Latte is a test-time adaptation method for pre-trained vision-language models in federated settings, where each client sees only a small stream of unlabeled images from its own distribution. The central idea is to give each client a local memory of its own high-confidence embeddings plus an external memory of class prototypes retrieved from the most similar clients, so knowledge is shared without forcing every client to use the same global model. The paper proves that this design makes the expected error decrease as the number of in-distribution clients grows, while out-of-distribution clients do not worsen the bound, and it reports consistent accuracy gains over local-only and global-memory baselines on domain adaptation and corruption benchmarks. The practical payoff is that clients can adapt offline and communicate rarely, with communication cost below a fraction of a percent of the model size.

What carries the argument

The load-bearing mechanism is the two-memory architecture. Each client keeps a local memory $L^i$ of per-class priority queues holding the lowest-entropy embeddings of its own test stream, and the server keeps a global memory $G$ with one entropy-weighted prototype per client and class; each client downloads the top-$k_e$ prototypes most similar to its own prototype into an external memory $E^i$. At inference, local and external memories are merged per class, and the memory logits are formed by aggregating embeddings with weights $\exp(\beta f^\top m)\exp(-\gamma H(m))$, so both similarity to the test image and prediction confidence matter. Under $\alpha, \beta \to \infty$ this aggregation collapses to a 1-nearest-neighbor rule, and the proofs then use sphere-cap volume bounds to control how far the top-$k$ memory entries can be from their asymptotic limits, which is what converts $n_{\mathrm{ID}} \cdot N$ pooled samples into a smaller error bound.

What would settle it

Run a controlled federation where one group has a single in-distribution client and another has ten, each client seeing the same $N$ samples and using the same fixed memory sizes; if accuracy stops improving once the external memory is saturated, the claimed $1/(n_{\mathrm{ID}} N)$ dependence is not realized. Separately, add an out-of-distribution client whose class centers lie at distance 3 from the in-distribution centers, violating the separation assumption, and check whether increasing $n_{\mathrm{OOD}}$ degrades accuracy.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 4.3: on a target client with $n_{\mathrm{ID}}$ in-distribution clients and $n_{\mathrm{OOD}}$ out-of-distribution clients (each with $N$ seen samples), Latte's expected error satisfies $\epsilon_{\mathrm{post}} \le \epsilon_{\mathrm{asym}} + O\big( ( (k + \log(1/\delta)) / (n_{\mathrm{ID}} N) )^{1/(d+1)} \big)$, with a faster $O\big( ( (k + \log(1/\delta)) / (n_{\mathrm{ID}} N) )^{1/2} \big)$ rate when the asymptotic error is zero. The bound decreases with the total number of in-distribution samples $n_{\mathrm{ID}} N$ and does not depend on $n_{\mathrm{OOD}}$, which the paper reads as proof that Latte benefits from similar clients while staying unaffected by highly different ones. This rests on the observation that when the adapter weights become sharp ($\alpha, \beta \to \infty$), Latte behaves as a 1-nearest-neighbor classifier, so under a sufficient separation between distributions, OOD prototypes are never selected as nearest neighbors.

Load-bearing premise

The load-bearing premise is that every out-of-distribution client is cleanly separated from the in-distribution clients — all of its class centers are at distance more than 4 from every in-distribution center — and that the fixed-size external memory can be treated as a representative sample of all $n_{\mathrm{ID}} \cdot N$ in-distribution embeddings.

Editorial extensions

If this is right

  • In federated deployments with many clients sharing a similar distribution, per-client accuracy should improve as the total number of in-distribution test samples grows, without any raw data leaving the clients.
  • Because communication is decoupled from local inference, clients can update their external memory only every 50–200 samples with almost no loss in accuracy, making the method practical for bandwidth-limited networks.
  • A single shared global memory can cause negative transfer when domains differ, whereas retrieving only the most similar prototypes preserves personalization and avoids that degradation.
  • The adaptation adds only a few hundred thousand MACs against CLIP's 17.6G and transmits less than 0.4% of the visual encoder's size per communication round.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the external memory size $k_e$ is small, the benefit of adding more in-distribution clients may saturate in practice even though the theorem's bound keeps improving with $n_{\mathrm{ID}}$; a controlled experiment varying $n_{\mathrm{ID}}$ under fixed memory sizes would reveal this limit.
  • The OOD robustness result assumes class centers are separated by more than distance 4 between distributions; in real deployments with partially overlapping domains, stronger filtering than the current entropy threshold may be needed, which could be tested by adding OOD clients with centers at distance 3.
  • Because only prototypes are exchanged, the same two-memory retrieval scheme could be combined with privacy mechanisms such as prototype perturbation or secure aggregation, a direction the paper does not explore.
  • The memory-retrieval idea transfers to non-CLIP encoders or to other memory-based adaptation methods, since nothing in the mechanism depends on the specific text-image alignment of CLIP.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper proposes Latte, a federated test-time adaptation framework for vision-language models. Each client maintains a local memory of its own high-confidence test embeddings and an external memory of class prototypes retrieved from the most similar clients, coordinated by the server. Adaptation combines CLIP zero-shot logits with memory logits computed from embedding similarity and uncertainty. The authors provide a theoretical analysis claiming that the expected error decreases with the number of in-distribution clients n_ID and is unaffected by out-of-distribution clients n_OOD, and they support this with extensive experiments on domain adaptation and corruption benchmarks using two CLIP backbones, reporting accuracy, standard deviations, ablations, communication/computation costs, and hyperparameter sensitivity.

Significance. If the theoretical claims were fully established, Latte would be a valuable contribution to federated test-time adaptation: it is training-free, communication-efficient, and provides a principled balance between cross-client knowledge sharing and personalization. The empirical work is a genuine strength: five seeds with standard deviations are reported, the authors include ablations for each design component, sensitivity analyses over the main hyperparameters, two backbones, and a public code repository. The paper also gives a rare theoretical treatment of memory-based TTA, explicitly comparing the adapted classifier with the zero-shot baseline. However, the multi-client theorem, which is the headline theoretical contribution and is restated in the abstract and conclusion, has a load-bearing gap in its proof; this tempers the significance until the proof is repaired.

major comments (1)
  1. [Appendix C.5, Theorem 4.3 (Eqs. 26-27)] The OOD-robustness conclusion in Theorem 4.3 also depends on the retrieval step in Eq. (2), which the proof does not model. The proof assumes that OOD samples are never selected by the 1-NN classifier, but it does not show that the top-k_e retrieval rule, which compares the target client's own prototype with all other clients' prototypes, will select enough in-distribution prototypes when n_ID is small or when k_e is small, nor does it show that OOD prototypes cannot enter the merged memory and be selected during adaptation. A condition relating k_e to n_ID, or an explicit analysis of the retrieval threshold, is needed to justify 'robust to OOD clients' as stated in Section 4.
minor comments (1)
  1. [Figure 4 caption] Theorem 4.2 and Theorem 4.3 both state 'where k is the merged memory size', but Eq. (3) caps the merged memory at k_l vectors, so the relationship between k, k_l, and k_e should be stated explicitly to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Latte's central bound is derived from explicit geometric assumptions, not from a fitted parameter or a self-citation chain.

full rationale

The paper's main theoretical claim (Theorem 4.3) is obtained by a real derivation chain: Assumptions C.1-C.5 and C.16 fix the data model; Lemma C.10 shows that the alpha,beta->infinity limit of Eqs. (4)-(7) reduces Latte to 1-NN; Lemmas C.11-C.14 and Theorem C.15 derive single-client error bounds from concentration on sphere caps; Theorem C.18 then applies these bounds to the multi-client setting by using Assumption C.16 to exclude OOD nearest neighbors. No step in this chain is defined in terms of the conclusion it is supposed to prove, and no fitted quantity is renamed as a prediction. The authors do cite their own earlier work (e.g., ATP [2]) for the federated setting, but that citation is not load-bearing: the uniqueness claims and bounds are proved in the appendix rather than imported from prior self-citations. The one substantive weakness is in the last step of Theorem C.18: the proof asserts that 'Latte selects no more than k samples from in total n_ID * N samples' and applies Theorem C.15, while Latte's actual memory (Eq. 3) holds a fixed number of local embeddings plus averaged prototypes, not n_ID*N i.i.d. draws. That is a proof gap about whether the averaged prototypes satisfy the concentration assumptions of Lemma C.14, not a circularity: the bound does not reduce to its own input by construction. The experimental evaluation against CLIP, TDA, DMN-ZS, TPT, and Zero provides independent empirical support. Overall, the derivation is self-contained, so no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on five per-dataset hyperparameters and three domain assumptions (balanced two-sphere data, a non-degenerate initial classifier, and a strong OOD separation). No new physical entities are introduced. The theoretical bound's dependence on n_ID further relies on an unproven step in Appendix C.5, flagged separately.

free parameters (5)
  • α (memory logit weight) = 0.3 (VLCS), 1.5 (TerraIncognita), 1.0 (CIFAR-10-C), 0.7 (CIFAR-100-C)
    Tuned per dataset; controls the balance between zero-shot and memory predictions (Eq. 7).
  • β (similarity sharpness) = 6.0 / 35.0 / 60.0 / 60.0
    Tuned per dataset; controls similarity weighting in Eq. 4.
  • γ (uncertainty sharpness) = 6.0 / 10.0 / 1.5 / 1.5
    Tuned per dataset; controls entropy weighting in Eq. 4 and Eq. 1.
  • k_l (local memory size per class) = 15 / 2 / 12 / 8
    Tuned per dataset; capacity of the local memory queue.
  • k_e (external memory size per class) = 12 / 20 / 9 / 5
    Tuned per dataset; bounds communication cost and the number of retrieved prototypes.
assumptions (4)
  • domain assumption Data are generated from two unit hyperspheres (Assumption C.1).
    Used for all theorems; idealizes real image embeddings and enables volume-based concentration proofs.
  • domain assumption Initial CLIP classifier is linear with error ϵ_pre<1/2 and aligned with truth (Assumptions C.3, C.5).
    Rules out label-swap degeneracy and complete misclassification of a class; reasonable but not guaranteed in practice.
  • domain assumption OOD clients' class centers are at distance >4 from all ID centers (Assumption C.16).
    Load-bearing for Theorem 4.3; ensures ID samples are always closer than OOD samples. This strong separation may not hold in real heterogeneous FL.
  • standard math Standard volume formulas, Wendel's inequality, Chernoff bound (Appendix C.2).
    Used in the proofs of the sphere-cap concentration bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learning." pith.science (2026). https://pith.science/paper/NUFNKOI5

@misc{pith2026250721494,
  author       = {Pith},
  title        = {Pith review of: Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUFNKOI5}},
  note         = {Machine review of arXiv:2507.21494}
}
read the original abstract

Test-time adaptation with pre-trained vision-language models has gained increasing attention for addressing distribution shifts during testing. Among these approaches, memory-based algorithms stand out due to their training-free nature and ability to leverage historical test data. However, existing test-time adaptation methods are typically designed for a single domain with abundant data. In decentralized settings such as federated learning, applying these methods individually to each client suffers from limited test data, while directly sharing a single global memory via the server prevents proper personalization to each client's unique distribution. To address this, we propose Latte, a novel framework where each client maintains a local memory to store embeddings from its own historical test data and an external memory to store class prototypes from other relevant clients. During communication, each client retrieves prototypes from similar clients under the server's coordination to expand its memory. For local adaptation, Latte utilizes both embedding similarity and uncertainty to enhance model performance. Our theoretical analysis shows that Latte effectively leverages in-distribution clients while remaining robust to out-of-distribution clients. Extensive experiments on domain adaptation and corruption benchmarks validate that Latte achieves superior performance in decentralized settings, while introducing only negligible communication and computation costs. Our code is available at https://github.com/baowenxuan/Latte .

Figures

Figures reproduced from arXiv: 2507.21494 by the authors.

Figure 1
Figure 1. Overview of our proposed framework Latte. (Left) Each client performs test-time adaptation using local and external memory. (Right) Clients communicate with the server to update external memory external memory E i . (Subsection 3.3) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Performance of Latte with different communication pe￾riod on TerraIncognita. methods (TPT, Zero) require model inference on multiple augmentations, leading to a significant increase in compu￾tation compared to CLIP. Additionally, TPT introduces ex￾tra overhead due to prompt tuning. In contrast, Latte, like other memory-based methods (e.g., DMN-ZS), achieves high computation efficiency. It only involves lightweight m… view at source ↗
Figure 6
Figure 6. Ablation study on VLCS. 5 10 15 20 25 Local Memory Size kl 65.0 65.5 66.0 66.5 67.0 67.5 68.0 68.5 69.0 Accuracy (%) 5 10 15 20 25 External Memory Size ke 0.2 0.5 1 2 5 10 Latte CLIP [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: t-SNE [32] visualization of local and merged memories. cation for every test sample. In [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 8
Figure 8. Figure 8: Hyperparameter sensitivity analysis on CIFAR-10-C. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Comparison of memory-based TTA methods under [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 58 canonical work pages

  1. [1]

    Dusen- berry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan

    James Urquhart Allingham, Jie Ren, Michael W. Dusen- berry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text- image models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 547–568. PMLR, 2023. 1

    work page 2023

  2. [2]

    Adaptive test-time personalization for federated learning

    Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. Adaptive test-time personalization for federated learning. In Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. 1, 2, 6

    work page 2023

  3. [3]

    Recogni- tion in terra incognita

    Sara Beery, Grant Van Horn, and Pietro Perona. Recogni- tion in terra incognita. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part XVI , pages 472–489. Springer,

    work page 2018

  4. [4]

    Cross- device collaborative test-time adaptation

    Guohao Chen, Shuaicheng Niu, Deyu Chen, Shuhai Zhang, Changsheng Li, Yuanqing Li, and Mingkui Tan. Cross- device collaborative test-time adaptation. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. 2

    work page 2024

  5. [5]

    Marsden, Tobias Raichle, and Bin Yang

    Mario D ¨obler, Robert A. Marsden, Tobias Raichle, and Bin Yang. A lost opportunity for vision-language models: A comparative study of online test-time adaptation for vision- language models. In Computer Vision - ECCV 2024 Work- shops - Milan, Italy, September 29-October 4, 2024, Pro- ceedings, Part XVIII, pages 117–133. Springer, 2024. 6, 7, 25

    work page 2024

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Rep- resentations, ICLR 20...

    2021

  7. [7]

    Rockmore

    Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased met- ric learning: On the utilization of multiple datasets and web images for softening bias. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, Decem- ber 1-8, 2013 , pages 1657–1664. IEEE Computer Society,

    work page 2013

  8. [8]

    Frustratingly easy test-time adaptation of vision-language models

    Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimil- iano Mancini, and Elisa Ricci. Frustratingly easy test-time adaptation of vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Sys- tems, 2024. 2, 6, 7, 25

    work page 2024

Show all 62 references
  1. [9]

    Diverse data augmentation with diffusions for effective test-time prompt tuning

    Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In IEEE/CVF Inter- national Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 2704–2714. IEEE, 2023. 1, 2

    2023

  2. [10]

    In search of lost domain generalization

    Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net, 2021. 24

    2021

  3. [11]

    pfedprompt: Learn- ing personalized prompt for vision-language models in fed- erated learning

    Tao Guo, Song Guo, and Junxiao Wang. pfedprompt: Learn- ing personalized prompt for vision-language models in fed- erated learning. In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages 1364–1374. ACM, 2023. 12

    2023

  4. [12]

    Promptfl: Let federated participants cooper- atively learn prompts instead of models - federated learning in age of foundation model

    Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated participants cooper- atively learn prompts instead of models - federated learning in age of foundation model. IEEE Trans. Mob. Comput., 23 (5):5179–5194, 2024. 12

    2024

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. 6

    2016

  6. [14]

    Dietterich

    Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and per- turbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 6, 24

    2019

  7. [15]

    Aug- mix: A simple data processing method to improve robustness and uncertainty

    Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Aug- mix: A simple data processing method to improve robustness and uncertainty. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, A...

    2020

  8. [16]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In Proceedings of the 38th International Conference on Mach...

    2021

  9. [17]

    Test-time robust personaliza- tion for federated learning

    Liangze Jiang and Tao Lin. Test-time robust personaliza- tion for federated learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. 1, 2

    2023

  10. [18]

    Brendan McMahan, Brendan Avent, Aur´elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A

    Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aur´elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett...

    2021

  11. [19]

    Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El-Saddik, and Eric P. Xing. Efficient test-time adaptation of vision-language models. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , pages 14162–14171. IEEE,

    2024

  12. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  13. [21]

    A comprehensive sur- vey on test-time adaptation under distribution shifts

    Jian Liang, Ran He, and Tieniu Tan. A comprehensive sur- vey on test-time adaptation under distribution shifts. Inter- national Journal of Computer Vision, pages 1–34, 2024. 2

    2024

  14. [22]

    Fedclip: Fast generalization and personalization for CLIP in federated learning

    Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedclip: Fast generalization and personalization for CLIP in federated learning. IEEE Data Eng. Bull., 46(1):52–66, 2023. 12

    2023

  15. [23]

    Communication- efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Ag ¨uera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Arti- ficial Intelligence and Statistics, AISTATS 2017, 20...

    2017

  16. [24]

    Federated learning from vision-language foundation models: Theoretical analy- sis and method

    Bikang Pan, Wei Huang, and Ye Shi. Federated learning from vision-language foundation models: Theoretical analy- sis and method. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 12

    2024

  17. [25]

    Federated text-driven prompt generation for vision- language models

    Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Federated text-driven prompt generation for vision- language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1...

    2024

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  19. [27]

    Jameel Abdul Samadh, Hanan Gani, Noor Hussein, Muham- mad Uzair Khattak, Muzammal Naseer, Fahad Shahbaz Khan, and Salman H. Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot general- ization. In Advances in Neural Information Processing Sy...

    2023

  20. [28]

    Ronny Huang, and Tom Goldstein

    Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neu- ral network overparameterization on gradient confusion and stochastic gradient descent. In Proceedings of the 37th In- ternational Conference on Machine Learning, ICML 2020, ...

    2020

  21. [29]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. In Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural Informa- t...

    2022

  22. [30]

    Just shift it: Test-time prototype shifting for zero-shot general- ization with vision-language models

    Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot general- ization with vision-language models. In IEEE/CVF Win- ter Conference on Applications of Computer Vision, WACV 2025, Tucson, AZ, USA, February 26 - March 6, 2025,...

    2025

  23. [31]

    Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning

    Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Proc...

    2023

  24. [32]

    Visualizing data using t-sne

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 (86):2579–2605, 2008. 8

    2008

  25. [33]

    Ol- shausen, and Trevor Darrell

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net, 2021. 6, 12

    2021

  26. [34]

    J. G. Wendel. Note on the gamma function. The American Mathematical Monthly, 55(9):563–564, 1948. 15

    1948

  27. [35]

    Zehao Xiao and Cees G. M. Snoek. Beyond model adapta- tion at test time: A survey. CoRR, abs/2411.03687, 2024. 2

    2024 arXiv

  28. [36]

    A joint training-calibration framework for test-time personalization with label shift in federated learning

    Jian Xu and Shao-Lun Huang. A joint training-calibration framework for test-time personalization with label shift in federated learning. In Proceedings of the 32nd ACM Interna- tional Conference on Information and Knowledge Manage- ment, CIKM 2023, Birmingham, United Kingdom, ...

    2023

  29. [37]

    Fair federated learning with biased vision- language models

    Huimin Zeng, Zhenrui Yue, Yang Zhang, Lanyu Shang, and Dong Wang. Fair federated learning with biased vision- language models. In Findings of the Association for Com- putational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pages 10002–1001...

    2024

  30. [38]

    Sycara, and Yaqi Xie

    Ce Zhang, Simon Stepputtis, Katia P. Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024. 1, 2, 3

    2024

  31. [39]

    Robust test- time adaptation for zero-shot prompt tuning

    Dingchu Zhang, Zhi Zhou, and Yufeng Li. Robust test- time adaptation for zero-shot prompt tuning. In Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo...

    2024

  32. [40]

    Historical test-time prompt tuning for vision foundation models

    Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Historical test-time prompt tuning for vision foundation models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2

    2024

  33. [41]

    Enabling collaborative test- time adaptation in dynamic environment via federated learn- ing

    Jiayuan Zhang, Xuefeng Liu, Yukang Zhang, Guogang Zhu, Jianwei Niu, and Shaojie Tang. Enabling collaborative test- time adaptation in dynamic environment via federated learn- ing. In Proceedings of the 30th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, KDD 20...

    2024

  34. [42]

    MEMO: test time robustness via adaptation and augmentation

    Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: test time robustness via adaptation and augmentation. In Ad- vances in Neural Information Processing Systems 35: An- nual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November...

    2022

  35. [43]

    Tip- adapter: Training-free adaption of CLIP for few-shot classi- fication

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of CLIP for few-shot classi- fication. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, ...

    2022

  36. [44]

    Boostadapter: Improving vision- language test-time adaptation via regional bootstrapping

    Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. Boostadapter: Improving vision- language test-time adaptation via regional bootstrapping. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2, 5

    2024

  37. [45]

    Adanpc: Ex- ploring non-parametric classifier for test-time adaptation

    Yifan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Adanpc: Ex- ploring non-parametric classifier for test-time adaptation. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , pages 416...

    2023

  38. [46]

    Dual memory networks: A ver- satile adaptation approach for vision-language models

    Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A ver- satile adaptation approach for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pag...

    2024

  39. [47]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16795–16804. IEEE, 2022. 1

    2022

  40. [48]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. Int. J. Comput. Vis., 130(9):2337–2348, 2022. 1 Appendix Contents A . More Discussion 12 A.1 . Additional Related Works . . . . . . . . . . . . . . . . . . . . . . . . ....

    2022

  41. [52]

    Label distribution: The prior label distribution Pr(y = 0) = Pr(y = 1) = 1 2,

  42. [53]

    Remark C.2

    Conditional feature distribution: for each class y, the image embedding is uniformly sampled from a d-dimensional unit hypersphere Sy = {f : ∥f − (2y − 1)µ∥2 ≤ 1}, with center (2y − 1)µ (i.e., ±µ) and radius 1. Remark C.2. For clarity, we consider a simple, balanced binary cla...

  43. [54]

    Let pρ := V ol(G(+µ, h(m∗

    − ρ′) = G(h(m∗ 1 − ρ′wpre)) ⊂ G(0) ∩ B(m∗ 1, ρ). Let pρ := V ol(G(+µ, h(m∗

  44. [55]

    − ρ′)) + V ol(G(−µ, h(m∗

  45. [56]

    Since 0 < pρ < 1, and 1[fi∈R(h(m∗ 1 −ρ′wpre))] ∼ Bern(pρ), then for any κ ∈ [k], as n → ∞, Pr[∥m1,κ − m∗ 1∥ > ρ] = Pr[ m1,κ /∈ G(0) ∩ B(m∗ 1, ρ)] ≤ Pr[m1,κ /∈ G(h(m∗

    − ρ′)) V ol(B(+µ, 1)) + V ol(B(−µ, 1)) . Since 0 < pρ < 1, and 1[fi∈R(h(m∗ 1 −ρ′wpre))] ∼ Bern(pρ), then for any κ ∈ [k], as n → ∞, Pr[∥m1,κ − m∗ 1∥ > ρ] = Pr[ m1,κ /∈ G(0) ∩ B(m∗ 1, ρ)] ≤ Pr[m1,κ /∈ G(h(m∗

  46. [57]

    − ρ′)] ≤ Pr[∃κ ∈ [k] : m1,κ /∈ G(h(m∗

  47. [58]

    − ρ′)] = Pr h arg topk fi: fi∈G(0) h(fi) ̸⊂ G(h(m∗

  48. [59]

    This implies that m1,κ P → m∗ 1 for every κ ∈ [k]

    − ρ′) i = Pr X i∈[n] 1[fi∈G(h(m∗ 1 )−ρ′)] < k = k−1X j=0 n j pj ρ(1 − pρ)n−j = (1 − pρ)n k−1X j=0 n j pj ρ(1 − pρ)−j = (1 − pρ)n k−1X j=0 O(nj) = (1 − pρ)n · O(nk−1) → 0. This implies that m1,κ P → m∗ 1 for every κ ∈ [k]. Similarly, we can show that m0,κ P → m∗ 0 for every κ ∈...

  49. [60]

    So this is proved

    Notice that µ⊤wasym ≥ µ⊤wpre To prove this, µ⊤wpost ≥ µ⊤wpre ⇔ µ⊤µ + µ⊤wpre ≥ ∥µ + wpre∥2 · (µ⊤wpre) ⇔ (µ⊤µ + µ⊤wpre)2 ≥ ∥µ + wpre∥2 2 · (µ⊤wpre)2 ⇔ (µ⊤µ + 2µ⊤wpre)(µ⊤µ − (µ⊤wpre)2) ≥ 0 Notice that µ⊤µ + 2µ⊤wpre > 0, and µ⊤µ = ∥µ∥2 2 ≥ (µ⊤wpre)2 since ∥wpre∥2 = 1. So this is p...

  50. [61]

    When ϵasym > 0, V ol(A2) can be bounded by V ol(A2) Vsphere(d) < Vsphere(d − 1) · ∆ Vsphere(d) = π d−1 2 Γ( d+1 2 ) · Γ( d 2 + 1) π d 2 · ∆ ≤ 1√π · r d + 1 2 · ∆ (Wendel’s inequality and Corollary C.9) < 1√π · r d + 1 2 · 3θ ≤ 1√π · r d + 1 2 · 3 · π 2 8√π · √ 2d + 4 · log(2/δ...

  51. [62]

    itap of a {class}

    When ϵasym = 0, we can have a tighter bound V ol(A2) Vsphere(d) < Vsphere cap(d, arccos(1 − ∆)) Vsphere(d) Notice that arccos(1 − ∆) ≤ π 2 · √ ∆, ∀∆ ∈ [0, 1], therefore, V ol(A2) Vsphere(d) < Vsphere cap(d, arccos(1 − ∆)) Vsphere(d) ≤ Vsphere cap(d, π 2 √ ∆) Vsphere(d) ≤ 1√π ·...

  52. [2021]

    OpenReview.net, 2021. 6

    2021

  53. [2024]

    1, 2, 3, 6, 7, 25, 26

  54. [8763]

    1, 2, 3, 6, 7, 25, 26

    PMLR, 2021. 1, 2, 3, 6, 7, 25, 26

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.