{"id":"ed8b51bd-5199-4d58-bc23-60a2ece6f145","arxiv_id":"2507.21494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Latte improves federated test-time adaptation of CLIP by merging each client's local memory with prototypes retrieved from similar clients.","lead":"Latte adds a memory-sharing step to test-time adaptation of vision-language models in federated learning: each client keeps its own memory and retrieves class prototypes only from similar clients. The result is better accuracy than running memory-based adaptation alone or sharing one global memory, at a small communication cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.3's proof replaces Latte's fixed-size prototype memory with an oracle 1-NN over n_ID·N raw samples; without deriving how prototypes represent those samples, the claimed n_ID dependence is unproven.","rationale":"The reader's weakest_assumption accurately identifies the gap: the multi-client proof does not connect the fixed-size external memory to the pooled n_ID·N sample count. In the single-client analysis, Lemma C.14 relies on the memory entries being i.i.d. uniform draws from the class hyperspheres; the merged memory in Latte is not such a sample. The external prototypes are normalized averages of entropy-selected local embeddings, and the memory size is bounded by kl and ke. Therefore Theorem 4.3's denominator is not derived. This is the most load-bearing concern because it directly affects the headline theoretical claim. I do not see a similarly severe flaw in the experiments: the empirical protocol is reasonable, results are reported over five seeds with standard deviations, and hyperparameter selection is described. The lack of comparison to TSA is a scope limitation, not a correctness issue. The alpha,beta to infinity limit is a gap between theory and practice but secondary to the unproven sample-count step. The recommended verdict remains CONDITIONAL: the empirical contribution is credible, but the paper should either prove the prototype-to-sample connection (or an alternative rate) or soften the theoretical claim.","tokens_in":33600,"tokens_out":9929,"duration_ms":129774,"concrete_test":"Re-derive Theorem C.18 for the minimal case k_l=k_e=1: the class-y merged memory consists of one local embedding and one normalized entropy-weighted ID prototype (Eq. 1). Test whether Lemma C.14's concentration argument can be applied with the candidate set replaced by this prototype, yielding the same theta as Eq. (22) with N replaced by n_ID·N. If the Chernoff bound requires k independent uniform samples and does not carry over to a deterministic function of kl selected embeddings, then the proof's 'apply Theorem C.15 with n_ID·N' step is invalid. Run this derivation; if it cannot be completed, the theorem's n_ID dependence is unproven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.3 (Appendix C.5) is the headline theoretical justification for collaborative TTA. The proof has two steps: (i) under Assumption C.16, OOD embeddings are never the nearest neighbor, so only ID content matters; (ii) 'Latte selects no more than k samples from in total n_ID·N samples', and Theorem C.15 is applied with N replaced by n_ID·N. Step (ii) is not derived for the actual algorithm. Latte's merged memory (Eq. 3) holds kl local embeddings plus at most ke external prototypes per class; the external prototypes are normalized entropy-weighted averages of kl local embeddings (Eq. 1), not i.i.d. draws from the class hypersphere. Lemma C.14's concentration bound requires the k memory points to be among N i.i.d. uniform samples from the class spheres. Neither the fixed memory size nor the averaging operation is compatible with that requirement. Consequently, the n_ID·N denominator in Eqs. (26)-(27) is the rate for an oracle 1-NN classifier with direct access to all pooled raw samples, not a proven property of Latte's prototype-based memory. The 'unaffected by n_OOD' conclusion inherits this gap. This is load-bearing because Section 4 explicitly claims the error decreases with n_ID, and the abstract and conclusion restate this as a core contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Latte, a federated test-time adaptation framework for vision-language models. Each client maintains a local memory of its own high-confidence test embeddings and an external memory of class prototypes retrieved from the most similar clients, coordinated by the server. Adaptation combines CLIP zero-shot logits with memory logits computed from embedding similarity and uncertainty. The authors provide a theoretical analysis claiming that the expected error decreases with the number of in-distribution clients n_ID and is unaffected by out-of-distribution clients n_OOD, and they support this with extensive experiments on domain adaptation and corruption benchmarks using two CLIP backbones, reporting accuracy, standard deviations, ablations, communication/computation costs, and hyperparameter sensitivity.","tokens_in":33864,"tokens_out":4822,"duration_ms":62547,"significance":"If the theoretical claims were fully established, Latte would be a valuable contribution to federated test-time adaptation: it is training-free, communication-efficient, and provides a principled balance between cross-client knowledge sharing and personalization. The empirical work is a genuine strength: five seeds with standard deviations are reported, the authors include ablations for each design component, sensitivity analyses over the main hyperparameters, two backbones, and a public code repository. The paper also gives a rare theoretical treatment of memory-based TTA, explicitly comparing the adapted classifier with the zero-shot baseline. However, the multi-client theorem, which is the headline theoretical contribution and is restated in the abstract and conclusion, has a load-bearing gap in its proof; this tempers the significance until the proof is repaired.","major_comments":[{"comment":"The OOD-robustness conclusion in Theorem 4.3 also depends on the retrieval step in Eq. (2), which the proof does not model. The proof assumes that OOD samples are never selected by the 1-NN classifier, but it does not show that the top-k_e retrieval rule, which compares the target client's own prototype with all other clients' prototypes, will select enough in-distribution prototypes when n_ID is small or when k_e is small, nor does it show that OOD prototypes cannot enter the merged memory and be selected during adaptation. A condition relating k_e to n_ID, or an explicit analysis of the retrieval threshold, is needed to justify 'robust to OOD clients' as stated in Section 4.","section":"Appendix C.5, Theorem 4.3 (Eqs. 26-27)"}],"minor_comments":[{"comment":"Theorem 4.2 and Theorem 4.3 both state 'where k is the merged memory size', but Eq. (3) caps the merged memory at k_l vectors, so the relationship between k, k_l, and k_e should be stated explicitly to avoid ambiguity.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The central empirical contribution is solid and well presented, and the paper has a good chance of being accepted after the theory is repaired. The main issue is not the experimental soundness but the fact that the multi-client theorem, which the paper advertises in the abstract and conclusion, currently proves a bound for an oracle memory rather than for Latte's actual prototype-based memory. I recommend major revision rather than rejection because the gap is localizable and may be fixable with additional assumptions or a more careful proof; if the authors can establish the n_ID dependence for the actual algorithm, the paper would be a strong fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a credible, well-executed empirical core—collaborative memory-based TTA across federated clients, with a practical server-coordinated prototype retrieval scheme—but the headline theoretical claim (error decreases with n_ID, unaffected by n_OOD) is not actually proven for the algorithm. The proof in Appendix C.5 analyzes an oracle 1-NN on pooled raw samples, not Latte's merged memory of entropy-weighted prototypes. That doesn't sink the paper, but it does mean the theory currently overclaims.\n\nWhat's new: the per-client external memory, where the server keeps one prototype per client and each client retrieves the top-ke similar ones, is a legitimate extension of DMN-ZS to decentralized settings. The decoupled communication (offline adaptation, adjustable communication period) is a practical point that matters in FL. The experiments are thorough: four benchmarks, two backbones, five seeds with standard deviations in the appendix, ablations, hyperparameter sensitivity, communication cost, and a comparison of local vs global memory baselines. The finding that naive global memory hurts on heterogeneous domains, while Latte's selective retrieval helps, is useful evidence for the main design.\n\nWhere it's soft: the theory is the weak spot. Theorem 4.3's proof (Appendix C.5) assumes OOD prototypes are never the nearest neighbor, then applies the single-client bound with N replaced by n_ID·N. But Latte's merged memory holds a fixed kl local embeddings plus ke prototypes, where each prototype is an entropy-weighted average of kl embeddings. The concentration argument in Lemma C.14 requires the memory points to be among N i.i.d. uniform samples from the class spheres; neither the fixed memory size nor the averaging respects that. So the n_ID·N denominator is the rate for an oracle classifier with direct access to all pooled samples, not for Latte. The 'unaffected by n_OOD' conclusion inherits the gap. This is load-bearing in the sense that the abstract and Section 4 advertise it as a core theoretical contribution.\n\nAlso of note: no empirical comparison with TSA, the closest federated TTA method, though the paper does cite it and explains the setting difference. That's a minor omission; I'd like to see it addressed in a revision.\n\nWho this is for: anyone working on test-time adaptation for foundation models in decentralized settings. The empirical framework is worth knowing about even if you skip the theory. Does it deserve a serious referee? Yes. The experimental design is careful and reproducible (code link provided), and the theoretical gap, while real, is contained and fixable by either weakening the claim or proving that prototype retrieval actually preserves the sample-count dependency.\n\nMy recommendation: send it out, but flag the theory for the authors to address. If the theory isn't fixable, the contribution still stands on the empirical side, but the advertised guarantee should be retracted.","headline":"Solid empirical contribution on collaborative TTA for VLMs in federated settings; the headline multi-client theory does not go through as written and needs to be fixed or weakened before the paper is cited for that claim.","tokens_in":34461,"tokens_out":2762,"would_cite":true,"duration_ms":30701,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Latte claims that each federated client can adapt CLIP at test time by combining local memory with prototypes from similar clients, and proves the error falls as in-distribution clients grow and is unaffected by out-of-distribution clients.","keywords":["test-time adaptation","vision-language models","federated learning","memory-based adaptation","CLIP","distribution shift","personalization","prototype retrieval"],"falsifier":"Run a controlled federation where one group has a single in-distribution client and another has ten, each client seeing the same $N$ samples and using the same fixed memory sizes; if accuracy stops improving once the external memory is saturated, the claimed $1/(n_{\\mathrm{ID}} N)$ dependence is not realized. Separately, add an out-of-distribution client whose class centers lie at distance 3 from the in-distribution centers, violating the separation assumption, and check whether increasing $n_{\\mathrm{OOD}}$ degrades accuracy.","tokens_in":33361,"feed_emoji":"🔄","tokens_out":10091,"duration_ms":104008,"temperature":0.7,"pith_summary":"Latte is a test-time adaptation method for pre-trained vision-language models in federated settings, where each client sees only a small stream of unlabeled images from its own distribution. The central idea is to give each client a local memory of its own high-confidence embeddings plus an external memory of class prototypes retrieved from the most similar clients, so knowledge is shared without forcing every client to use the same global model. The paper proves that this design makes the expected error decrease as the number of in-distribution clients grows, while out-of-distribution clients do not worsen the bound, and it reports consistent accuracy gains over local-only and global-memory baselines on domain adaptation and corruption benchmarks. The practical payoff is that clients can adapt offline and communicate rarely, with communication cost below a fraction of a percent of the model size.","feed_headline":"CLIP adapts at test time by borrowing prototypes from similar clients","feed_subtitle":"Local memory plus similar-client prototypes beats global sharing on domain and corruption benchmarks.","key_machinery":"The load-bearing mechanism is the two-memory architecture. Each client keeps a local memory $L^i$ of per-class priority queues holding the lowest-entropy embeddings of its own test stream, and the server keeps a global memory $G$ with one entropy-weighted prototype per client and class; each client downloads the top-$k_e$ prototypes most similar to its own prototype into an external memory $E^i$. At inference, local and external memories are merged per class, and the memory logits are formed by aggregating embeddings with weights $\\exp(\\beta f^\\top m)\\exp(-\\gamma H(m))$, so both similarity to the test image and prediction confidence matter. Under $\\alpha, \\beta \\to \\infty$ this aggregation collapses to a 1-nearest-neighbor rule, and the proofs then use sphere-cap volume bounds to control how far the top-$k$ memory entries can be from their asymptotic limits, which is what converts $n_{\\mathrm{ID}} \\cdot N$ pooled samples into a smaller error bound.","core_discovery":"The paper's central claim is Theorem 4.3: on a target client with $n_{\\mathrm{ID}}$ in-distribution clients and $n_{\\mathrm{OOD}}$ out-of-distribution clients (each with $N$ seen samples), Latte's expected error satisfies $\\epsilon_{\\mathrm{post}} \\le \\epsilon_{\\mathrm{asym}} + O\\big( ( (k + \\log(1/\\delta)) / (n_{\\mathrm{ID}} N) )^{1/(d+1)} \\big)$, with a faster $O\\big( ( (k + \\log(1/\\delta)) / (n_{\\mathrm{ID}} N) )^{1/2} \\big)$ rate when the asymptotic error is zero. The bound decreases with the total number of in-distribution samples $n_{\\mathrm{ID}} N$ and does not depend on $n_{\\mathrm{OOD}}$, which the paper reads as proof that Latte benefits from similar clients while staying unaffected by highly different ones. This rests on the observation that when the adapter weights become sharp ($\\alpha, \\beta \\to \\infty$), Latte behaves as a 1-nearest-neighbor classifier, so under a sufficient separation between distributions, OOD prototypes are never selected as nearest neighbors.","pith_inferences":["If the external memory size $k_e$ is small, the benefit of adding more in-distribution clients may saturate in practice even though the theorem's bound keeps improving with $n_{\\mathrm{ID}}$; a controlled experiment varying $n_{\\mathrm{ID}}$ under fixed memory sizes would reveal this limit.","The OOD robustness result assumes class centers are separated by more than distance 4 between distributions; in real deployments with partially overlapping domains, stronger filtering than the current entropy threshold may be needed, which could be tested by adding OOD clients with centers at distance 3.","Because only prototypes are exchanged, the same two-memory retrieval scheme could be combined with privacy mechanisms such as prototype perturbation or secure aggregation, a direction the paper does not explore.","The memory-retrieval idea transfers to non-CLIP encoders or to other memory-based adaptation methods, since nothing in the mechanism depends on the specific text-image alignment of CLIP."],"forward_implications":["In federated deployments with many clients sharing a similar distribution, per-client accuracy should improve as the total number of in-distribution test samples grows, without any raw data leaving the clients.","Because communication is decoupled from local inference, clients can update their external memory only every 50–200 samples with almost no loss in accuracy, making the method practical for bandwidth-limited networks.","A single shared global memory can cause negative transfer when domains differ, whereas retrieving only the most similar prototypes preserves personalization and avoids that degradation.","The adaptation adds only a few hundred thousand MACs against CLIP's 17.6G and transmits less than 0.4% of the visual encoder's size per communication round."],"supporting_citations":[{"why":"It supplies the pretrained CLIP model whose zero-shot classifier and image/text embeddings are adapted at test time.","marker":"[26]"},{"why":"It introduces the training-free cache-and-adapter mechanism that memory-based test-time adaptation, including Latte, builds on.","marker":"[43]"},{"why":"It provides the dual-memory baseline variants compared in the experiments and the hyperparameter selection strategy Latte adopts.","marker":"[46]"},{"why":"It provides the efficient memory-based TTA baseline (TDA) against which Latte is measured in local and global forms.","marker":"[19]"},{"why":"It defines the closest collaborative federated test-time adaptation setting, which Latte contrasts by decoupling communication from inference.","marker":"[41]"},{"why":"It supplies the federated personalization setting and the data partitioning strategy used in the experimental design.","marker":"[2]"},{"why":"It gives prior k-NN-based theory for memory classifiers that the paper's analysis extends beyond noise-free memory assumptions.","marker":"[45]"},{"why":"It provides a prior theoretical analysis of k-NN memory adaptation whose assumptions the paper relaxes.","marker":"[44]"},{"why":"It supplies the gamma-function inequality used to bound sphere-cap volume ratios in the convergence proofs.","marker":"[34]"},{"why":"It provides the CIFAR-10-C and CIFAR-100-C corruption benchmarks used in the experiments.","marker":"[14]"}],"fun_headline_variants":["CLIP borrows prototypes from similar clients to adapt at test time","Federated test-time adaptation: CLIP shares prototypes with similar clients","CLIP adapts via similar-client prototypes in federated test-time","Borrowing prototypes from similar clients: federated CLIP adaptation","Latte: collaborative test-time adaptation with borrowed prototypes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every out-of-distribution client is cleanly separated from the in-distribution clients — all of its class centers are at distance more than 4 from every in-distribution center — and that the fixed-size external memory can be treated as a representative sample of all $n_{\\mathrm{ID}} \\cdot N$ in-distribution embeddings.","fun_headline_variants_meta":{"raw":{"variants":["CLIP borrows prototypes from similar clients to adapt at test time","Federated test-time adaptation: CLIP shares prototypes with similar clients","CLIP adapts via similar-client prototypes in federated test-time","Borrowing prototypes from similar clients: federated CLIP adaptation","Latte: collaborative test-time adaptation with borrowed prototypes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2976,"prompt_tokens":1024,"completion_tokens":1952,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":1863}},"tokens_in":640,"tokens_out":1952,"duration_ms":16705,"temperature":1.0,"reasoning_tokens":1863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:42:53.330524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled federation where one group has a single in-distribution client and another has ten, each client seeing the same $N$ samples and using the same fixed memory sizes; if accuracy stops improving once the external memory is saturated, the claimed $1/(n_{\\mathrm{ID}} N)$ dependence is not realized. Separately, add an out-of-distribution client whose class centers lie at distance 3 from the in-distribution centers, violating the separation assumption, and check whether increasing $n_{\\mathrm{OOD}}$ degrades accuracy.","supporting_citations":[{"cited_title":"Tip- adapter: Training-free adaption of CLIP for few-shot classi- fication","cited_arxiv_id":null,"evidence_quote":"It introduces the training-free cache-and-adapter mechanism that memory-based test-time adaptation, including Latte, builds on."},{"cited_title":"Dual memory networks: A ver- satile adaptation approach for vision-language models","cited_arxiv_id":null,"evidence_quote":"It provides the dual-memory baseline variants compared in the experiments and the hyperparameter selection strategy Latte adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the efficient memory-based TTA baseline (TDA) against which Latte is measured in local and global forms."},{"cited_title":"Enabling collaborative test- time adaptation in dynamic environment via federated learn- ing","cited_arxiv_id":null,"evidence_quote":"It defines the closest collaborative federated test-time adaptation setting, which Latte contrasts by decoupling communication from inference."},{"cited_title":"Adaptive test-time personalization for federated learning","cited_arxiv_id":null,"evidence_quote":"It supplies the federated personalization setting and the data partitioning strategy used in the experimental design."},{"cited_title":"Adanpc: Ex- ploring non-parametric classifier for test-time adaptation","cited_arxiv_id":null,"evidence_quote":"It gives prior k-NN-based theory for memory classifiers that the paper's analysis extends beyond noise-free memory assumptions."},{"cited_title":"Boostadapter: Improving vision- language test-time adaptation via regional bootstrapping","cited_arxiv_id":null,"evidence_quote":"It provides a prior theoretical analysis of k-NN memory adaptation whose assumptions the paper relaxes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the gamma-function inequality used to bound sphere-cap volume ratios in the convergence proofs."},{"cited_title":"Dietterich","cited_arxiv_id":null,"evidence_quote":"It provides the CIFAR-10-C and CIFAR-100-C corruption benchmarks used in the experiments."}],"review_version":1}