Pith. sign in

REVIEW 4 major objections 6 minor 65 references

Federated Adapter on Foundation Models: An Out-Of-Distribution Approach

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the aggregated global model in federated foundation-model fine-tuning already carries out-of-distribution robustness, and that a feature-distance regularizer that pulls each client's personalized encoder toward the…

desk verdict Plausible empirical recipe for personalized federated adapters, but the advertised OOD generalization proof breaks on an invalid inequality and an unverified head-optimality assumption. read the letter →

arxiv 2505.01075 v1 pith:GY34B5CC submitted 2025-05-02 cs.LG

Yiyuan Yang , Guodong Long , Tianyi Zhou , Qinghua Lu , Shanshan Ye , Jing Jiang This is my paper · ORCID
classification cs.LG
keywords federatedlearningfoundationmodelsout-of-distributiongeneralizationparameter-efficientfine-tuningadapterspersonalizationinvariantfeatureconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how out-of-distribution (OOD) generalization can survive in federated fine-tuning of foundation models, where models are huge, clients see different tasks or domains, and only a small adapter is trained and communicated. It argues that the conventional aggregated global model already has OOD robustness: because the frozen pre-trained head is assumed optimal for every task, the global objective reduces to learning invariant features. It then shows that a personalized model's OOD error is controlled by the expected distance between that client's features and the global model's features. On that ground it proposes FedOA, which keeps a personal adapter per client and regularizes it with $D(\Phi_e(X), \Phi_g(X))$, and proves convergence to a stationary point at rate $O(1/T)$ under non-convex assumptions. The empirical portion reports that this regularizer outperforms parameter-regularization and conventional-FL OOD baselines on leave-one-task-out NLP benchmarks.

What carries the argument

The central object is the feature-distance regularizer $D(\Phi_e(X), \Phi_g(X))$, applied in representation space rather than parameter space. It appears in the optimization objective as $\min_{\Phi_e} R_e(\Phi_e) + \lambda D(\Phi_e(X_e), \Phi_g^*(X_e))$ with the global encoder $\Phi_g^*$ solving the aggregated objective. The regularizer is what converts the Theorem 3.2 generalization bound into a training signal; it also sidesteps the structural heterogeneity of parameter-efficient modules such as low-rank adapters, because it shapes adapter parameters indirectly through their output features. In the convergence analysis it is instantiated as the squared L2 distance, and the proof tracks how this term bounds client drift between the personalized and global adapters.

What would settle it

Take a foundation model whose pre-training corpus excludes the federated clients' task family, train FedOA with a learnable head instead of a frozen one, and measure leave-one-task-out accuracy: if the feature-distance regularizer no longer improves OOD accuracy or the aggregated global model degrades, the fixed-head optimality premise is violated.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.2: for a personalized hypothesis $(w, \Phi_e)$ in a federated foundation model with fixed head $w$, the excess OOD risk is bounded by the invariant-feature distance $M \cdot E_{X \sim P_e}[D(\Phi_e(X), \Phi_g(X))]$ plus the global model's own bound. Since the global model's bound is tight when the frozen head is optimal across environments (Theorem 3.1), minimizing the feature distance is a principled way to bring personalized models' OOD guarantees close to the global model's. The proposed FedOA implements this by training two adapters per client: a personalized adapter updated with the regularizer and a global adapter that is aggregated by the server; only the global adapter is communicated. The paper also proves that under smoothness, bounded-gradient, and bounded-diversity assumptions, the personalized objectives reach a stationary point at rate $O(1/T)$.

Load-bearing premise

The argument rests on the claim that the frozen pre-trained head is already optimal for every possible environment, so the global model's objective collapses to invariant feature learning; if some environment needs a different head, the global model's OOD guarantee and the regularizer's rationale weaken.

Editorial extensions

If this is right

  • Conventional federated averaging over adapters already provides OOD robustness, so even without personalization the global model can be deployed on unseen clients or tasks.
  • Personalized adapters can be made OOD-robust without modifying the aggregated global model, so the method preserves communication efficiency.
  • The same regularization also reduces feature drift between clients and the global model, which the experiments tie to flatter loss surfaces and more stable generalization.
  • The convergence guarantee holds for non-convex objectives, so the method's extra regularizer does not change the asymptotic communication-computation trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's argument suggests that if the head is no longer frozen, the same feature-distance logic would have to be extended to head parameters; the experiments do not test head tuning, so that extension remains an open prediction rather than a demonstrated result.
  • A sharper distance function than L2, for example a distributional discrepancy between feature batches, might capture the Theorem 3.2 term more directly; the paper only compares L2, cosine, and Pearson, so this is a testable direction.
  • The worst-case guarantee in the paper is about reaching a stationary point for each client's personalized objective, not about minimizing the worst environment risk; in strongly heterogeneous federations the regularizer may still leave large inter-client gaps.
  • One could test the fixed-head premise directly by measuring whether the pre-trained head is near-optimal for held-out environments; a positive measurement would isolate when the OOD gain of the regularizer should appear.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FedOA, a federated adapter method for foundation models that adds a per-client personalized adapter and a feature-distance regularizer pulling personalized features toward the globally aggregated model. It claims three theoretical results: that a conventional aggregated global model in FedFM inherently retains OOD generalization ability, that the personalized model's OOD risk is bounded by the feature distance between personalized and global encoders, and that FedOA converges at rate O(1/T) under non-convex assumptions. Experiments on four NLP tasks derived from Flan under a leave-one-task-out protocol, together with ablations, a scalability study, and a ViT/OfficeHome extension, report that FedOA outperforms the personalized baselines on average.

Significance. If the theoretical claims were correct, FedOA would be a useful contribution to federated foundation-model personalization with OOD guarantees. The empirical evaluation is reasonably broad: four NLP tasks, a leave-one-task-out evaluation, ablations over lambda and the distance function, convergence curves, a 30-client scalability experiment, and an OfficeHome/ViT extension. However, the advertised guarantees rest on an invalid proof step in Theorem 3.2 and on an unvalidated optimal-head premise in Theorem 3.1, so the central theoretical contribution is not established. The paper also provides no code release or machine-checked proofs, which limits independent verification of the empirical claims.

major comments (4)
  1. [Appendix D, Eq. (14)] Step (b) of the proof of Theorem 3.2 asserts |E[g(z)] - E[g(z')]| <= M * E_X[D(Phi_e(X), Phi_g(X))] from the boundedness condition |g| <= M for a generic distance D. With D chosen as L2 distance in Section 4, this is false without a Lipschitz assumption on g. For example, with X ~ U[0,1], Phi_e(X)=X, Phi_g(X)=X+eps/2, and g(z)=M * 1_{floor(2z/eps) even}, the left-hand side equals M while M * E[D] = M*eps/2, so the inequality fails for eps < 2. A Lipschitz condition on the composition loss o head would repair the step, but none is stated. Consequently, Eq. (12) and the claim that the regularizer in Eq. (3) provably improves OOD generalization are unsupported.
  2. [Section 3.1 / Appendix D (Theorem 3.1 proof)] The proof assumes that a single frozen pre-trained head w satisfies w in argmin_w R_e(w, Phi_g) for all e in E_all and justifies this by the scale of pre-training. This is an unvalidated premise; the paper offers no evidence that one fixed head is simultaneously optimal for all downstream environments. In addition, Assumption 2.1 only equates conditional means E[Y^e | Phi(X^e)=z], which is insufficient to make the discrepancy d_F(P_e,P_e') vanish for general losses. For instance, under squared loss, equal conditional means with different conditional variances still produce a nonzero discrepancy. Theorem 3.1's conclusion therefore lacks support.
  3. [Theorem 3.2 / Appendix D, Eq. (12)] Even if the proof step in Eq. (14) were repaired, the bound in Eq. (12) is stated for R_e(f_e) with the distance E_{X~P_e}[D(Phi_e(X), Phi_g(X))] evaluated on the client's own distribution. This is not an out-of-distribution bound for an unseen environment e'; the regularizer in Eq. (3) directly minimizes this same training-distribution distance, so the claimed OOD guarantee is partly a statement about the training objective itself. To support the OOD claim, the paper would need a bound involving E_{X~P_e'}[D(Phi_e(X), Phi_g(X))] for the target environment.
  4. [Section 4, Theorem 4.4 / Appendix E.2] The convergence statement does not match the objective being optimized. The client update minimizes R_e(phi) + lambda * D(Phi(phi), Phi(phi_g)), but Theorem 4.4 bounds (1/T) * sum_t E||nabla R_e(phi_e^{t-1})||^2, which omits the lambda * nabla_phi D term; stationarity of the regularized objective would require the full gradient to vanish. Moreover, the step-size choices eta_l = O(1/(T K L sigma)) and eta_g = O(1/(T L)) do not satisfy the theorem's own conditions eta_l <= 1/(8 sqrt(3(1+3T)) T (1+2K) K lambda sigma L) and eta_g <= 1/(2 sqrt(6(1+3T)) T L), and the rates in Eq. (8) are dimensionally inconsistent with the substitutions that produce them. In addition, Lemma E.4 uses ||Phi(phi_1)-Phi(phi_2)|| <= L ||phi_1-phi_2||, which follows from Assumption 4.1 only if Phi is L-Lipschitz, not merely L-smooth. These issues leave the advertised O(1/T) convergence claim unsubstantiated.
minor comments (6)
  1. [Throughout] The word 'turning' appears where 'tuning' is intended, for example in Section 3.1 ('during the turning') and in the discussion of the fixed head.
  2. [Appendix C.2] The description of FedSDR says 'we adapt pFedMe to the FedFM setting' but the surrounding text refers to FedSDR; this appears to be a copy-paste error.
  3. [Page 8, Section 5.1] The metric is written 'ROGUE-1' but should be 'ROUGE-1'.
  4. [Section 3.1 / Appendix D] The statement of Theorem 3.2 in the main text uses E[D(Phi_e(X_{e'}), Phi_g(X_{e'}))], while the appendix proof and Eq. (12) use E_{X~P_e}; the notation should be aligned.
  5. [Eq. (2)] The variable z is used as the label in the empirical risk R_e(Phi) but is not defined before the equation, making the feature-space loss ambiguous.
  6. [Algorithm 1 and Eq. (3)] The symbol D is used both for the distance function and for a dataset in Table 1; consider renaming one of them to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regularizer is chosen to minimize an upper bound from Theorem 3.2 rather than being the definition of the claimed OOD guarantee, though Theorem 3.2's proof contains a non-circular correctness gap.

full rationale

The paper's central theoretical chain is not circular. Theorem 3.1 imports the global-model generalization bound from Konstantinov and Lampert [22] and adds the pre-training-based assumption that the frozen head w is optimal for all environments; that assumption is strong and empirically unsupported, but it is an explicit premise, not an output smuggled back into the conclusion. Theorem 3.2 attempts to upper-bound the personalized model's excess risk by a term involving E[D(Phi_e(X),Phi_g(X))], and the method then adopts exactly this feature-distance term as a regularizer in Eq. (3). This is a standard bound-driven algorithm design: the bound's left-hand side is expected risk, not the feature distance, so minimizing the distance is a proposed way to tighten an upper bound rather than a tautological restatement of the objective. The empirical evaluation uses held-out tasks (leave-one-task-out), which is independent evidence. The one overlapping-author citation, [55], appears only in related work to describe a dual-adapter approach and is not load-bearing. The notable weakness is in the proof of Theorem 3.2: step (b) in Appendix D asserts |E_{z~Phi_e(X)}[g(z)] - E_{z~Phi_g(X)}[g(z)]| <= M E[D(Phi_e(X),Phi_g(X))] solely from |g|<=M, which is false for the L2 distance used in Section 4. That is a correctness/rigor problem, not a circularity problem, because a repaired proof with a suitable probability metric or Lipschitz assumption would still connect the bound to the regularizer without the regularizer being assumed into the bound. Therefore the circularity score is 0, with the proof gap and unsupported head-optimality assumption flagged as separate technical risks.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the personalized and global adapters are architectural components built from existing LoRA modules. The free parameters are hyperparameters selected by ablation or standard practice, and the axioms are a mix of standard ML assumptions and a strong domain-specific claim about frozen heads.

free parameters (3)
  • λ = 0.5
    Regularization weight in Eq. (3); selected via ablation on the held-out task (Table 3), with performance increasing up to λ=2.
  • D = L2 distance
    Feature distance function chosen by ablation (Table 4) as best among Cosine, Pearson, L2.
  • LoRA rank r = 8
    Rank of low-rank adapters, chosen without justification; standard value.
assumptions (4)
  • domain assumption Assumption 2.1: exists representation Φ with E[Y|Φ(X)=z] equal across environments
    Standard invariant-learning condition invoked in the proofs of Theorems 3.1 and 3.2; not validated on the studied data.
  • ad hoc to paper A single fixed pre-trained head w is optimal for all environments e ∈ E_all
    Lets the global objective reduce to objective (2) in Theorem 3.1; asserted as a property of foundation models without evidence.
  • standard math Smoothness, unbiased/bounded stochastic gradients, bounded gradient diversity (Assumptions 4.1-4.3)
    Standard FL convergence assumptions used for Theorem 4.4.
  • domain assumption Test distributions belong to E_all and the bounded loss condition ℓ ≤ M
    Standard OOD framework; the bound M is also reused to bound initial parameter distance, which is unjustified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Adapter on Foundation Models: An Out-Of-Distribution Approach." pith.science (2026). https://pith.science/paper/GY34B5CC

@misc{pith2026250501075,
  author       = {Pith},
  title        = {Pith review of: Federated Adapter on Foundation Models: An Out-Of-Distribution Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GY34B5CC}},
  note         = {Machine review of arXiv:2505.01075}
}
read the original abstract

As foundation models gain prominence, Federated Foundation Models (FedFM) have emerged as a privacy-preserving approach to collaboratively fine-tune models in federated learning (FL) frameworks using distributed datasets across clients. A key challenge for FedFM, given the versatile nature of foundation models, is addressing out-of-distribution (OOD) generalization, where unseen tasks or clients may exhibit distribution shifts leading to suboptimal performance. Although numerous studies have explored OOD generalization in conventional FL, these methods are inadequate for FedFM due to the challenges posed by large parameter scales and increased data heterogeneity. To address these, we propose FedOA, which employs adapter-based parameter-efficient fine-tuning methods for efficacy and introduces personalized adapters with feature distance-based regularization to align distributions and guarantee OOD generalization for each client. Theoretically, we demonstrate that the conventional aggregated global model in FedFM inherently retains OOD generalization capabilities, and our proposed method enhances the personalized model's OOD generalization through regularization informed by the global model, with proven convergence under general non-convex settings. Empirically, the effectiveness of the proposed method is validated on benchmark datasets across various NLP tasks.

Figures

Figures reproduced from arXiv: 2505.01075 by the authors.

Figure 1
Figure 1. Average accuracy varies as communication rounds on read￾ing comprehension task. 2 4 6 8 10 12 14 16 18 20 Communication Rounds 0.0 0.5 1.0 1.5 2.0 2.5 Feature Distance Generalization on RC FedOA PRADA pFedMe [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 25 canonical work pages

  1. [1]

    Out of distribution generalization in machine learning

    Martin Arjovsky. Out of distribution generalization in machine learning. PhD thesis, New York University, 2020. 10

    work page 2020

  2. [2]

    Invariant risk mini- mization

    Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv:1907.02893, 2019

    arXiv 1907

  3. [3]

    Slora: Federated parameter efficient fine-tuning of language models

    Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06522, 2023

    arXiv 2023

  4. [4]

    Towards federated foundation models: Scalable dataset pipelines for group-structured learning

    Zachary Charles, Nicole Mitchell, Krishna Pillutla, Michael Reneer, and Zachary Garrett. Towards federated foundation models: Scalable dataset pipelines for group-structured learning. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  5. [5]

    On bridging generic and personalized federated learning for image classification

    Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00778, 2021

    arXiv 2021

  6. [6]

    Heterogeneous lora for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

    work page 2023

  7. [7]

    Exploiting shared representations for personalized federated learning

    Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089–2099. PMLR, 2021

    2021

  8. [8]

    Dingo: Distributed newton-type method for gradient-norm optimization

    Rixon Crane and Fred Roosta. Dingo: Distributed newton-type method for gradient-norm optimization. Advances in neural information processing systems, 32, 2019

    work page 2019

Show all 65 references
  1. [9]

    Distributionally robust federated averaging

    Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Distributionally robust federated averaging. Advances in neural information processing systems , 33:15111–15122, 2020

    2020

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [11]

    The risk of federated learning to skew fine-tuning features and underperform out-of-distribution robustness

    Mengyao Du, Miao Zhang, Yuwen Pu, Kai Xu, Shouling Ji, and Quanjun Yin. The risk of federated learning to skew fine-tuning features and underperform out-of-distribution robustness. arXiv preprint arXiv:2401.14027, 2024

    2024 arXiv

  4. [12]

    Fednew: A communication-efficient and privacy-preserving newton-type method for federated learning

    Anis Elgabli, Chaouki Ben Issaid, Amrit Singh Bedi, Ketan Rajawat, Mehdi Bennis, and Vaneet Aggarwal. Fednew: A communication-efficient and privacy-preserving newton-type method for federated learning. In International conference on machine learning, pages 5861–5877. PMLR, 2022

    2022

  5. [13]

    Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557–3568, 2020

    2020

  6. [14]

    Active invariant causal prediction: Experiment selection through stability

    Juan L Gamella and Christina Heinze-Deml. Active invariant causal prediction: Experiment selection through stability. Advances in Neural Information Processing Systems, 33:15464– 15475, 2020

    2020

  7. [15]

    Out-of-distribution generaliza- tion of federated learning via implicit invariant relationships

    Yaming Guo, Kai Guo, Xiaofeng Cao, Tieru Wu, and Yi Chang. Out-of-distribution generaliza- tion of federated learning via implicit invariant relationships. In International Conference on Machine Learning, pages 11905–11933. PMLR, 2023

    2023

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  9. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  10. [18]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

    Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933, 2023

    2023 arXiv

  11. [19]

    Test-time robust personalization for federated learning

    Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In ICLR, 2023

    2023

  12. [20]

    Breaking the centralized barrier for cross-device federated learning

    Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian U Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device federated learning. Advances in Neural Information Processing Systems , 34:28663–28676, 2021. 11

    2021

  13. [21]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020

    2020

  14. [22]

    Robust learning from untrusted sources

    Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In International conference on machine learning, pages 3488–3498. PMLR, 2019

    2019

  15. [23]

    When is invariance useful in an out-of-distribution generalization problem? arXiv preprint arXiv:2008.01883, 2020

    Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in an out-of-distribution generalization problem? arXiv preprint arXiv:2008.01883, 2020

    2008 arXiv

  16. [24]

    Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning

    Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309.00363, 2023

    2023 arXiv

  17. [25]

    Ditto: Fair and robust federated learning through personalization

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357–6368. PMLR, 2021

    2021

  18. [26]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020

    2020

  19. [27]

    Fedbn: Federated learning on non-iid features via local batch normalization

    Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021

    2021 arXiv

  20. [28]

    Federated domain generalization: A survey

    Ying Li, Xingwei Wang, Rongfei Zeng, Praveen Kumar Donta, Ilir Murturi, Min Huang, and Schahram Dustdar. Federated domain generalization: A survey. arXiv preprint arXiv:2306.01334, 2023

    2023 arXiv

  21. [29]

    No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier

    Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5319–5329, 2023

    2023

  22. [30]

    Heterogeneous risk minimization

    Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In International Conference on Machine Learning, pages 6804–6814. PMLR, 2021

    2021

  23. [31]

    Towards out-of-distribution generalization: A survey

    Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021

    2021 arXiv

  24. [32]

    Summary of chatgpt-related research and perspective towards the future of large language models

    Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology, page 100017, 2023

    2023

  25. [33]

    Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring

    Zhengquan Luo, Yunlong Wang, Zilei Wang, Zhenan Sun, and Tieniu Tan. Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring. arXiv preprint arXiv:2206.06818, 2022

    2022 arXiv

  26. [34]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  27. [35]

    Stochastic gradient methods for distributionally robust optimization with f-divergences

    Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. Advances in neural information processing systems, 29, 2016

    2016

  28. [36]

    Fedsr: A simple and effective domain generalization method for federated learning

    A Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing Systems, 35:38831–38843, 2022

    2022

  29. [37]

    Regularizing towards causal invariance: Linear models with proxies

    Michael Oberst, Nikolaj Thams, Jonas Peters, and David Sontag. Regularizing towards causal invariance: Linear models with proxies. In International Conference on Machine Learning, pages 8260–8270. PMLR, 2021

    2021

  30. [38]

    Effect of scale on catastrophic forgetting in neural networks

    Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021

    2021

  31. [39]

    Advances and open challenges in federated learning with foundation models

    Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Anran Li, Yulan Gao, Alysa Ziying Tan, Bo Zhao, Xiaoxiao Li, Zengxiang Li, et al. Advances and open challenges in federated learning with foundation models. arXiv preprint arXiv:2404.15381, 2024. 12

    2024 arXiv

  32. [40]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  33. [41]

    Fedbpt: Efficient federated black-box prompt tuning for large language models

    Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models. arXiv preprint arXiv:2310.01467, 2023

    2023 arXiv

  34. [42]

    Improving lora in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313, 2024

    2024 arXiv

  35. [43]

    Personalized federated learning with moreau envelopes

    Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in neural information processing systems, 33:21394–21405, 2020

    2020

  36. [44]

    Towards personalized federated learning

    Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022

    2022

  37. [45]

    Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning

    Yue Tan, Chen Chen, Weiming Zhuang, Xin Dong, Lingjuan Lyu, and Guodong Long. Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  38. [46]

    Learning personalized causally invariant representations for heterogeneous federated clients

    Xueyang Tang, Song Guo, Jie Zhang, and Jingcai Guo. Learning personalized causally invariant representations for heterogeneous federated clients. In The Twelfth International Conference on Learning Representations, 2023

    2023

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [48]

    Deep hashing network for unsupervised domain adaptation

    Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017

    2017

  41. [49]

    A field guide to federated optimization

    Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al- Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021

    2021 arXiv

  42. [50]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  43. [51]

    Perada: Parameter-efficient federated learning personalization with generalization guarantees

    Chulin Xie, De-An Huang, Wenda Chu, Daguang Xu, Chaowei Xiao, Bo Li, and Anima Anandkumar. Perada: Parameter-efficient federated learning personalization with generalization guarantees. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  44. [52]

    Fwdllm: Efficient fedllm using forward gradient

    M Xu, D Cai, Y Wu, X Li, and S Wang. Fwdllm: Efficient fedllm using forward gradient. 2024

    2024

  45. [53]

    Federated adversarial domain hallucination for privacy-preserving domain generalization

    Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yi-Yan Wu, and Yanfeng Wang. Federated adversarial domain hallucination for privacy-preserving domain generalization. IEEE Transactions on Multimedia, 26:1–14, 2023

    2023

  46. [54]

    Causalvae: Disentangled representation learning via neural structural causal models

    Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9593–9602, 2021

    2021

  47. [55]

    Dual- personalizing adapter for federated foundation models

    Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein. Dual- personalizing adapter for federated foundation models. arXiv preprint arXiv:2403.19211 , 2024

    2024 arXiv

  48. [56]

    Fedlora: Model-heterogeneous personal- ized federated learning with lora tuning

    Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personal- ized federated learning with lora tuning. arXiv preprint arXiv:2310.13283, 2023

    2023 arXiv

  49. [57]

    Federated foundation models: Privacy-preserving and collaborative learning for large models

    Sixing Yu, J Pablo Muñoz, and Ali Jannesari. Federated foundation models: Privacy-preserving and collaborative learning for large models. arXiv preprint arXiv:2305.11414, 2023

    2023 arXiv

  50. [58]

    What do we mean by generalization in federated learning? arXiv preprint arXiv:2110.14216, 2021

    Honglin Yuan, Warren Morningstar, Lin Ning, and Karan Singhal. What do we mean by generalization in federated learning? arXiv preprint arXiv:2110.14216, 2021

    2021 arXiv

  51. [59]

    Towards building the federated gpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen. Towards building the federated gpt: Federated instruction tuning. arXiv preprint arXiv:2305.05644, 2023. 13

    2023 arXiv

  52. [60]

    Federated learning with domain generalization

    Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization. arXiv preprint arXiv:2111.10487, 2021

    2021 arXiv

  53. [61]

    Feder- ated domain generalization with generalization adjustment

    Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Feder- ated domain generalization with generalization adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3954–3963, 2023

    2023

  54. [62]

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre- trained language models

    Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre- trained language models. In Annual Meeting of the Association of Computational Linguistics 2023, pages ...

    2023

  55. [63]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  56. [64]

    Fedfa: Federated learning with feature anchors to align features and classifiers for heterogeneous data

    Tailin Zhou, Jun Zhang, and Danny HK Tsang. Fedfa: Federated learning with feature anchors to align features and classifiers for heterogeneous data. IEEE Transactions on Mobile Computing, 2023

    2023

  57. [65]

    leave-one-task-out

    Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546, 2023. A Appendix The Appendix is organized as follows: • Appendix B provides related works. • Appendix ...

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.