Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FedProj prevents federated-learning clients from forgetting the global decision boundary by projecting local gradients against a global-knowledge gradient, and the paper reports that it outperforms existing non-IID methods on vision and…

desk verdict A promising FL method with a real algorithmic inconsistency between the theory and the printed pseudocode; the empirical gains are plausible but the paper needs a major revision before the results can be trusted. read the letter →

arxiv 2505.20485 v3 pith:SXAAJVYW submitted 2025-05-26 cs.LG cs.AIcs.CVcs.DCcs.PF

Abhijit Chunduru , Majid Morafah , Mahdi Morafah , Vishnu Pandi Chellapandi , Ang Li This is my paper
classification cs.LGcs.AIcs.CVcs.DCcs.PF MSC 68T0768T05
keywords federatedlearningnon-IIDdatacatastrophicforgettinggradientprojectionknowledgedistillationdecisionboundaryensemblelogitspublicdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated learning with non-IID data degrades because each client, training only on its own skewed data, forgets the global decision boundary learned by the server; the paper demonstrates this forgetting in a toy experiment, even when starting from pre-trained weights. To prevent it, the paper introduces FedProj, which during local training constrains each gradient update to stay aligned with a 'global knowledge gradient' computed as the KL divergence between the client's logits and the server's averaged ensemble logits on a small public dataset. At the server, it then distills the ensembled client models into the global model with a weight-divergence regularizer. The paper reports that FedProj outperforms existing methods on CIFAR-10/100, CINIC-10, MNLI, SST-2, and MARC under Dirichlet skew.

What carries the argument

The central mechanism is a projected-gradient update that preserves global knowledge while minimizing the local loss. The projection solves $\min_{g_{\text{proj}}} \tfrac{1}{2}\|g_{\text{new}} - g_{\text{proj}}\|^2$ subject to $\langle g_{\text{proj}}, g_{\text{glob}}\rangle \geq 0$, whose closed-form solution removes the component of $g_{\text{new}}$ that conflicts with the global-knowledge gradient. The global-knowledge gradient itself is defined as the gradient of the KL divergence between the client's softmax logits and the server's averaged ensemble logits on a small public dataset; on the server side, ensemble distillation with a weight-divergence regularizer fuses the client models.

What would settle it

Run FedProj on a non-IID split while replacing the public dataset with one drawn from a very different domain (for example, random noise images), and check whether the reported accuracy gains vanish; or measure $L_{\text{mem}}$ during 20 local epochs to verify that it never increases despite the first-order constraint, which would indicate higher-order effects dominate.

Watch

Extended reading notes

Core claim

The paper claims that catastrophic forgetting of the global decision boundary is a primary cause of FedAvg's poor performance on non-IID data, and that this forgetting can be avoided by a first-order gradient constraint: replace the local gradient $g_{\text{new}}$ with its projection $g_{\text{proj}}$ that is non-negatively correlated with $g_{\text{glob}} = \nabla L_{\text{mem}}$, where $L_{\text{mem}}$ measures the KL divergence to the server's ensemble logits on public data. The closed-form projection $g_{\text{proj}} = g_{\text{new}}$ if $\langle g_{\text{new}}, g_{\text{glob}}\rangle \geq 0$, and $g_{\text{proj}} = g_{\text{new}} - \frac{\langle g_{\text{new}}, g_{\text{glob}}\rangle}{\|g_{\text{glob}}\|^2 + \epsilon} g_{\text{glob}}$ otherwise, guarantees that the memory loss does not increase to first order. Combined with server-side ensemble distillation and weight regularization, this preserves the global boundary and yields the reported accuracy gains.

Load-bearing premise

The method assumes that the server-averaged logits computed on a small public dataset truly represent the global knowledge that must be preserved; if that public dataset is unrepresentative of the clients' private distributions, the projection preserves a proxy 'global boundary' instead of the true one.

Editorial extensions

If this is right

  • Clients can train with a simple closed-form gradient modification, so FedProj adds negligible per-step cost to FedAvg.
  • The method works across vision and language tasks, suggesting that preserving global logits on a public set transfers across modalities.
  • Because the server-side fusion uses only logits and weights, FedProj does not require clients to share data or model architectures.
  • The ablation on projection rate shows that the constraint is the main driver of accuracy under high skew, pinning the mechanism to the gradient-alignment step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The first-order constraint only guarantees non-increase of memory loss locally; a natural testable extension is to incorporate a second-order or trust-region correction when curvature is high, which could further stabilize long local training.
  • FedProj's reliance on a public dataset could be relaxed by synthesizing proxy logits from the server model itself, as in data-free distillation, at the cost of some accuracy.
  • The gradient projection can be interpreted as a proximal operator with respect to the global-knowledge loss, which may connect FedProj to existing convergence theory for proximal federated optimization.
  • The reported gains with public sets that are not class-aligned to the private data suggest the public distribution can deviate substantially; quantifying how far it can deviate before gains vanish would be a useful robustness study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FedProj, a federated learning method for non-IID data. It first presents a pilot study on the Iris dataset claiming that standard FL methods (FedAvg, FedDF) catastrophically forget the global decision boundary during local training. The main method combines two components: a client-side gradient projection that constrains local updates to preserve global knowledge, and a server-side ensemble knowledge distillation step with a weight divergence regularizer. The authors report experiments on CIFAR-10, CIFAR-100, CINIC-10, MNLI, SST-2, and MARC, claiming that FedProj outperforms FedAvg, FedProx, FedNova, FedDyn, MOON, FedRCL, FedDF, and FedET by large margins under Dirichlet heterogeneity with beta in {0.3, 0.5}.

Significance. If the method is specified consistently and the reported gains are reproducible, FedProj would be a meaningful contribution: it targets catastrophic forgetting at the local-update level, provides an anonymous code link, reports gains across both CV and NLP benchmarks, and includes ablations showing the importance of the projection component. The KKT derivation in Appendix A.1 for the conditional projection is correct for the stated quadratic program. However, the paper currently contains load-bearing inconsistencies between the conceptual derivation, the printed Algorithm 1, and the server-side objective, so the empirical claims are not reproducible from the text alone. The significance is therefore conditional on resolving these inconsistencies.

major comments (4)
  1. [§4.2, Eq. (3)–(5) and Algorithm 1, lines 8–10] The paper defines two incompatible versions of the core projection step. In Section 4.2, Lmem is the KL divergence between softmaxed server ensemble logits and local logits on an unlabeled public dataset, and the projection in Eq. (14) is applied only when the angle condition ⟨gnew, gglob⟩ < 0 holds. In Algorithm 1, line 9 defines gglob as the gradient of supervised cross-entropy on labeled public data, and line 10 always subtracts the gglob component from gnew, regardless of the sign of the inner product. Unconditional subtraction is not the solution of the quadratic program in Eq. (5), and it can increase Lmem when the original gradient already reduces it. Because the entire 'global knowledge preservation' claim depends on what gglob actually is, the manuscript does not specify a single method. Please align the algorithm with the derivation, or present the implemented variant as the method and justify why the conditional constraint is replaced by unconditional subtraction.
  2. [§4.2, Eq. (3)] The memory loss Lmem is not computable at the point where it is used. Eq. (3) defines Zserver(xm) as the average of f_l(xm; θ(t+1)_l) over selected clients, i.e., it uses the client models after the current round's local training. During client k's local updates, those models do not yet exist, so the gradient gglob = ∇θk Lmem(θk) cannot be evaluated. Please specify an available ensemble source, such as the previous global model or the previous round's client models, and update Eq. (3) and Algorithm 1 accordingly.
  3. [§4.3 and Algorithm 1, line 24] The weight divergence regularizer is defined inconsistently. Section 4.3 defines Ldiv = ||θk − θg||², where θk is a local model parameter, but Eq. (8) uses Ldiv(θg) as a term in the server-side objective, and Algorithm 1 line 24 defines Ldiv = α ||θg − θ(t)_g||², i.e., divergence from the previous global model. These are different regularizers and lead to different server updates. Please specify the exact server-side objective and the meaning of the hyperparameter reported in Table 3b.
  4. [§1 and §4.3] The paper claims feature distillation as part of the server-side fusion. Section 1 mentions 'ensemble knowledge distillation using both logits and feature representations', and Section 4.3 says the server update combines 'logit distillation, feature distillation, and weight divergence regularization', but Eqs. (7)–(8) and Algorithm 1 lines 23–25 define only logit distillation and weight divergence. No feature-level loss is provided. Please either define the feature distillation objective or remove this claim from the method description.
minor comments (5)
  1. [§4.2, Eq. (7) and Algorithm 1, line 23] The temperature scaling is inconsistent: Eq. (7) is written as plain KL divergence, Algorithm 1 line 23 uses T²·KL(σ(Zstudent/T), σ(Zteacher/T)), and Section 5.1 states T=3. Please align the notation and state clearly whether the T² factor is used in the reported experiments.
  2. [§3] The pilot study in Figure 1 includes FedProj but does not describe FedProj's hyperparameters (e.g., memory buffer size, projection settings, server distillation epochs) or whether the full method was used. Please specify the setup so the comparison is reproducible.
  3. [§5.3, Figure 3a] The projection dropout ablation is not described in enough detail: it is unclear whether the projection is omitted per client, per layer, per step, or with some other granularity, and how the 'projection rate' is varied. Please provide the implementation details.
  4. [References] References [50] and [51] are duplicates of the same paper (Zhu et al., 'Data-free knowledge distillation for heterogeneous federated learning'). Please remove the duplicate.
  5. [General] There are several typos and formatting issues, including 'allieviate' in Section 2, 'nosiy' in Section 4.3, and inconsistent spacing in 'F EDPROJ' in Section 5.1. A careful proofread is needed.

Circularity Check

1 steps flagged · score 2.0 of 10

Global-knowledge target in Eq. (3) is self-referential, but the reported accuracies are externally measured and not forced by the projection derivation.

  1. self definitional [Section 4.2, Eq. (3) and Eq. (4)]
    "Let Zserver(xm) = 1/|St| X ℓ∈St fℓ(xm; θ(t+1) ℓ ), be the server-aggregated logits ... Then we define Lmem(θk) = 1/|M| X (xm)∈M KL σ(Zserver(xm)), σ(Zk(xm; θk)) (3) ... min θk Llocal(θk) subject to Lmem(θk) ≤ Lmem(θold k) (4)"

    The 'global knowledge' that FedProj claims to preserve is defined as the server ensemble of the very local models being constrained: Zserver averages fℓ(·; θ(t+1)_ℓ), the post-update client models, while Lmem penalizes the current local model's KL divergence to that average. Since θ(t+1)_ℓ includes the model being updated, the objective defines the thing to be preserved in terms of the optimization variables that are supposed to preserve it; satisfying the constraint means 'do not move far from the ensemble of which you are a member.' This is a definitional circularity in the stated formulation, though the benchmark accuracies are held-out measurements and are not forced by this construction.

full rationale

The core derivation—the quadratic program in Eq. (5) and its solution Eq. (14)—is a standard projection and does not assume its conclusion; the KKT proof in Appendix A.1 is algebraically self-contained. The reported accuracies in Tables 1 and 2 are measured on held-out test sets, not read off from the projection rule, so no fitted parameter is later renamed as a prediction. The only definitional circularity is in Section 4.2, where 'global knowledge' is defined via Zserver, the ensemble of the local models being constrained, making Lmem a self-referential target. This is not the source of the benchmark numbers because Algorithm 1 actually computes gglob from labeled public data using cross-entropy rather than from the ensemble-KL loss, and the abstract's 'unlabeled dataset' claim is inconsistent with that algorithm. That inconsistency is a reproducibility/correctness risk, not a circularity that forces the results. Self-citations (e.g., [33] FedZoo and related-work items [34]-[36]) are used for implementation context and related work, not to justify the central claim. Overall circularity is minor and confined to the paper's stated definition of global knowledge.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the heuristic first-order projection being a sufficient forgetting constraint, the availability and representativeness of a public proxy dataset, and the ensemble-logits-as-global-knowledge identification. No new entities are postulated; the method is a combination of known techniques. The main free parameters (weight divergence, temperature, distillation epochs) are tuned on test performance and not fully disclosed for the headline results.

free parameters (4)
  • Weight divergence coefficient α (or λ) = 0.1, 0.3, 0.5, or 0, selected per dataset and β in Table 3b
    Regularization strength in server-side objective (Eq. 8) was tuned on test accuracy; the value used in the main results (Table 1) is not stated.
  • Distillation temperature T = 3
    Used in server-side KL distillation (Algorithm 1 line 23); chosen by hand.
  • Distillation epochs Ed = 1 epoch (CIFAR-10, CINIC-10, NLP) or 3 epochs (CIFAR-100)
    Number of server-side optimization steps, set per dataset.
  • Memory buffer size |M|
    Subset of Dpub used for Lmem; size not specified in the paper.
assumptions (3)
  • domain assumption First-order Taylor approximation of the memory-loss constraint is sufficient
    Section 4.2 replaces Lmem(θk) ≤ Lmem(θold) with the infinitesimal condition ⟨gproj, gglob⟩ ≥ 0; no higher-order analysis is provided.
  • domain assumption A small public dataset Dpub is available and is a faithful proxy for the global decision boundary
    The whole method (Eq. 3, server distillation) depends on Dpub being representative; the paper uses datasets like CIFAR-100 for CIFAR-10, which are related but not the same distribution.
  • domain assumption Average ensemble logits over selected clients encode the global knowledge
    Lmem and teacher logits assume the mean of client logits on the public data is the global target; this is asserted, not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data." pith.science (2026). https://pith.science/paper/SXAAJVYW

@misc{pith2026250520485,
  author       = {Pith},
  title        = {Pith review of: Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXAAJVYW}},
  note         = {Machine review of arXiv:2505.20485}
}
read the original abstract

The inevitable presence of data heterogeneity has made federated learning very challenging. There are numerous methods to deal with this issue, such as local regularization, better model fusion techniques, and data sharing. Though effective, they lack a deep understanding of how data heterogeneity can affect the global decision boundary. In this paper, we bridge this gap by performing an experimental analysis of the learned decision boundary using a toy example. Our observations are surprising: (1) we find that the existing methods suffer from forgetting and clients forget the global decision boundary and only learn the perfect local one, and (2) this happens regardless of the initial weights, and clients forget the global decision boundary even starting from pre-trained optimal weights. In this paper, we present FedProj, a federated learning framework that robustly learns the global decision boundary and avoids its forgetting during local training. To achieve better ensemble knowledge fusion, we design a novel server-side ensemble knowledge transfer loss to further calibrate the learned global decision boundary. To alleviate the issue of learned global decision boundary forgetting, we further propose leveraging an episodic memory of average ensemble logits on a public unlabeled dataset to regulate the gradient updates at each step of local training. Experimental results demonstrate that FedProj outperforms state-of-the-art methods by a large margin.

Figures

Figures reproduced from arXiv: 2505.20485 by the authors.

Figure 1
Figure 1. Visualization of Catastrophic Forgetting of Global Decision Boundaries under Non￾IID Federated Learning. This figure illustrates that standard federated learning methods (FedAvg and FedDF) experience significant catastrophic forgetting of global decision boundaries after local training on Non-IID client data, leading to poor global model performance (64%). In comparison, our proposed FedProj method successfully pres… view at source ↗
Figure 2
Figure 2. Gradient Projection in FedProj: Local gradient updates (glocal) are projected onto a subspace orthogonal to the global gradient (gglob), resulting in the projected vector (gproj) for better knowledge retention [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dissecting Federated-Graph Aggregation under Domain Shift: Importance-Aware Aggregation via Empirical Analysis

    cs.LG 2025-09 conditional novelty 5.0 of 10

    A top-rho gradient masking plus influence-weighted averaging method (FedIA) improves federated graph learning accuracy and stability under domain shift.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Federated learning based on dynamic regularization

    Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021

    arXiv 2021

  2. [2]

    A large annotated corpus for learning natural language inference

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015

    arXiv 2015

  3. [3]

    On the convergence of decentralized federated learning under imperfect information sharing

    Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, and Stanislaw H ˙Zak. On the convergence of decentralized federated learning under imperfect information sharing. IEEE Control Systems Letters, 2023

    work page 2023

  4. [4]

    Brinton, Stanislaw H Zak, and Zi- ran Wang

    Vishnu Pandi Chellapandi, Liangqi Yuan, Christopher G. Brinton, Stanislaw H Zak, and Zi- ran Wang. Federated learning for connected and automated vehicles: A survey of existing approaches and challenges. IEEE Transactions on Intelligent Vehicles, November 2023

    work page 2023

  5. [5]

    A Survey of Federated Learning for Connected and Automated Vehicles

    Vishnu Pandi Chellapandi, Liangqi Yuan, Stanislaw H Zak, and Ziran Wang. A survey of federated learning for connected and automated vehicles. arXiv preprint arXiv:2303.10677, 2023

    work page Pith review arXiv 2023

  6. [6]

    Fednmut – federated noisy model update tracking convergence analysis

    Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, and Stanislaw H ˙Zak. Fednmut – federated noisy model update tracking convergence analysis. arXiv preprint arXiv:2403.13247, 2024

    arXiv 2024

  7. [7]

    Fedgems: Federated learning of larger server models via selective knowledge fusion

    Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. Fedgems: Federated learning of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027, 2021

    arXiv 2021

  8. [8]

    Hetero- geneous ensemble knowledge transfer for training large models in federated learning

    Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Hetero- geneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703, 2022

    arXiv 2022

Show all 51 references
  1. [9]

    Exploiting shared representations for personalized federated learning

    Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089–2099. PMLR, 2021

    2021

  2. [10]

    Cinic-10 is not imagenet or cifar-10

    Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505, 2018

    2018 arXiv

  3. [11]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  4. [12]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics , pages 3762–3773. PMLR, 2020

    2020

  5. [13]

    Sharp bounds for federated averaging (local sgd) and continuous perspective

    Margalit R Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. In International Conference on Artificial Intelligence and Statistics, pages 9050–9090. PMLR, 2022

    2022

  6. [14]

    Twitter sentiment classification using distant super- vision

    Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant super- vision. CS224N project report, Stanford, 1(12):2009, 2009

    2009

  7. [15]

    Preserving privacy in federated learning with ensemble cross-domain knowledge distillation

    Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doer- mann, and Arun Innanje. Preserving privacy in federated learning with ensemble cross-domain knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 3...

    2022

  8. [16]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017. 10

    2017

  9. [17]

    Group knowledge transfer: Fed- erated learning of large cnns at the edge

    Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Fed- erated learning of large cnns at the edge. Advances in neural information processing systems, 33:14068–14080, 2020

    2020

  10. [18]

    Measuring the effects of non-identical data distribution for federated visual classification

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019

    1909 arXiv

  11. [19]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019

    1909 arXiv

  12. [20]

    Advances and open problems in federated learning.Foundations and trends® in machine learning, 14(1–2):1–210, 2021

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur ´elien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning.Foundations and trends® in machine learning, 14...

    2021

  13. [21]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020

    2020

  14. [22]

    The multilingual amazon reviews corpus

    Phillip Keung, Yichao Lu, Gy ¨orgy Szarvas, and Noah A Smith. The multilingual amazon reviews corpus. arXiv preprint arXiv:2010.02573, 2020

    2010 arXiv

  15. [23]

    Learning multiple layers of features from tiny im- ages.(2009), 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages.(2009), 2009

    2009

  16. [24]

    Fedmd: Heterogenous federated learning via model distillation

    Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019

    1910 arXiv

  17. [25]

    Model-contrastive federated learning

    Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10713– 10722, 2021

    2021

  18. [26]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020

    2020

  19. [27]

    Ditto: Fair and robust feder- ated learning through personalization

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust feder- ated learning through personalization. In International conference on machine learning, pages 6357–6368. PMLR, 2021

    2021

  20. [28]

    On the conver- gence of fedavg on non-iid data

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the conver- gence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019

    1907 arXiv

  21. [29]

    Fedbn: Federated learning on non-iid features via local batch normalization

    Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021

    2021 arXiv

  22. [30]

    Trgp: Trust region gradient projection for continual learning

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.02931, 2022

    2022 arXiv

  23. [31]

    Ensemble distillation for robust model fusion in federated learning

    Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in neural information processing systems , 33: 2351–2363, 2020

    2020

  24. [32]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  25. [33]

    A practical recipe for federated learning under statistical heterogeneity experimental design

    Mahdi Morafah, Weijia Wang, and Bill Lin. A practical recipe for federated learning under statistical heterogeneity experimental design. IEEE Transactions on Artificial Intelligence , 5 (4):1708–1717, 2023. 11

    2023

  26. [34]

    Large scale delocalized federated learning over a huge diversity of devices in emerging next-generation edge intelligence environments

    Mahdi Morafah, Hojin Chang, and Bill Lin. Large scale delocalized federated learning over a huge diversity of devices in emerging next-generation edge intelligence environments. InPro- ceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design , pages 1–8, 2024

    2024

  27. [35]

    Stable diffusion-based data augmentation for federated learning with non-iid data.arXiv preprint arXiv:2405.07925, 2024

    Mahdi Morafah, Matthias Reisser, Bill Lin, and Christos Louizos. Stable diffusion-based data augmentation for federated learning with non-iid data.arXiv preprint arXiv:2405.07925, 2024

    2024 arXiv

  28. [36]

    Clustered federated learning: A review

    Majid Morafah and Mahdi Morafah. Clustered federated learning: A review. Federated Learning-A Systematic Review, 2025

    2025

  29. [37]

    Continual learning with scaled gradient projection

    Gobinda Saha and Kaushik Roy. Continual learning with scaled gradient projection. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 37, pages 9677–9685, 2023

    2023

  30. [38]

    Gradient projection memory for continual learn- ing

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learn- ing. arXiv preprint arXiv:2103.09762, 2021

    2021 arXiv

  31. [39]

    Relaxed contrastive learning for federated learning

    Seonguk Seo, Jinkyu Kim, Geeho Kim, and Bohyung Han. Relaxed contrastive learning for federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12279–12288, 2024

    2024

  32. [40]

    Personalized federated learning using hypernetworks

    Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International conference on machine learning , pages 9489–9502. PMLR, 2021

    2021

  33. [41]

    Recursive deep models for semantic compositionality over a sen- timent treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sen- timent treebank. In Proceedings of the 2013 conference on empirical methods in natural lan- guage p...

    2013

  34. [42]

    Tackling the ob- jective inconsistency problem in heterogeneous federated optimization

    Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the ob- jective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020

    2020

  35. [43]

    Uncertainty estimation and reduction of pre-trained models for text regression

    Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. Uncertainty estimation and reduction of pre-trained models for text regression. Transactions of the Association for Computational Linguistics, 10:680–696, 2022

    2022

  36. [44]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017

    2017 arXiv

  37. [45]

    Minibatch vs local sgd for het- erogeneous distributed learning

    Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for het- erogeneous distributed learning. Advances in Neural Information Processing Systems , 33: 6281–6292, 2020

    2020

  38. [46]

    Uncertainty quantification with pre-trained language models: A large-scale empirical analysis

    Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04714, 2022

    arXiv

  39. [47]

    Federated continual learning via knowledge fusion: A survey

    Xin Yang, Hao Yu, Xin Gao, Hao Wang, Junbo Zhang, and Tianrui Li. Federated continual learning via knowledge fusion: A survey. IEEE Transactions on Knowledge and Data Engi- neering, 36(8):3832–3850, 2024

    2024

  40. [48]

    Continual learning of context-dependent processing in neural networks

    Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364–372, 2019

    2019

  41. [49]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

  42. [50]

    Data-free knowledge distillation for hetero- geneous federated learning

    Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for hetero- geneous federated learning. In International conference on machine learning , pages 12878– 12889. PMLR, 2021

    2021

  43. [51]

    Data-free knowledge distillation for hetero- geneous federated learning

    Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for hetero- geneous federated learning. In International conference on machine learning , pages 12878– 12889. PMLR, 2021. 12 A Appendix A.1 Detailed Derivation and Proof for Gradient Projection We pro...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.