Think Locally, Act Globally: Federated Learning with Local and Global Representations
Reviewed by Pithpith:MI47VKV6open to challenge →
read the original abstract
Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
FedSAF shifts prototype alignment in heterogeneous federated learning from coordinate matching to inter-class structural relations and reports up to 3.52% gains over prior methods.
-
FedOBP: Federated Optimal Brain Personalization through Cloud-Edge Element-wise Decoupling
FedOBP introduces a quantile-thresholded importance score based on a federated first-order Taylor approximation to select a small set of parameters for personalization, claiming better performance than prior PFL methods.
-
Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning
FedTSP builds class prototypes from LLM-generated text descriptions via PLMs and trainable prompts to preserve semantic relationships and reduce heterogeneity effects in federated learning.
-
Tuning-Free Efficient Estimation for Multi-Source Data via Covariance-Aware Shrinkage
Proposes a covariance-aware tuning-free shrinkage framework and sequential algorithm for multi-source estimation that attains oracle risk asymptotically and improves on single-step methods.
-
FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
FedCoE proposes a coordinated dual-level MoE framework for federated learning that improves global and personalized accuracy while enabling strong cold-start performance for new clients.
-
FIRMA: FIbonacci Ring Model Aggregation for Privacy-preserving Federated Learning
FIRMA introduces Fibonacci ring aggregation protocols for server-free federated learning that maintain private heads and achieve higher accuracy than FedAvg under label skew across multiple benchmarks and heterogeneit...
-
On the Tradeoffs of On-Device Generative Models in Federated Predictive Maintenance Systems
Experiments on real industrial time series show that partial model sharing improves diffusion model performance in bandwidth-limited non-IID settings, while full sharing stabilizes GAN training but offers less robustn...
-
From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learning
A data-centric survey of federated learning that ranks non-IID data traits by influence on convergence, links splitting protocols to real phenomena, and examines data-related defenses under clean and adversarial conditions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.