Decentralized SGD achieves high-probability convergence with order-optimal rates and linear speedup under standard cost assumptions matching those for MSE convergence.
hub Canonical reference
Federated Learning: Strategies for Improving Communication Efficiency
Canonical reference. 90% of citing Pith papers cite this work as background.
47
Pith papers citing it
Background
90% of classified citations
abstract
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.
hub tools
citation-role summary
background 9
method 1
citation-polarity summary
representative citing papers
For decentralized secure aggregation with at least U surviving users and at most T colluders, the optimal two-round rates are R1 ≥ 1 and R2 ≥ 1/(U-T-1) when U > T+1, and the task is impossible otherwise.
Inkheart SGD and M4 use bidirectional compression to achieve time complexities in distributed SGD that improve with worker count n and surpass prior lower bounds under a necessary structural assumption.
Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.
FSCLB scales federated linear contextual bandits with sketching to achieve over 90% lower computation and communication costs while preserving a near-optimal regret bound of O(sqrt(l d T)).
XFED is the first aggregation-agnostic non-collusive model poisoning attack that bypasses eight state-of-the-art defenses on six benchmark datasets without attacker coordination.
A scalar-projection federated zeroth-order method for model-free LQR policy learning that reduces per-agent communication from O(d) to O(1) with convergence rate improving in the number of agents.
SketchGuard decouples Byzantine filtering from aggregation in decentralized federated learning by exchanging k-dimensional Count Sketches for screening and full models only from accepted neighbors, achieving up to 50-70% communication savings while proving convergence and matching SOTA robustness.
DMBA maintains attack success rates above 80% for all backdoors in a distributed multi-target FL setting where baselines drop below 50%.
New analysis framework yields tighter linear convergence for FedExProx on non-strongly convex quadratics and PL functions, proving outperformance over GD once communication costs are counted.
T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colossal Clean Crawled Corpus.
Introduces FedHybrid and FedNewton for DP federated M-estimation, with finite-sample MSE bounds, minimax lower bound, and evaluations on vision datasets.
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and better empirical performance.
FAR-SIGN achieves adversary-resilient fully asynchronous optimization via signed directional projections and two-timescale correction, with almost-sure convergence to stationary points at rates O(n^{-1/4+ε}) first-order and O(n^{-1/6+ε}) zeroth-order.
Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.
MoR lets clients train local reward models on private preferences and uses a learned Mixture-of-Rewards with GRPO on the server to align a shared base VLM without exchanging parameters, architectures, or raw data.
The paper derives tight information-theoretic bounds on communication and key rates for secure multi-server aggregation under heterogeneous security constraints and arbitrary collusion, with matching schemes in most regimes and a bounded-gap scheme in the rest.
For hierarchical secure aggregation with groupwise keys of size G>1, the optimal rate region is fully characterized with user and relay rates at least 1 and minimum groupwise key rate max of two combinatorial terms.
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
Jellyfish enables zero-shot federated unlearning through synthetic proxy data generation, channel-restricted knowledge disentanglement, and a composite loss with repair to forget target data while retaining model utility.
A trust-region stabilized proximal point method enforces a displacement condition to achieve linear descent for general nonsmooth convex problems.
DeepFedNAS delivers up to 1.21% higher accuracy and 61x faster architecture search for federated learning on heterogeneous IoT by replacing random supernet sampling with Pareto-optimal elite architectures and using a multi-objective fitness function as a zero-cost proxy.
Sufficient conditions using the Wasserstein metric of order 1 are derived to calibrate Laplace noise for pufferfish privacy in multi-user aggregated queries, with relaxations for binary data that reduce noise while preserving indistinguishability.
DFedReweighting is a unified reweighting method for decentralized federated learning that customizes aggregation via target metrics and strategies to improve fairness, Byzantine robustness, and other objectives while proving linear convergence under standard assumptions.
citing papers explorer
-
High-Probability Convergence Guarantees of Decentralized SGD
Decentralized SGD achieves high-probability convergence with order-optimal rates and linear speedup under standard cost assumptions matching those for MSE convergence.
-
Information-Theoretic Decentralized Secure Aggregation with User Dropouts
For decentralized secure aggregation with at least U surviving users and at most T colluders, the optimal two-round rates are R1 ≥ 1 and R2 ≥ 1/(U-T-1) when U > T+1, and the task is impossible otherwise.
-
Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits
Inkheart SGD and M4 use bidirectional compression to achieve time complexities in distributed SGD that improve with worker count n and surpass prior lower bounds under a necessary structural assumption.
-
Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.
-
Scaling Federated Linear Contextual Bandits via Sketching
FSCLB scales federated linear contextual bandits with sketching to achieve over 90% lower computation and communication costs while preserving a near-optimal regret bound of O(sqrt(l d T)).
-
XFED: Non-Collusive Model Poisoning Attack Against Byzantine-Robust Federated Classifiers
XFED is the first aggregation-agnostic non-collusive model poisoning attack that bypasses eight state-of-the-art defenses on six benchmark datasets without attacker coordination.
-
Scalar Federated Learning for Linear Quadratic Regulator
A scalar-projection federated zeroth-order method for model-free LQR policy learning that reduces per-agent communication from O(d) to O(1) with convergence rate improving in the number of agents.
-
SketchGuard: Scaling Byzantine-Robust Decentralized Federated Learning via Sketch-Based Screening
SketchGuard decouples Byzantine filtering from aggregation in decentralized federated learning by exchanging k-dimensional Count Sketches for screening and full models only from accepted neighbors, achieving up to 50-70% communication savings while proving convergence and matching SOTA robustness.
-
Act in Collusion: Distributed Multi-Target Backdoor Attacks in Federated Learning
DMBA maintains attack success rates above 80% for all backdoors in a distributed multi-target FL setting where baselines drop below 50%.
-
Tighter Performance Theory of FedExProx
New analysis framework yields tighter linear convergence for FedExProx on non-strongly convex quadratics and PL functions, proving outperformance over GD once communication costs are counted.
-
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colossal Clean Crawled Corpus.
-
Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning
Introduces FedHybrid and FedNewton for DP federated M-estimation, with finite-sample MSE bounds, minimax lower bound, and evaluations on vision datasets.
-
Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and better empirical performance.
-
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
FAR-SIGN achieves adversary-resilient fully asynchronous optimization via signed directional projections and two-timescale correction, with almost-sure convergence to stationary points at rates O(n^{-1/4+ε}) first-order and O(n^{-1/6+ε}) zeroth-order.
-
Response Time Enhances Alignment with Heterogeneous Preferences
Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.
-
Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
MoR lets clients train local reward models on private preferences and uses a learned Mixture-of-Rewards with GRPO on the server to align a shared base VLM without exchanging parameters, architectures, or raw data.
-
Multi-Server Secure Aggregation with Arbitrary Collusion and Heterogeneous Security Constraints
The paper derives tight information-theoretic bounds on communication and key rates for secure multi-server aggregation under heterogeneous security constraints and arbitrary collusion, with matching schemes in most regimes and a bounded-gap scheme in the rest.
-
On the Capacity of Hierarchical Secure Aggregation with Groupwise Keys
For hierarchical secure aggregation with groupwise keys of size G>1, the optimal rate region is fully characterized with user and relay rates at least 1 and minimum groupwise key rate max of two combinatorial terms.
-
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
-
Jellyfish: Zero-Shot Federated Unlearning Scheme with Knowledge Disentanglement
Jellyfish enables zero-shot federated unlearning through synthetic proxy data generation, channel-restricted knowledge disentanglement, and a composite loss with repair to forget target data while retaining model utility.
-
Stabilized Proximal Point Method via Trust Region Control
A trust-region stabilized proximal point method enforces a displacement condition to achieve linear descent for general nonsmooth convex problems.
-
DeepFedNAS: Efficient Hardware-Aware Architecture Adaptation for Heterogeneous IoT Federations via Pareto-Guided Supernet Training
DeepFedNAS delivers up to 1.21% higher accuracy and 61x faster architecture search for federated learning on heterogeneous IoT by replacing random supernet sampling with Pareto-optimal elite architectures and using a multi-objective fitness function as a zero-cost proxy.
-
Multi-user Pufferfish Privacy
Sufficient conditions using the Wasserstein metric of order 1 are derived to calibrate Laplace noise for pufferfish privacy in multi-user aggregated queries, with relaxations for binary data that reduce noise while preserving indistinguishability.
-
DFedReweighting: A Unified Framework for Objective-Oriented Reweighting in Decentralized Federated Learning
DFedReweighting is a unified reweighting method for decentralized federated learning that customizes aggregation via target metrics and strategies to improve fairness, Byzantine robustness, and other objectives while proving linear convergence under standard assumptions.
-
Federated Learning with Nonvacuous Generalisation Bounds
Federated learning trains private local randomised predictors whose aggregation yields a global predictor with nonvacuous PAC-Bayesian generalisation bounds and near-centralized accuracy.
-
Federated Learning with Non-IID Data
Non-IID data causes up to 55% accuracy loss in federated learning due to weight divergence measured by earth mover's distance; 5% globally shared data recovers 30% accuracy on CIFAR-10.
-
Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback
SPEAR enables online federated LLM fine-tuning by using feedback-guided self-play to create contrastive pairs trained with maximum likelihood on correct completions and confidence-weighted unlikelihood on incorrect ones, outperforming baselines without ground-truth contexts.
-
Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
SSF enables efficient federated learning under heterogeneous data by optimizing in a low-dimensional subspace with projected corrections and backfill updates, achieving a non-asymptotic convergence rate of order O~(1/T + 1/sqrt(NKT)).
-
FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
Fed-FSTQ reduces uplink traffic by 46x and improves time-to-accuracy by 52% in federated LLM fine-tuning using Fisher-guided token quantization and selection.
-
Enhanced Privacy and Communication Efficiency in Non-IID Federated Learning with Adaptive Quantization and Differential Privacy
Adaptive bit-length schedulers plus Laplacian DP in non-IID FL reduce communicated data by up to 52.64% on MNIST and 45% on CIFAR-10 while keeping competitive accuracy and privacy.
-
PubSwap: Public-Data Off-Policy Coordination for Federated RLVR
PubSwap uses a small public dataset for selective off-policy response swapping in federated RLVR to improve coordination and performance over standard baselines on math and medical reasoning tasks.
-
Representation-Aligned Multi-Scale Personalization for Federated Learning
FRAMP generates client-specific models from compact descriptors in federated learning, trains tailored submodels, and aligns representations to balance personalization with global consistency.
-
Communication-Efficient Gluon in Federated Learning
Compressed Gluon variants using unbiased/contraction compressors and SARAH-style variance reduction achieve convergence guarantees and lower communication costs in federated learning under layer-wise smoothness.
-
Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation
A complete pipeline for federated unlearning via knowledge distillation for efficient removal and a GAN-integrated classifier for visual evaluation of forgetting capacity.
-
On Signal Peak Power Constraint of Over-the-Air Federated Learning
Over-the-air federated learning exceeds amplifier peak-power limits in practice, and clipping-filtering mitigation degrades performance especially in multi-carrier OFDM systems.
-
Optimizing Split Federated Learning with Unstable Client Participation
The paper derives the first convergence upper bound for split federated learning under activation upload, gradient download, and aggregation failures, then jointly optimizes client sampling and model splitting to minimize the bound, with simulations on EMNIST and CIFAR-10.
-
Stochastic versus Deterministic in Stochastic Gradient Descent
Treating stochastic and deterministic gradients separately in mini-batch SGD yields faster convergence and smaller error radius than uniform treatment, with further gains under strong convexity.
-
Decentralized Rank Scheduling for Energy-Constrained Multi-Task Federated Fine-Tuning in Edge-Assisted IoV Networks
Presents a hierarchical energy-aware framework with UCB-DUAL bandit for decentralized rank scheduling in multi-task federated fine-tuning for IoV networks.
-
BoBa: Boosting Backdoor Detection through Data Distribution Inference in Federated Learning
BoBa uses data distribution inference and overlapping clustering with voting to detect backdoor attacks in non-IID federated learning, claiming attack success rates below 0.001.
-
Centralized vs Decentralized Federated Learning: A trade-off performance analysis
Experimental analysis of performance trade-offs across CFL, DFL, and SDFL using Fedstellar simulator, MNIST, and MLP.
-
Understanding Communication Backends in Cross-Silo Federated Learning
Benchmarks of MPI, gRPC, and PyTorch RPC in cross-silo FL plus a new gRPC+S3 hybrid backend deliver up to 3.8x speedup for large-model transmission under realistic network conditions.
-
Privacy-Preserving Federated Learning: Integrating Zero-Knowledge Proofs in Scalable Distributed Architectures
A hybrid federated learning architecture using zero-knowledge proofs for computation verification retains 94.2% accuracy under adversarial conditions across 1,000 nodes.
-
Split and Aggregation Learning for Foundation Models Over Mobile Embodied AI Network (MEAN): A Comprehensive Survey
The paper surveys split and aggregation learning for foundation models in 6G networks to improve efficiency, resource use, and data privacy in distributed AI.
-
A Survey on Foundation Models for Personalized Federated Intelligence
The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.
-
Active Learning Solution on Distributed Edge Computing
A hybrid approach applies active learning at edge devices and federated learning at fog nodes to reduce training data volume and communication cost for image classification in distributed edge-fog setups.
-
Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
A survey organizing knowledge distillation techniques for addressing privacy, heterogeneity, communication, and personalization challenges in federated learning.
- Modulated learning for private and distributed regression with just a single sample per client device