Decentralized SGD achieves high-probability convergence with order-optimal rates and linear speedup under standard cost assumptions matching those for MSE convergence.
hub Canonical reference
Federated Learning: Strategies for Improving Communication Efficiency
Canonical reference. 90% of citing Pith papers cite this work as background.
47
Pith papers citing it
Background
90% of classified citations
abstract
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.
hub tools
citation-role summary
background 9
method 1
citation-polarity summary
representative citing papers
For decentralized secure aggregation with at least U surviving users and at most T colluders, the optimal two-round rates are R1 ≥ 1 and R2 ≥ 1/(U-T-1) when U > T+1, and the task is impossible otherwise.
Inkheart SGD and M4 use bidirectional compression to achieve time complexities in distributed SGD that improve with worker count n and surpass prior lower bounds under a necessary structural assumption.
Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.
FSCLB scales federated linear contextual bandits with sketching to achieve over 90% lower computation and communication costs while preserving a near-optimal regret bound of O(sqrt(l d T)).
XFED is the first aggregation-agnostic non-collusive model poisoning attack that bypasses eight state-of-the-art defenses on six benchmark datasets without attacker coordination.
A scalar-projection federated zeroth-order method for model-free LQR policy learning that reduces per-agent communication from O(d) to O(1) with convergence rate improving in the number of agents.
SketchGuard decouples Byzantine filtering from aggregation in decentralized federated learning by exchanging k-dimensional Count Sketches for screening and full models only from accepted neighbors, achieving up to 50-70% communication savings while proving convergence and matching SOTA robustness.
DMBA maintains attack success rates above 80% for all backdoors in a distributed multi-target FL setting where baselines drop below 50%.
New analysis framework yields tighter linear convergence for FedExProx on non-strongly convex quadratics and PL functions, proving outperformance over GD once communication costs are counted.
T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colossal Clean Crawled Corpus.
Introduces FedHybrid and FedNewton for DP federated M-estimation, with finite-sample MSE bounds, minimax lower bound, and evaluations on vision datasets.
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and better empirical performance.
FAR-SIGN achieves adversary-resilient fully asynchronous optimization via signed directional projections and two-timescale correction, with almost-sure convergence to stationary points at rates O(n^{-1/4+ε}) first-order and O(n^{-1/6+ε}) zeroth-order.
Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.
MoR lets clients train local reward models on private preferences and uses a learned Mixture-of-Rewards with GRPO on the server to align a shared base VLM without exchanging parameters, architectures, or raw data.
The paper derives tight information-theoretic bounds on communication and key rates for secure multi-server aggregation under heterogeneous security constraints and arbitrary collusion, with matching schemes in most regimes and a bounded-gap scheme in the rest.
For hierarchical secure aggregation with groupwise keys of size G>1, the optimal rate region is fully characterized with user and relay rates at least 1 and minimum groupwise key rate max of two combinatorial terms.
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
Jellyfish enables zero-shot federated unlearning through synthetic proxy data generation, channel-restricted knowledge disentanglement, and a composite loss with repair to forget target data while retaining model utility.
A trust-region stabilized proximal point method enforces a displacement condition to achieve linear descent for general nonsmooth convex problems.
DeepFedNAS delivers up to 1.21% higher accuracy and 61x faster architecture search for federated learning on heterogeneous IoT by replacing random supernet sampling with Pareto-optimal elite architectures and using a multi-objective fitness function as a zero-cost proxy.
Sufficient conditions using the Wasserstein metric of order 1 are derived to calibrate Laplace noise for pufferfish privacy in multi-user aggregated queries, with relaxations for binary data that reduce noise while preserving indistinguishability.
DFedReweighting is a unified reweighting method for decentralized federated learning that customizes aggregation via target metrics and strategies to improve fairness, Byzantine robustness, and other objectives while proving linear convergence under standard assumptions.
citing papers explorer
-
Information-Theoretic Decentralized Secure Aggregation with User Dropouts
For decentralized secure aggregation with at least U surviving users and at most T colluders, the optimal two-round rates are R1 ≥ 1 and R2 ≥ 1/(U-T-1) when U > T+1, and the task is impossible otherwise.
-
Multi-Server Secure Aggregation with Arbitrary Collusion and Heterogeneous Security Constraints
The paper derives tight information-theoretic bounds on communication and key rates for secure multi-server aggregation under heterogeneous security constraints and arbitrary collusion, with matching schemes in most regimes and a bounded-gap scheme in the rest.
-
On the Capacity of Hierarchical Secure Aggregation with Groupwise Keys
For hierarchical secure aggregation with groupwise keys of size G>1, the optimal rate region is fully characterized with user and relay rates at least 1 and minimum groupwise key rate max of two combinatorial terms.
-
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
-
Split and Aggregation Learning for Foundation Models Over Mobile Embodied AI Network (MEAN): A Comprehensive Survey
The paper surveys split and aggregation learning for foundation models in 6G networks to improve efficiency, resource use, and data privacy in distributed AI.