Decentralized SGD achieves high-probability convergence with order-optimal rates and linear speedup under standard cost assumptions matching those for MSE convergence.
hub Canonical reference
Federated Learning: Strategies for Improving Communication Efficiency
Canonical reference. 90% of citing Pith papers cite this work as background.
59
Pith papers citing it
Background
90% of classified citations
abstract
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. We consider learning algorithms for this setting where on each round, each client independently computes an update to the current model based on its local data, and communicates this update to a central server, where the client-side updates are aggregated to compute a new global model. The typical clients in this setting are mobile phones, and communication efficiency is of the utmost importance. In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, where we learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling before sending it to the server. Experiments on both convolutional and recurrent networks show that the proposed methods can reduce the communication cost by two orders of magnitude.
hub tools
citation-role summary
background 9
method 1
citation-polarity summary
representative citing papers
TIGER turns the low-rank attention gradient subspace into a differentiable objective for continuous embedding optimization, improving reconstruction quality and robustness over prior discrete token tests especially under noise or DP.
The capacity region among randomness for security, key-distribution communication, and aggregation communication is completely characterized for T-colluding secure aggregation with N users under a general two-phase user-to-user key distribution framework.
For decentralized secure aggregation with at least U surviving users and at most T colluders, the optimal two-round rates are R1 ≥ 1 and R2 ≥ 1/(U-T-1) when U > T+1, and the task is impossible otherwise.
Inkheart SGD and M4 use bidirectional compression to achieve time complexities in distributed SGD that improve with worker count n and surpass prior lower bounds under a necessary structural assumption.
Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.
FSCLB scales federated linear contextual bandits with sketching to achieve over 90% lower computation and communication costs while preserving a near-optimal regret bound of O(sqrt(l d T)).
XFED is the first aggregation-agnostic non-collusive model poisoning attack that bypasses eight state-of-the-art defenses on six benchmark datasets without attacker coordination.
A scalar-projection federated zeroth-order method for model-free LQR policy learning that reduces per-agent communication from O(d) to O(1) with convergence rate improving in the number of agents.
SketchGuard decouples Byzantine filtering from aggregation in decentralized federated learning by exchanging k-dimensional Count Sketches for screening and full models only from accepted neighbors, achieving up to 50-70% communication savings while proving convergence and matching SOTA robustness.
DMBA maintains attack success rates above 80% for all backdoors in a distributed multi-target FL setting where baselines drop below 50%.
New analysis framework yields tighter linear convergence for FedExProx on non-strongly convex quadratics and PL functions, proving outperformance over GD once communication costs are counted.
T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colossal Clean Crawled Corpus.
Proposes a covariance-aware tuning-free shrinkage framework and sequential algorithm for multi-source estimation that attains oracle risk asymptotically and improves on single-step methods.
GASLoC generalizes communication acceleration to the outer optimizer to enable gossip-based decentralized LLM pretraining that supports adaptive optimizers, local steps, and outperforms prior decentralized methods on standard tasks while matching DiLoCo in multi-step regimes.
PrivacyCredit is a machine learning method that combines traditional and alternative data for credit risk prediction while satisfying privacy-preserving, model-confidential, and lossless properties.
ELCP integrates auxiliary data with a density-ratio-weighted kernel to enhance localized conformal prediction sets, maintaining marginal coverage and improving asymptotic local coverage.
DIST-FL distributes TEE-guarded servers into an append-only ledger to ensure linearizable FL aggregation and counter rollback plus I/O attacks while matching single-TEE speed.
FlashbackCL adds time-decayed label counts, class-balanced replay, and coreset curation to Flashback, yielding 6.9-10% gains and up to 68% less temporal forgetting on CIFAR-10 under controlled shifts.
PDR uses sparse random projection to reduce server computation for Byzantine-robust FL aggregation to O(Mp) while preserving near-optimal convergence rates up to a tunable error inflation factor of (1+ε)/(1-ε).
Nonlinear kernel integration (NKI) with graph regularization enables accurate alignment of nonlinearly obfuscated decentralized data representations, outperforming linear methods on image classification.
Introduces FedHybrid and FedNewton for DP federated M-estimation, with finite-sample MSE bounds, minimax lower bound, and evaluations on vision datasets.
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and better empirical performance.
FAR-SIGN achieves adversary-resilient fully asynchronous optimization via signed directional projections and two-timescale correction, with almost-sure convergence to stationary points at rates O(n^{-1/4+ε}) first-order and O(n^{-1/6+ε}) zeroth-order.
citing papers explorer
-
High-Probability Convergence Guarantees of Decentralized SGD
Decentralized SGD achieves high-probability convergence with order-optimal rates and linear speedup under standard cost assumptions matching those for MSE convergence.
-
Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
Two randomized Hadamard transforms suffice to make coordinate marginals O(d^{-1/2})-close to Gaussian for most quantization methods, with three needed for vector quantization to match uniform random rotations asymptotically.
-
Scaling Federated Linear Contextual Bandits via Sketching
FSCLB scales federated linear contextual bandits with sketching to achieve over 90% lower computation and communication costs while preserving a near-optimal regret bound of O(sqrt(l d T)).
-
SketchGuard: Scaling Byzantine-Robust Decentralized Federated Learning via Sketch-Based Screening
SketchGuard decouples Byzantine filtering from aggregation in decentralized federated learning by exchanging k-dimensional Count Sketches for screening and full models only from accepted neighbors, achieving up to 50-70% communication savings while proving convergence and matching SOTA robustness.
-
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
T5 casts all NLP tasks as text-to-text generation, systematically explores pre-training choices, and reaches strong performance on summarization, QA, classification and other tasks via large-scale training on the Colossal Clean Crawled Corpus.
-
Unifying Local Communications and Local Updates for LLM Pretraining
GASLoC generalizes communication acceleration to the outer optimizer to enable gossip-based decentralized LLM pretraining that supports adaptive optimizers, local steps, and outperforms prior decentralized methods on standard tasks while matching DiLoCo in multi-step regimes.
-
Privacy-Preserving Credit Risk Prediction with Alternative Data
PrivacyCredit is a machine learning method that combines traditional and alternative data for credit risk prediction while satisfying privacy-preserving, model-confidential, and lossless properties.
-
FlashbackCL: Mitigating Temporal Forgetting in Federated Learning
FlashbackCL adds time-decayed label counts, class-balanced replay, and coreset curation to Flashback, yielding 6.9-10% gains and up to 68% less temporal forgetting on CIFAR-10 under controlled shifts.
-
Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee
PDR uses sparse random projection to reduce server computation for Byzantine-robust FL aggregation to O(Mp) while preserving near-optimal convergence rates up to a tunable error inflation factor of (1+ε)/(1-ε).
-
Nonlinear Data Integration via Kernel Methods for Data Collaboration Analysis
Nonlinear kernel integration (NKI) with graph regularization enables accurate alignment of nonlinearly obfuscated decentralized data representations, outperforming linear methods on image classification.
-
Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and better empirical performance.
-
Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates
FAR-SIGN achieves adversary-resilient fully asynchronous optimization via signed directional projections and two-timescale correction, with almost-sure convergence to stationary points at rates O(n^{-1/4+ε}) first-order and O(n^{-1/6+ε}) zeroth-order.
-
Modulated learning for private and distributed regression with just a single sample per client device
Introduces modulated learning for private distributed regression allowing one sample per client via calibrated noise injection on samples and aggregation of transformed representations to achieve unbiased gradients in expectation.
-
Response Time Enhances Alignment with Heterogeneous Preferences
Response times modeled as drift-diffusion processes enable consistent estimation of population-average preferences from heterogeneous anonymous binary choices.
-
DeepFedNAS: Efficient Hardware-Aware Architecture Adaptation for Heterogeneous IoT Federations via Pareto-Guided Supernet Training
DeepFedNAS delivers up to 1.21% higher accuracy and 61x faster architecture search for federated learning on heterogeneous IoT by replacing random supernet sampling with Pareto-optimal elite architectures and using a multi-objective fitness function as a zero-cost proxy.
-
DFedReweighting: A Unified Framework for Objective-Oriented Reweighting in Decentralized Federated Learning
DFedReweighting is a unified reweighting method for decentralized federated learning that customizes aggregation via target metrics and strategies to improve fairness, Byzantine robustness, and other objectives while proving linear convergence under standard assumptions.
-
Federated Learning with Nonvacuous Generalisation Bounds
Federated learning trains private local randomised predictors whose aggregation yields a global predictor with nonvacuous PAC-Bayesian generalisation bounds and near-centralized accuracy.
-
Federated Learning with Non-IID Data
Non-IID data causes up to 55% accuracy loss in federated learning due to weight divergence measured by earth mover's distance; 5% globally shared data recovers 30% accuracy on CIFAR-10.
-
Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback
SPEAR enables online federated LLM fine-tuning by using feedback-guided self-play to create contrastive pairs trained with maximum likelihood on correct completions and confidence-weighted unlikelihood on incorrect ones, outperforming baselines without ground-truth contexts.
-
Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
SSF enables efficient federated learning under heterogeneous data by optimizing in a low-dimensional subspace with projected corrections and backfill updates, achieving a non-asymptotic convergence rate of order O~(1/T + 1/sqrt(NKT)).
-
PubSwap: Public-Data Off-Policy Coordination for Federated RLVR
PubSwap uses a small public dataset for selective off-policy response swapping in federated RLVR to improve coordination and performance over standard baselines on math and medical reasoning tasks.
-
Representation-Aligned Multi-Scale Personalization for Federated Learning
FRAMP generates client-specific models from compact descriptors in federated learning, trains tailored submodels, and aligns representations to balance personalization with global consistency.
-
Communication-Efficient Gluon in Federated Learning
Compressed Gluon variants using unbiased/contraction compressors and SARAH-style variance reduction achieve convergence guarantees and lower communication costs in federated learning under layer-wise smoothness.
-
Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation
A complete pipeline for federated unlearning via knowledge distillation for efficient removal and a GAN-integrated classifier for visual evaluation of forgetting capacity.
-
Decentralized Rank Scheduling for Energy-Constrained Multi-Task Federated Fine-Tuning in Edge-Assisted IoV Networks
Presents a hierarchical energy-aware framework with UCB-DUAL bandit for decentralized rank scheduling in multi-task federated fine-tuning for IoV networks.
-
BoBa: Boosting Backdoor Detection through Data Distribution Inference in Federated Learning
BoBa uses data distribution inference and overlapping clustering with voting to detect backdoor attacks in non-IID federated learning, claiming attack success rates below 0.001.
-
FoggyTrust: Robust Federated Learning with Hierarchical Trust Networks
FoggyTrust is a hierarchical extension of FLTrust that localizes trust computation to fog nodes and combines it with heterogeneity-aware optimizers, reporting over 50% gains on CIFAR-10 under Krum and Trim attacks.
-
Centralized vs Decentralized Federated Learning: A trade-off performance analysis
Experimental analysis of performance trade-offs across CFL, DFL, and SDFL using Fedstellar simulator, MNIST, and MLP.
-
Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
A survey organizing knowledge distillation techniques for addressing privacy, heterogeneity, communication, and personalization challenges in federated learning.
- FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices