REVIEW 7 cited by
On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Existing theory predicts that data heterogeneity will degrade the performance of the Federated Averaging (FedAvg) algorithm in federated learning. However, in practice, the simple FedAvg algorithm converges very well. This paper explains the seemingly unreasonable effectiveness of FedAvg that contradicts the previous theoretical predictions. We find that the key assumption of bounded gradient dissimilarity in previous theoretical analyses is too pessimistic to characterize data heterogeneity in practical applications. For a simple quadratic problem, we demonstrate there exist regimes where large gradient dissimilarity does not have any negative impact on the convergence of FedAvg. Motivated by this observation, we propose a new quantity, average drift at optimum, to measure the effects of data heterogeneity, and explicitly use it to present a new theoretical analysis of FedAvg. We show that the average drift at optimum is nearly zero across many real-world federated training tasks, whereas the gradient dissimilarity can be large. And our new analysis suggests FedAvg can have identical convergence rates in homogeneous and heterogeneous data settings, and hence, leads to better understanding of its empirical success.
Forward citations
Cited by 7 Pith papers
-
Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression
A two-stage Local GD with learning-rate warmup achieves O(1/(K R)) convergence for heterogeneous distributed logistic regression, proving that local steps can provably reduce communication rounds.
-
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Local SGD provably improves over Mini-batch SGD under bounded second-order heterogeneity in the general convex setting, with nearly tight upper and lower bounds.
-
Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability
For separable logistic regression, Local GD with any step size and any communication interval converges at rate O~(1/(eta K R)) after O~(eta K M) unstable rounds, beating the general O(1/R) worst-case bound.
-
Federated Learning from Molecules to Processes: A Perspective
Federated learning lets chemical companies train shared models on private data, and two case studies show it approaches centralized accuracy while outperforming isolated training.
-
Task Arithmetic Through The Lens Of One-Shot Federated Learning
Task arithmetic is exactly one-shot FedAvg with outer step size beta = lambda T, and FedNova, FedGMA, Median, and CCLIP can often improve merged model performance.
-
What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness
Under bounded second-order heterogeneity, local updates are shown to achieve faster convergence than mini-batch SGD in several convex and non-convex regimes, with matching lower bounds.
-
How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy
A practical, extremely thorough survey of differentially private synthetic data generation: methods, privacy units, evaluation metrics, and end-to-end system components across four data modalities.
Discussion (0). Continue with ORCID to comment.