REVIEW 4 cited by
Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Federated learning (FL) has emerged as a prominent distributed learning paradigm. FL entails some pressing needs for developing novel parameter estimation approaches with theoretical guarantees of convergence, which are also communication efficient, differentially private and Byzantine resilient in the heterogeneous data distribution settings. Quantization-based SGD solvers have been widely adopted in FL and the recently proposed SIGNSGD with majority vote shows a promising direction. However, no existing methods enjoy all the aforementioned properties. In this paper, we propose an intuitively-simple yet theoretically-sound method based on SIGNSGD to bridge the gap. We present Stochastic-Sign SGD which utilizes novel stochastic-sign based gradient compressors enabling the aforementioned properties in a unified framework. We also present an error-feedback variant of the proposed Stochastic-Sign SGD which further improves the learning performance in FL. We test the proposed method with extensive experiments using deep neural networks on the MNIST dataset and the CIFAR-10 dataset. The experimental results corroborate the effectiveness of the proposed method.
Forward citations
Cited by 4 Pith papers
-
Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness
First high-probability bounds for SignSGD with batching or majority voting under (L0, L1)-smoothness and heavy-tailed noise, with near-optimal epsilon-dependencies.
-
Improved Analysis for Sign-based Methods with Momentum Updates
SignSGD with momentum attains O(d^{1/2}T^{-1/4}) gradient-norm convergence under standard L2 smoothness and O(T^{-1/4}) under L-infinity smoothness, with improved distributed majority-vote rates.
-
Byzantine Outside, Curious Inside: Reconstructing Data Through Malicious Updates
A single malicious client in federated learning can amplify gradient inversion by poisoning its own updates, and common defenses such as robust aggregation or local differential privacy may fail or even increase data leakage.
-
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.
Discussion (0). Continue with ORCID to comment.