Pith. sign in

REVIEW 5 cited by

Distributed Learning with Compressed Gradient Differences

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1901.09269 v3 pith:JIQOTZ7J submitted 2019-01-26 cs.LG math.OCstat.ML

Konstantin Mishchenko , Eduard Gorbunov , Martin Takáč , Peter Richtárik This is my paper · ORCID
classification cs.LGmath.OCstat.ML
keywords analysisdifferencesdistributedlearningquantizationcompressiongradientmethod
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods based on the compression (e.g., sparsification and/or quantization) of updates were recently proposed, including QSGD (Alistarh et al., 2017), TernGrad (Wen et al., 2017), SignSGD (Bernstein et al., 2018), and DQGD (Khirirat et al., 2018). However, none of these methods are able to learn the gradients, which renders them incapable of converging to the true optimum in the batch mode. In this work we propose a new distributed learning method -- DIANA -- which resolves this issue via compression of gradient differences. We perform a theoretical analysis in the strongly convex and nonconvex settings and show that our rates are superior to existing rates. We also provide theory to support non-smooth regularizers study the difference between quantization schemes. Our analysis of block-quantization and differences between $\ell_2$ and $\ell_{\infty}$ quantization closes the gaps in theory and practice. Finally, by applying our analysis technique to TernGrad, we establish the first convergence rate for this method.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Federated Majorize-Minimization: Beyond Parameter Aggregation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    By averaging surrogate-function parameters across clients and then minimizing the aggregated surrogate on the server, federated learning can converge under heterogeneity where parameter averaging diverges.

  2. Beyond Communication Overhead: A Multilevel Monte Carlo Approach for Mitigating Compression Bias in Distributed Learning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A multilevel Monte Carlo framework debiases biased gradient compressors, preserving SGD convergence guarantees while reducing communication cost, with adaptive variance-minimizing level selection.

  3. Optimization Methods and Software for Federated Learning

    cs.LG 2025-09 conditional novelty 4.0 of 10

    A thesis that packages the author's published federated learning work, whose main new theoretical result is an improved complexity bound for error-feedback compression.

  4. Quantized Rank Reduction: A Communications-Efficient Federated Learning Scheme for Network-Critical Applications

    cs.LG 2025-07 conditional novelty 3.0 of 10

    A federated learning scheme that compresses client gradients with truncated SVD or Tucker decomposition and then quantizes the components, cutting transmitted bits by about 90 to 97 percent at some accuracy cost.

  5. Event-Driven Online Vertical Federated Learning

    cs.LG 2025-06

Pith tools