Pith. sign in

REVIEW 2 minor 142 references

Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers

T0 review · 0 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient clipping removes the maximum delay dependence from the oracle complexity of asynchronous SGD.

desk verdict Clipping removes max delay from async SGD rates under sub-Weibull noise and delivers the first high-probability bound, but the tail assumption carries the result. read the letter →

arxiv 2606.13287 v1 pith:OQKJTO36 submitted 2026-06-11 cs.LG cs.DCmath.OC

Samuel Erickson , Mikael Johansson This is my paper
classification cs.LGcs.DCmath.OC
keywords asynchronousSGDgradientclippingstragglersconvergenceanalysissub-Weibullnoisedistributedtrainingfederatedlearninghigh-probability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that applying gradient clipping to asynchronous stochastic gradient descent removes the impact of the slowest workers on the convergence rate. This holds with constant step sizes under a model of gradient noise that covers heavy-tailed distributions observed in deep learning. The analysis establishes convergence both in expectation and with high probability, the latter appearing for the first time in asynchronous optimization. The result supplies a theoretical account for why clipping stabilizes training in distributed and federated settings.

What carries the argument

Gradient clipping applied to asynchronous updates, paired with a sub-Weibull model of gradient noise.

What would settle it

An experiment that measures oracle complexity with clipping and shows the rate still grows with the maximum delay.

Watch

Extended reading notes

Core claim

By clipping gradients before they are applied, the oracle complexity of asynchronous SGD becomes independent of the maximum delay among workers, yielding both expectation and high-probability convergence guarantees under a sub-Weibull model of gradient noise.

Load-bearing premise

Gradient noise is modeled as sub-Weibull distributed to derive the delay-independent bound.

Editorial extensions

If this is right

  • Asynchronous training can proceed without waiting for any worker, raising hardware utilization.
  • Convergence rates remain stable even when some workers are much slower than others.
  • High-probability bounds become available for the first time in this setting.
  • The same clipping step applies equally in distributed and federated environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same clipping step could be tested in other asynchronous first-order methods to check for similar delay independence.
  • Empirical checks of gradient tail behavior in large models would test whether the noise model fits observed data.
  • Scheduling policies that tolerate variable worker speeds might become preferable once clipping is in place.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript claims that gradient clipping in asynchronous SGD (ASGD), including distributed and federated variants, removes the dependence on the maximum delay au_max from the oracle complexity. This holds under a sub-Weibull model of gradient noise that generalizes sub-Gaussian and sub-exponential tails. The authors establish convergence in expectation and, for the first time in asynchronous optimization, convergence with high probability.

Significance. If the central claims hold, the work supplies a theoretical account for the empirical stabilization effect of clipping in asynchronous deep learning training on heterogeneous hardware. The sub-Weibull noise model is motivated by observed heavy-tailed gradients and enables both the delay-independent bound and the novel high-probability guarantee. These strengths—machine-checked-style theoretical results under a realistic noise model and the first HP convergence in the async setting—would make the contribution notable for robust parallel optimization.

minor comments (2)
  1. [Abstract] Abstract: states the main results but supplies no derivation details, error bounds, or explicit assumptions beyond the sub-Weibull model; adding a brief statement of the achieved rate (e.g., dependence on clipping threshold and tail parameter) would improve readability.
  2. [Introduction] The sub-Weibull assumption is load-bearing for both the delay-independent claim and the high-probability result; a short remark clarifying what happens under weaker moment conditions (even if outside the paper's scope) would help readers assess the result's robustness.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive evaluation of the manuscript and for recommending minor revision. The referee's summary correctly reflects our central claims regarding the effect of gradient clipping on the oracle complexity of asynchronous SGD under a sub-Weibull noise model, as well as the high-probability convergence result.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: standard convergence analysis under explicit noise assumption

full rationale

The paper derives delay-independent oracle complexity and high-probability convergence for clipped async SGD by applying standard martingale and concentration arguments to the clipped gradient updates under an explicit sub-Weibull noise model. No equations reduce to self-definitions, no fitted parameters are relabeled as predictions, and no load-bearing steps rely on self-citations or imported uniqueness theorems. The sub-Weibull tail parameter is stated as an assumption motivated by DL observations, not derived from the target bound, so the derivation chain remains independent of its own outputs.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the sub-Weibull noise model as the key domain assumption enabling the delay-independent bound.

assumptions (1)
  • domain assumption Gradient noise follows a sub-Weibull distribution
    Generalizes sub-Gaussian and sub-exponential distributions and is motivated by empirical observations in deep learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers." pith.science (2026). https://pith.science/paper/OQKJTO36

@misc{pith2026260613287,
  author       = {Pith},
  title        = {Pith review of: Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQKJTO36}},
  note         = {Machine review of arXiv:2606.13287}
}
read the original abstract

In modern machine learning, parallelization of training is an important strategy for increasing scale. Asynchronous stochastic gradient descent (ASGD), which maximizes the utilization of available hardware by avoiding waiting for slow workers. However, with constant step sizes, the convergence of ASGD is nonetheless affected negatively by slow workers due to large delays in updates. At the same time, it has been empirically observed in asynchronous training of deep learning models that gradient clipping "stabilizes" training. In this work, we provide a theoretical justification for this behavior, as we show that clipping removes the dependence of the maximum delay in the oracle complexity. We employ a sub-Weibull model of gradient noise which generalizes sub-Gaussian and sub-exponential distributions to more heavy-tailed distributions, motivated by empirical observations in deep learning. We show convergence in expectation, and the first time in asynchronous optimization, convergence with high probability.

Figures

Figures reproduced from arXiv: 2606.13287 by the authors.

Figure 1
Figure 1. Histograms of (a) gradient errors in training of a ResNet￾18 model on CIFAR-10 and (b) simulated sub-Weibull distribution. The empirical estimate of the tail parameter is 𝜃 = 2.71. This model of the gradient noise is used by Li & Liu (2022) and (Madden et al., 2024), who provide high probability guarantees for serial SGD. Note that if a random variable is sub-Weibull then it also has bounded second-moment. In partic… view at source ↗
Figure 2
Figure 2. shows the results for delay factors 𝐷 ∈ {4, 8}. Clipped ASGD consistently improves over Vanilla ASGD and Delay-adaptive ASGD across all delay settings. Here, Clipped ASGD reduces the minimum wall-clock time by 1.8× relative to Vanilla ASGD, and by 1.5× relative to Delay-adaptive and Ringmaster ASGD. As predicted by the theory, increasing the delay primarily affects Vanilla ASGD, which requires substantially smaller … view at source ↗
Figure 3
Figure 3. Simulated wall-clock time to reach test perplexity 5.0 on Shakespeare dataset with LSTM architecture, when half of the 16 workers are 𝐷 times slower than the other half. The average time per oracle call here is 0.108 and 0.123 time units for 𝐷 = 4 and 𝐷 = 8, respectively. pling scheme described in Section 5. As before, we tune the clipping radius over 𝑐 ∈ {2 𝑘 : 𝑘 = −1, ... , 2}. Note however that in practical FL de… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Clipped ASGD consistently improves over both baselines across heterogeneity levels. With delay factor 𝐷 = 4, clipping reduces the minimum wall-clock time by 1.2× relative to Vanilla ASGD and Ringleader ASGD. With 𝐷 = 8, Clipped ASGD is 1.3× faster than Vanilla ASGD, an…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

142 extracted references · 11 canonical work pages

  1. [1]

    Sharper Convergence Guarantees for Asynchronous

    Koloskova, Anastasiia and Stich, Sebastian U and Jaggi, Martin , booktitle =. Sharper Convergence Guarantees for Asynchronous

  2. [2]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Delay-Adaptive Step-sizes for Asynchronous Learning , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  3. [3]

    Asynchronous Stochastic Optimization Robust to Arbitrary Delays , url =

    Cohen, Alon and Daniely, Amit and Drori, Yoel and Koren, Tomer and Schain, Mariano , booktitle =. Asynchronous Stochastic Optimization Robust to Arbitrary Delays , url =

  4. [4]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Asynchronous Stochastic Gradient Descent with Delay Compensation , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =

    2017

  5. [6]

    Revisiting Distributed Synchronous

    Jianmin Chen and Rajat Monga and Samy Bengio and Rafal Jozefowicz , year =. Revisiting Distributed Synchronous

  6. [7]

    International Conference on Learning Representations , year =

    Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity , author =. International Conference on Learning Representations , year =

  7. [8]

    Improved Analysis of Clipping Algorithms for Non-convex Optimization , url =

    Zhang, Bohang and Jin, Jikai and Fang, Cong and Wang, Liwei , booktitle =. Improved Analysis of Clipping Algorithms for Non-convex Optimization , url =

  8. [9]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

Show all 142 references
  1. [10]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  2. [11]

    Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach , url =

    Fallah, Alireza and Mokhtari, Aryan and Ozdaglar, Asuman , booktitle =. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach , url =

  3. [12]

    2017 , editor =

    McMahan, Brendan and Moore, Eider and Ramage, Daniel and Hampson, Seth and Arcas, Blaise Aguera y , booktitle =. 2017 , editor =

    2017

  4. [13]

    arXiv preprint arXiv:2306.08393 , year =

    Provably personalized and robust federated learning , author =. arXiv preprint arXiv:2306.08393 , year =

  5. [14]

    2022 IEEE International Smart Cities Conference (ISC2) , pages =

    Personalized federated learning via convex clustering , author =. 2022 IEEE International Smart Cities Conference (ISC2) , pages =. 2022 , organization =

    2022

  6. [15]

    Electronics , volume =

    A Personalized Federated Learning Method Based on Clustering and Knowledge Distillation , author =. Electronics , volume =. 2024 , publisher =

    2024

  7. [16]

    Advances in Neural Information Processing Systems , volume =

    An efficient framework for clustered federated learning , author =. Advances in Neural Information Processing Systems , volume =

  8. [17]

    arXiv preprint arXiv:2003.13461 , year =

    Adaptive personalized federated learning , author =. arXiv preprint arXiv:2003.13461 , year =

    2003

  9. [18]

    Ando, Rie and Zhang, Tong , journal =

  10. [19]

    arXiv preprint arXiv:2310.01973 , year =

    Federated wasserstein distance , author =. arXiv preprint arXiv:2310.01973 , year =

  11. [20]

    Cédric Villani , title =

  12. [21]

    Asynchronous

    Mishchenko, Konstantin and Bach, Francis and Even, Mathieu and Woodworth, Blake E , booktitle =. Asynchronous

  13. [22]

    Stat , volume =

    Sub-Weibull distributions: Generalizing sub-Gaussian and sub-Exponential properties to heavier tailed distributions , author =. Stat , volume =. 2020 , publisher =

    2020

  14. [23]

    Localized Upper and Lower Bounds for Some Estimation Problems

    Zhang, Tong. Localized Upper and Lower Bounds for Some Estimation Problems. Learning Theory. 2005

    2005

  15. [24]

    Better Theory for

    Ahmed Khaled and Peter Richt. Better Theory for. Transactions on Machine Learning Research , issn =. 2023 , url =

    2023

  16. [25]

    IEEE Transactions on Automatic Control , volume =

    An asynchronous mini-batch algorithm for regularized stochastic optimization , author =. IEEE Transactions on Automatic Control , volume =. 2016 , publisher =

    2016

  17. [26]

    Advances in neural information processing systems , volume =

    Hogwild!: A lock-free approach to parallelizing stochastic gradient descent , author =. Advances in neural information processing systems , volume =

  18. [27]

    Advances in Neural Information Processing Systems , volume =

    Why are adaptive methods good for attention models? , author =. Advances in Neural Information Processing Systems , volume =

  19. [28]

    The heavy-tail phenomenon in

    Gurbuzbalaban, Mert and Simsekli, Umut and Zhu, Lingjiong , booktitle =. The heavy-tail phenomenon in. 2021 , organization =

    2021

  20. [29]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    High Probability Guarantees for Nonconvex Stochastic Gradient Descent with Heavy Tails , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  21. [30]

    Journal of Machine Learning Research , year =

    Liam Madden and Emiliano Dall'Anese and Stephen Becker , title =. Journal of Machine Learning Research , year =

  22. [31]

    2018 , editor =

    Nguyen, Lam and NGUYEN, PHUONG HA and van Dijk, Marten and Richtarik, Peter and Scheinberg, Katya and Takac, Martin , booktitle =. 2018 , editor =

    2018

  23. [32]

    Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16) , year =

    Wei Zhang and Suyog Gupta and Xiangru Lian and Ji Liu , title =. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16) , year =

  24. [33]

    Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , pages =

    AdaDelay: Delay Adaptive Distributed Stochastic Optimization , author =. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , pages =. 2016 , editor =

    2016

  25. [34]

    Advances in Neural Information Processing Systems , volume =

    Delay-tolerant algorithms for asynchronous distributed online learning , author =. Advances in Neural Information Processing Systems , volume =

  26. [35]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis , title =

  27. [36]

    Advances in neural information processing systems , volume =

    Distributed delayed stochastic optimization , author =. Advances in neural information processing systems , volume =

  28. [38]

    IEEE transactions on automatic control , volume =

    Distributed asynchronous deterministic and stochastic gradient optimization algorithms , author =. IEEE transactions on automatic control , volume =. 2003 , publisher =

    2003

  29. [39]

    International conference on artificial intelligence and statistics , pages =

    Federated learning with buffered asynchronous aggregation , author =. International conference on artificial intelligence and statistics , pages =. 2022 , organization =

    2022

  30. [40]

    Federated learning and analytics in practice: algorithms, systems, applications, and opportunities , year =

    Tackling the data heterogeneity in asynchronous federated learning with cached update calibration , author =. Federated learning and analytics in practice: algorithms, systems, applications, and opportunities , year =

  31. [41]

    Forty-second International Conference on Machine Learning , year =

    Faster Stochastic Optimization with Arbitrary Delays via Adaptive Asynchronous Mini-Batching , author =. Forty-second International Conference on Machine Learning , year =

  32. [42]

    Journal of Machine Learning Research , volume =

    Hamid Reza Feyzmahdavian and Mikael Johansson , title =. Journal of Machine Learning Research , volume =

  33. [43]

    ACM Computing Surveys (CSUR) , volume =

    Demystifying parallel and distributed deep learning: An in-depth concurrency analysis , author =. ACM Computing Surveys (CSUR) , volume =. 2019 , publisher =

    2019

  34. [44]

    Proceedings of Machine Learning and Systems , volume =

    Papaya: Practical, private, and scalable federated learning , author =. Proceedings of Machine Learning and Systems , volume =

  35. [45]

    OPT2020: 12th Annual Workshop on Optimization for Machine Learning , year =

    Asynchronous Federated Optimization , author =. OPT2020: 12th Annual Workshop on Optimization for Machine Learning , year =

  36. [46]

    2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages =

    Unbounded gradients in federated learning with buffered asynchronous aggregation , author =. 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages =. 2022 , organization =

    2022

  37. [47]

    Proceedings of the 41st International Conference on Machine Learning , series =

    FADAS: Towards Federated Adaptive Asynchronous Optimization , author =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , address =

    2024

  38. [48]

    Understanding Priors in

    Vladimirova, Mariia and Verbeek, Jakob and Mesejo, Pablo and Arbel , Julyan , booktitle =. Understanding Priors in. 2019 , editor =

    2019

  39. [49]

    Information and Inference: A Journal of the IMA , volume =

    Moving beyond sub-Gaussianity in high-dimensional statistics: Applications in covariance estimation and linear regression , author =. Information and Inference: A Journal of the IMA , volume =. 2022 , publisher =

    2022

  40. [50]

    Presentation at Google, Mountain View, 2nd April , volume =

    Statistical language models based on neural networks , author =. Presentation at Google, Mountain View, 2nd April , volume =

  41. [51]

    Mathematical Programming , volume =

    On the projected subgradient method for nonsmooth convex optimization in a Hilbert space , author =. Mathematical Programming , volume =. 1998 , publisher =

    1998

  42. [52]

    Book in preparation for MIT Press , author =

    Deep learning. Book in preparation for MIT Press , author =

  43. [53]

    Proceedings of the 30th International Conference on Machine Learning , pages =

    On the difficulty of training recurrent neural networks , author =. Proceedings of the 30th International Conference on Machine Learning , pages =. 2013 , editor =

    2013

  44. [54]

    IEEE transactions on neural networks , volume =

    Learning long-term dependencies with gradient descent is difficult , author =. IEEE transactions on neural networks , volume =. 1994 , publisher =

    1994

  45. [55]

    Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages =

    Deep learning with differential privacy , author =. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages =

    2016

  46. [56]

    Advances in Neural Information Processing Systems , volume =

    Differentially private empirical risk minimization revisited: Faster and more general , author =. Advances in Neural Information Processing Systems , volume =

  47. [57]

    International colloquium on automata, languages, and programming , pages =

    Differential privacy , author =. International colloquium on automata, languages, and programming , pages =. 2006 , organization =

    2006

  48. [58]

    Journal of the ACM (JACM) , volume =

    Privacy aware learning , author =. Journal of the ACM (JACM) , volume =. 2014 , publisher =

    2014

  49. [59]

    arXiv preprint arXiv:2406.04443 , year =

    Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed , author =. arXiv preprint arXiv:2406.04443 , year =

  50. [60]

    Advances in Neural Information Processing Systems , volume =

    Stochastic optimization with heavy-tailed noise via accelerated gradient clipping , author =. Advances in Neural Information Processing Systems , volume =

  51. [61]

    International Conference on Machine Learning , pages =

    A tail-index analysis of stochastic gradient noise in deep neural networks , author =. International Conference on Machine Learning , pages =. 2019 , organization =

    2019

  52. [62]

    Science Meets Engineering of Deep Learning (SEDL) Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS) , year =

    Non-Gaussianity of Stochastic Gradient Noise , author =. Science Meets Engineering of Deep Learning (SEDL) Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS) , year =

  53. [63]

    From Gradient Clipping to Normalization for Heavy Tailed

    H. From Gradient Clipping to Normalization for Heavy Tailed. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics , pages =. 2025 , editor =

    2025

  54. [64]

    High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails , url =

    Cutkosky, Ashok and Mehta, Harsh , booktitle =. High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails , url =

  55. [65]

    Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise , volume =

    Nguyen, Ta Duy and Nguyen, Thien H and Ene, Alina and Nguyen, Huy , booktitle =. Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise , volume =

  56. [66]

    2014 IEEE 55th annual symposium on foundations of computer science , pages =

    Private empirical risk minimization: Efficient algorithms and tight error bounds , author =. 2014 IEEE 55th annual symposium on foundations of computer science , pages =. 2014 , organization =

    2014

  57. [67]

    Foundations and trends

    The algorithmic foundations of differential privacy , author =. Foundations and trends. 2014 , publisher =

    2014

  58. [68]

    Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data , booktitle =

    Nicolas Papernot and Mart. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data , booktitle =

  59. [69]

    Brendan McMahan and Daniel Ramage and Kunal Talwar and Li Zhang , title =

    H. Brendan McMahan and Daniel Ramage and Kunal Talwar and Li Zhang , title =. Proceedings of the 6th International Conference on Learning Representations (ICLR) , year =

  60. [70]

    , title =

    Mania, Horia and Pan, Xinghao and Papailiopoulos, Dimitris and Recht, Benjamin and Ramchandran, Kannan and Jordan, Michael I. , title =. SIAM Journal on Optimization , volume =. 2017 , doi =

    2017

  61. [71]

    Stich and Sai Praneeth Karimireddy , title =

    Sebastian U. Stich and Sai Praneeth Karimireddy , title =. Journal of Machine Learning Research , year =

  62. [72]

    Advances in neural information processing systems , volume =

    Large scale distributed deep networks , author =. Advances in neural information processing systems , volume =

  63. [73]

    Understanding gradient clipping in private

    Chen, Xiangyi and Wu, Steven Z and Hong, Mingyi , journal =. Understanding gradient clipping in private

  64. [74]

    Advances in Neural Information Processing Systems , volume =

    Optimal time complexities of parallel stochastic optimization methods under a fixed computation model , author =. Advances in Neural Information Processing Systems , volume =

  65. [75]

    Learning multiple layers of features from tiny images.(2009) , author =

    2009

  66. [76]

    2015 , howpublished =

    Karpathy, Andrej , title =. 2015 , howpublished =

    2015

  67. [77]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

    Deep residual learning for image recognition , author =. Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

  68. [78]

    Neural computation , volume =

    Long short-term memory , author =. Neural computation , volume =. 1997 , publisher =

    1997

  69. [80]

    2024 , howpublished =

    Muon: An Optimizer for Hidden Layers in Neural Networks , author =. 2024 , howpublished =

    2024

  70. [81]

    International Conference on Learning Representations (ICLR) , year =

    Adam: A Method for Stochastic Optimization , author =. International Conference on Learning Representations (ICLR) , year =

  71. [82]

    Ringmaster

    Arto Maranjyan and Alexander Tyurin and Peter Richt. Ringmaster. Forty-second International Conference on Machine Learning , year =

  72. [83]

    Ringleader

    Arto Maranjyan and Peter Richt. Ringleader. The Fourteenth International Conference on Learning Representations , year =

  73. [86]

    Workshop on Federated Learning for Data Privacy and Confidentiality , year =

    LEAF: A Benchmark for Federated Settings , author =. Workshop on Federated Learning for Data Privacy and Confidentiality , year =

  74. [87]

    B., Mironov, I., Talwar, K., and Zhang, L

    Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016

    2016

  75. [88]

    and Duchi, J

    Agarwal, A. and Duchi, J. C. Distributed delayed stochastic optimization. Advances in neural information processing systems, 24, 2011

    2011

  76. [89]

    I., Iusem, A

    Alber, Y. I., Iusem, A. N., and Solodov, M. V. On the projected subgradient method for nonsmooth convex optimization in a hilbert space. Mathematical Programming, 81 0 (1): 0 23--35, 1998

    1998

  77. [90]

    Private empirical risk minimization: Efficient algorithms and tight error bounds

    Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pp.\ 464--473. IEEE, 2014

    2014

  78. [91]

    and Hoefler, T

    Ben-Nun, T. and Hoefler, T. Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Computing Surveys (CSUR), 52 0 (4): 0 1--43, 2019

    2019

  79. [92]

    Bertsekas, D. P. and Tsitsiklis, J. N. Parallel and Distributed Computation: Numerical Methods. Prentice-Hall, 1989

    1989

  80. [93]

    Revisiting distributed synchronous SGD

    Chen, J., Monga, R., Bengio, S., and Jozefowicz, R. Revisiting distributed synchronous SGD . In International Conference on Learning Representations Workshop Track, 2016. URL https://arxiv.org/abs/1604.00981

    2016 arXiv

  81. [94]

    Z., and Hong, M

    Chen, X., Wu, S. Z., and Hong, M. Understanding gradient clipping in private SGD : A geometric perspective. Advances in Neural Information Processing Systems, 33: 0 13773--13782, 2020

    2020

  82. [95]

    Asynchronous stochastic optimization robust to arbitrary delays

    Cohen, A., Daniely, A., Drori, Y., Koren, T., and Schain, M. Asynchronous stochastic optimization robust to arbitrary delays. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ ...

    2021

  83. [96]

    and Mehta, H

    Cutkosky, A. and Mehta, H. High-probability bounds for non-convex stochastic optimization with heavy tails. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 4883--4895. Curran...

    2021

  84. [97]

    Large scale distributed deep networks

    Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  85. [98]

    Heterofl: Computation and communication efficient federated learning for heterogeneous clients

    Diao, E., Ding, J., and Tarokh, V. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020

    2010

  86. [99]

    The algorithmic foundations of differential privacy

    Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and trends in theoretical computer science , 9 0 (3--4): 0 211--407, 2014

    2014

  87. [100]

    Feyzmahdavian, H. R. and Johansson, M. Asynchronous iterations in optimization: New sequence results and sharper algorithmic guarantees. Journal of Machine Learning Research, 24 0 (158): 0 1--75, 2023

    2023

  88. [101]

    R., Aytekin, A., and Johansson, M

    Feyzmahdavian, H. R., Aytekin, A., and Johansson, M. An asynchronous mini-batch algorithm for regularized stochastic optimization. IEEE Transactions on Automatic Control, 61 0 (12): 0 3740--3754, 2016

    2016

  89. [102]

    The heavy-tail phenomenon in SGD

    Gurbuzbalaban, M., Simsekli, U., and Zhu, L. The heavy-tail phenomenon in SGD . In International Conference on Machine Learning, pp.\ 3964--3975. PMLR, 2021

    2021

  90. [103]

    and Yin, W

    Hannah, R. and Yin, W. On unbounded delays in asynchronous parallel fixed-point algorithms. Journal of Scientific Computing, 76 0 (1): 0 299--326, 2018. ISSN 1573-7691. doi:10.1007/s10915-017-0628-z. URL https://doi.org/10.1007/s10915-017-0628-z

    2018 doi

  91. [104]

    Fedml: A research library and benchmark for federated machine learning

    He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518, 2020

    2007

  92. [105]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  93. [106]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  94. [107]

    Papaya: Practical, private, and scalable federated learning

    Huba, D., Nguyen, J., Malik, K., Zhu, R., Rabbat, M., Yousefpour, A., Wu, C.-J., Zhan, H., Ustinov, P., Srinivas, H., et al. Papaya: Practical, private, and scalable federated learning. Proceedings of Machine Learning and Systems, 4: 0 814--832, 2022

    2022

  95. [108]

    From gradient clipping to normalization for heavy tailed SGD

    H \"u bler, F., Fatkhullin, I., and He, N. From gradient clipping to normalization for heavy tailed SGD . In Li, Y., Mandt, S., Agrawal, S., and Khan, E. (eds.), Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedin...

    2025

  96. [109]

    Muon: An optimizer for hidden layers in neural networks

    Jordan, K., Jin, Y., Boza, V., Yu, J., Cecista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Online article

    2024

  97. [110]

    char-rnn

    Karpathy, A. char-rnn. https://github.com/karpathy/char-rnn, 2015

    2015

  98. [111]

    U., and Jaggi, M

    Koloskova, A., Stich, S. U., and Jaggi, M. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 3...

    2022

  99. [112]

    Koloskova, A., Hendrikx, H., and Stich, S. U. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine...

    2023

  100. [113]

    Learning multiple layers of features from tiny images.(2009), 2009

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images.(2009), 2009

    2009

  101. [114]

    Kuchibhotla, A. K. and Chakrabortty, A. Moving beyond sub-gaussianity in high-dimensional statistics: Applications in covariance estimation and linear regression. Information and Inference: A Journal of the IMA, 11 0 (4): 0 1389--1456, 2022

    2022

  102. [115]

    and Liu, Y

    Li, S. and Liu, Y. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 ...

    2022

  103. [116]

    High probability convergence bounds for non-convex stochastic gradient descent with sub-weibull noise

    Madden, L., Dall'Anese, E., and Becker, S. High probability convergence bounds for non-convex stochastic gradient descent with sub-weibull noise. Journal of Machine Learning Research, 25 0 (241): 0 1--36, 2024. URL http://jmlr.org/papers/v25/23-0466.html

    2024

  104. [117]

    Mai, V. V. and Johansson, M. Stability and convergence of stochastic gradient clipping: Beyond lipschitz continuity and smoothness. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Lea...

    2021

  105. [118]

    Mania, H., Pan, X., Papailiopoulos, D., Recht, B., Ramchandran, K., and Jordan, M. I. Perturbed iterate analysis for asynchronous stochastic optimization. SIAM Journal on Optimization, 27 0 (4): 0 2202--2229, 2017. doi:10.1137/16M1057000. URL https://doi.org/10.1137/16M1057000

    2017 doi

  106. [119]

    and Richt \'a rik, P

    Maranjyan, A. and Richt \'a rik, P. Ringleader ASGD : The first asynchronous SGD with optimal time complexity under data heterogeneity. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=5wqTal0EuC

    2026

  107. [120]

    Ringmaster ASGD : The first asynchronous SGD with optimal time complexity

    Maranjyan, A., Tyurin, A., and Richt \'a rik, P. Ringmaster ASGD : The first asynchronous SGD with optimal time complexity. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Rkgn9KLHhd

    2025

  108. [121]

    and Streeter, M

    McMahan, B. and Streeter, M. Delay-tolerant algorithms for asynchronous distributed online learning. Advances in Neural Information Processing Systems, 27, 2014

    2014

  109. [122]

    B., Ramage, D., Talwar, K., and Zhang, L

    McMahan, H. B., Ramage, D., Talwar, K., and Zhang, L. Learning differentially private recurrent language models. Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018

    2018

  110. [123]

    Mikolov, T. et al. Statistical language models based on neural networks. Presentation at Google, Mountain View, 2nd April, 80 0 (26), 2012

    2012

  111. [124]

    Mishchenko, K., Bach, F., Even, M., and Woodworth, B. E. Asynchronous SGD beats minibatch SGD under arbitrary delays. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 420--433....

    2022

  112. [125]

    Distributed asynchronous incremental subgradient methods

    Nedić, A., Bertsekas, D., and Borkar, V. Distributed asynchronous incremental subgradient methods. In Butnariu, D., Censor, Y., and Reich, S. (eds.), Inherently Parallel Algorithms in Feasibility and Optimization and their Applications, volume 8 of Studies in Computational Mat...

    2001 doi

  113. [126]

    Federated learning with buffered asynchronous aggregation

    Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., and Huba, D. Federated learning with buffered asynchronous aggregation. In International conference on artificial intelligence and statistics, pp.\ 3581--3607. PMLR, 2022

    2022

  114. [127]

    H., van Dijk, M., Richtarik, P., Scheinberg, K., and Takac, M

    Nguyen, L., NGUYEN, P. H., van Dijk, M., Richtarik, P., Scheinberg, K., and Takac, M. SGD and hogwild! C onvergence without the bounded gradients assumption. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro...

    2018

  115. [128]

    D., Nguyen, T

    Nguyen, T. D., Nguyen, T. H., Ene, A., and Nguyen, H. Improved convergence in high probability of clipped gradient methods with heavy tailed noise. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sy...

    2023

  116. [129]

    Non-gaussianity of stochastic gradient noise

    Panigrahi, A., Somani, R., Goyal, N., and Netrapalli, P. Non-gaussianity of stochastic gradient noise. In Science Meets Engineering of Deep Learning (SEDL) Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019

    2019

  117. [130]

    On the difficulty of training recurrent neural networks

    Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp.\ 1310--...

    2013

  118. [131]

    Training deep learning models with norm-constrained lmos

    Pethick, T., Xie, W., Antonakopoulos, K., Zhu, Z., Silveti-Falls, A., and Cevher, V. Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529, 2025

    2025 arXiv

  119. [132]

    Hogwild!: A lock-free approach to parallelizing stochastic gradient descent

    Recht, B., Re, C., Wright, S., and Niu, F. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011

    2011

  120. [133]

    A tail-index analysis of stochastic gradient noise in deep neural networks

    Simsekli, U., Sagun, L., and Gurbuzbalaban, M. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pp.\ 5827--5837. PMLR, 2019

    2019

  121. [134]

    W., Li, M., and Smola, A

    Sra, S., Yu, A. W., Li, M., and Smola, A. Adadelay: Delay adaptive distributed stochastic optimization. In Gretton, A. and Robert, C. C. (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Lear...

    2016

  122. [135]

    Stich, S. U. and Karimireddy, S. P. The error-feedback framework: Sgd with delayed gradients. Journal of Machine Learning Research, 21 0 (237): 0 1--36, 2020. URL http://jmlr.org/papers/v21/19-748.html

    2020

  123. [136]

    Distributed asynchronous deterministic and stochastic gradient optimization algorithms

    Tsitsiklis, J., Bertsekas, D., and Athans, M. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE transactions on automatic control, 31 0 (9): 0 803--812, 2003

    2003

  124. [137]

    and Richt \'a rik, P

    Tyurin, A. and Richt \'a rik, P. Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems, 36: 0 16515--16577, 2023

    2023

  125. [138]

    Understanding priors in B ayesian neural networks at the unit level

    Vladimirova, M., Verbeek, J., Mesejo, P., and Arbel, J. Understanding priors in B ayesian neural networks at the unit level. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine...

    2019

  126. [139]

    Sub-weibull distributions: Generalizing sub-gaussian and sub-exponential properties to heavier tailed distributions

    Vladimirova, M., Girard, S., Nguyen, H., and Arbel, J. Sub-weibull distributions: Generalizing sub-gaussian and sub-exponential properties to heavier tailed distributions. Stat, 9 0 (1): 0 e318, 2020

    2020

  127. [140]

    Tackling the data heterogeneity in asynchronous federated learning with cached update calibration

    Wang, Y., Cao, Y., Wu, J., Chen, R., and Chen, J. Tackling the data heterogeneity in asynchronous federated learning with cached update calibration. In Federated learning and analytics in practice: algorithms, systems, applications, and opportunities, 2023

    2023

  128. [141]

    R., and Johansson, M

    Wu, X., Magnusson, S., Feyzmahdavian, H. R., and Johansson, M. Delay-adaptive step-sizes for asynchronous learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, vo...

    2022

  129. [142]

    Asynchronous federated optimization

    Xie, C., Koyejo, O., and Gupta, I. Asynchronous federated optimization. OPT2020: 12th Annual Workshop on Optimization for Machine Learning, 2020

    2020

  130. [143]

    Improved analysis of clipping algorithms for non-convex optimization

    Zhang, B., Jin, J., Fang, C., and Wang, L. Improved analysis of clipping algorithms for non-convex optimization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 15511--15521. Curran...

    2020

  131. [144]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Zhang, J., He, T., Sra, S., and Jadbabaie, A. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020 b . URL https://openreview.net/forum?id=BJgnXpVYwS

    2020

  132. [145]

    P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S

    Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020 c

    2020

  133. [146]

    Staleness-aware Async-SGD for distributed deep learning

    Zhang, W., Gupta, S., Lian, X., and Liu, J. Staleness-aware Async-SGD for distributed deep learning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16), 2016

    2016

  134. [147]

    Asynchronous stochastic gradient descent with delay compensation

    Zheng, S., Meng, Q., Wang, T., Chen, W., Yu, N., Ma, Z.-M., and Liu, T.-Y. Asynchronous stochastic gradient descent with delay compensation. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of M...

    2017

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.