Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Distributed momentum SGD provably converges at the last iterate

desk verdict Real but unverified generalization of centralized mSGD last-iterate analysis to distributed momentum methods; the main proof has a load-bearing gap as written, though the claims are plausible. read the letter →

arxiv 2505.10889 v1 pith:R6Z6YUXP submitted 2025-05-16 math.OC

Difei Cheng , Ruinan Jin , Hong Qiao , Bo Zhang This is my paper · ORCID
classification math.OC MSC 90C1590C2668W15
keywords distributedstochasticgradientdescentmomentumlast-iterateconvergencenon-convexoptimizationRobbins-MonrostepsizeconsensusaveragingEASGDD-PSGD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to close a gap between practice and theory for distributed stochastic gradient descent with momentum. Existing results for momentum-based PSASGD, EASGD, and D-PSGD only guaranteed time-averaged convergence, while practitioners care about the final model. The paper proves that, under Robbins-Monro step sizes and standard smoothness, bounded-gradient, and noise assumptions, the gradient norm of the averaged last iterate converges to zero almost surely and in mean square. It also derives a last-iterate rate under convexity and shows that a larger momentum coefficient accelerates early-stage progress. A sympathetic reader would see the paper's contribution as making the last iterate a theoretically legitimate output of these distributed momentum methods.

What carries the argument

The load-bearing object is the unified momentum iteration (8), which encodes three distributed algorithms through the mixing matrix $W_n$: PSASGD uses full averaging, D-PSGD uses a doubly stochastic gossip matrix, and EASGD augments the state with a central anchor variable and an elastic update block. The proof's engine is a pair of estimates: the momentum buffer satisfies $\mathbb{E}\|v_n\|^2=O(\epsilon_n^2)$ because gradients are globally bounded by $M$, and the consensus error $\mathbb{E}\|x_n^{(i)}-\bar{x}_n\|^2$ tends to zero through the spectral gap $\lambda_0<1$ of the mixing matrix. These estimates control the discrepancy between the averaged iterate and the true gradient, yielding a weighted supermartingale inequality; the divergence of $\sum_n \epsilon_n$ then upgrades a liminf conclusion to limsup convergence, giving both almost sure and mean-square last-iterate convergence.

What would settle it

Run the noiseless one-worker version of iteration (8) with $\alpha=0.9$, $\epsilon_n=1/\sqrt{n}$, and a smooth nonconvex objective with bounded gradients such as $g(x)=\log(1+x^2)$; if $\|\nabla g(x_n)\|$ does not converge to zero over a long horizon, Theorem II.1 is false. More sharply, recompute Eq. (10) for the quadratic $g(x)=x^2$, where the bounded-gradient assumption fails: $\mathbb{E}\|v_n\|^2$ need not be $O(\epsilon_n^2)$, so the paper's proof breaks exactly where that assumption enters.

Watch

Extended reading notes

Core claim

The central claim is Theorem II.1: for the unified momentum iteration $v_n=\alpha v_{n-1}+\epsilon_n G(X_n,\xi_n)$, $X_{n+1}=W_n(X_n-v_n)$, with $W_n$ a periodic doubly stochastic mixing matrix, the averaged iterate $x_n=\frac1m\sum_i x_n^{(i)}$ satisfies $\|\nabla g(x_n)\| o 0$ almost surely and $\mathbb{E}\|\nabla g(x_n)\|^2\to 0$. This extends centralized last-iterate momentum SGD guarantees to the distributed consensus setting, covering momentum-based PSASGD, EASGD, and D-PSGD in one argument. Under additional convexity and stability assumptions, the paper obtains the last-iterate rate $\mathbb{E}(g(u^\top X_T)-g(\theta^*))=O(\sqrt{m}\,\ln T/\sqrt{T})$. Theorem II.3 further states that the probability that the gradient norm has not yet fallen below a target $a_0$ by step $n$ decays like $O(e^{-s(1-\alpha)^{-2}\sum_{i=1}^n \epsilon_i})$, which formalizes the early-stage acceleration of larger momentum.

Load-bearing premise

The proof needs the gradient field to be globally bounded together with a global Lipschitz constant; for many realistic losses, including quadratics, gradients grow without bound, and if that premise fails the momentum buffer and consensus estimates that carry the argument are no longer controlled.

Editorial extensions

If this is right

  • Momentum-based PSASGD, EASGD, and D-PSGD can be deployed with the final averaged model as the output, because the last iterate is certified rather than only the time average.
  • The time-averaged gradient-norm convergence used in earlier analyses follows as an immediate corollary, so the new result is strictly stronger for nonconvex objectives.
  • Under convexity and the paper's stability condition, the last-iterate function error is $O(\sqrt{m}\,\ln T/\sqrt{T})$, quantifying how the convergence bound depends on the number of workers.
  • Larger momentum is provably beneficial in the early phase: the hitting-time tail bound decays with rate $s/(1-\alpha)^2$, matching the experiments where $\alpha=0.9$ converges fastest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof's reliance on the global bound $\|G(X)\|\le M$ suggests the first extension to target is unbounded-gradient losses; the $O(\epsilon_n^2)$ momentum-buffer estimate is the point where that constant is indispensable, so relaxing it would require a different control on the momentum term.
  • The rate $O(\sqrt{m}\,\ln T/\sqrt{T})$ may be improvable: the logarithmic factor comes from summing $1/k$ over the last $T$ steps, and a sharper convexity argument might remove it, while the $\sqrt{m}$ factor indicates a possible speedup from parallelism that could be tested against centralized momentum SGD.
  • The limsup-by-contradiction step, which rules out a second accumulation point of $\|\nabla g(x_n)\|^2$ using the divergence of $\sum_n \epsilon_n$, is a general template that could be applied to other stochastic approximation schemes with consensus noise.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies the momentum-based distributed iteration (8), which unifies momentum variants of PSASGD, EASGD, and D-PSGD. Under a global Lipschitz and bounded-gradient assumption together with the Robbins-Monro step-size schedule, Theorem II.1 claims almost-sure and L2 last-iterate convergence of the gradient norm evaluated at the averaged worker model. Theorem II.2 claims a last-iterate convergence rate under additional convexity and stability assumptions, and Theorem II.3 claims that larger momentum accelerates the early phase by bounding a first-hitting-time tail. The proofs in Appendices B-D adapt the almost-supermartingale framework of the authors' prior work [30,37] to the distributed setting, and the experiments train ResNet20 on CIFAR-10 and CIFAR-100.

Significance. If the proof of Theorem II.1 can be completed, the paper would provide the first last-iterate, rather than time-averaged, convergence guarantees for momentum-based distributed SGD in non-convex problems, and the unified treatment of the three algorithms is a useful contribution. The paper is honest that the asymptotic rate in Theorem II.2 is not improved by momentum and uses Theorem II.3 to address the early-stage regime. No machine-checked proofs or reproducible code are included, and the experiments plot loss and accuracy rather than the gradient norm targeted by the theory, so the empirical support is indirect. The main technical novelty is the distributed and consensus-error handling, which is exactly where the proof currently has gaps; the significance is therefore conditional on the repairs described below.

major comments (4)
  1. [Appendix B, Eqs. (19)-(22)] The descent inequality (23), which is the foundation of Theorem II.1, is not derived as written. Eq. (21) contains the term -sum_{s=1}^n alpha^{n-s} epsilon_s (u^T G(X_s))^T u^T G(X_s, xi_s), whose first factor is the average of local gradients u^T G(X_s). Eq. (22), however, analyzes (u^T G(I tensor u^T X_s))^T u^T G(X_s, xi_s), whose first factor is nabla g(x_s). Replacing one first factor by the other introduces the consensus error u^T G(X_s) - nabla g(x_s), and no estimate is provided for the resulting cross term, in expectation or almost surely. Since Eq. (23) is used to obtain the summability of epsilon_s E||nabla g(x_s)||^2, Theorem II.1 is not proved as written. A likely repair is to work with E||u^T G(X_s)||^2 and to lower-bound it by (1/2)||nabla g(x_s)||^2 minus a consensus-error term controlled by beta_s; the authors should supply this argument in full.
  2. [Appendix B, after Eq. (17)] The statement that Eq. (17) implies sum_{s=1}^n epsilon_s^2 beta_s -> 0 and beta_hat_n -> 0 is not correct. The double sum beta_hat_n = sum_{t=1}^n sum_{s=1}^t alpha^{t-s} epsilon_s beta_s is nondecreasing whenever beta_s is nonnegative, so it cannot converge to zero unless all beta_s vanish; and convergence of E beta_s to zero does not by itself give summability of epsilon_s^2 E beta_s. What is needed for the combination of Eqs. (26) and (28) is a rate for E beta_s from Eq. (17), for instance E beta_s = O(epsilon_s^2), together with boundedness rather than vanishing of beta_hat_n. This step is load-bearing for the conclusion sum epsilon_s E||nabla g(x_s)||^2 < infinity.
  3. [Theorem II.2 and Appendix C] The rate theorem is not valid as stated. The step-size choice epsilon_n = sqrt(m)/sqrt(n) gives sum epsilon_n^2 = m sum 1/n = infinity, contradicting Assumption II.2, so the hypothesis set of Theorem II.2 is empty if Assumption II.2 is enforced. If the intended step size is sqrt(m)/n, the proof must be reworked because its summations use sqrt(m)/sqrt(t)-type terms. Independently of this, the proof is incomplete: the constants l and l0 in the bound involving sum E((z_t - z_{T-k})^T nabla g(z_t)) are undefined; the replacement of u^T G(X_n) by nabla g(z_n) in Eq. (33) is asserted to incur an O(epsilon_n^2) error without derivation; and the proof concludes with a bound on E(g(z_T) - g(theta*)), while the theorem concerns E(g(u^T X_T) - g(theta*)), with the conversion between z_T and u^T X_T omitted. The rate claim therefore needs either a corrected statement with a complete proof or removal.
  4. [Appendix D, Theorem II.3] The proof does not match the theorem statement. The theorem defines the hitting time tau(a0) using the local gradient nabla g_i(x_n) of worker i, but the proof works with the global gradient nabla g(x_n) throughout. The Stolz-type inequality for the ratio G_i/(epsilon_i F_i) is not a valid weighted-average inequality as written: a weighted average of nonnegative ratios cannot be lower-bounded by the ratio of the last terms. The final step E(I_n^{(a0)}) <= F_{n+1}^{(a0)} is also unjustified because F is a weighted sum of E(I_t g(x_t)) and no lower bound relating g to the indicator is supplied; the inequality g(x) >= ||nabla g(x)||^2/(2L) from Lemma A.2 would need to be stated and used. Because Theorem II.3 is one of the advertised contributions, these gaps must be addressed.
minor comments (5)
  1. [Appendix A, Lemma A.4] Lemma A.4 is false as stated: for x_k = 1/k^2, the tail sum_{k=s}^n x_k is of order 1/s, which is not bounded by a constant multiple of x_s = 1/s^2. If this lemma is used anywhere in the proofs, the argument needs to be replaced.
  2. [Figure 1] Figure 1 appears corrupted in the manuscript: many panel labels and axis ticks are garbled strings such as '/s48 /s50/s48/s48', so the experimental results cannot be evaluated. Please regenerate the figure with readable text or provide the experimental curves in a verifiable form.
  3. [Assumption II.1(3)] Assumption II.1(3) requires the gradient to be globally bounded and globally Lipschitz, which excludes simple unbounded objectives such as quadratics. The paper should either discuss relaxation of this assumption or explicitly identify it as the main restriction of the theory.
  4. [Section III] The reported experiments show training loss and accuracy, not ||nabla g(x_n)||, so they do not directly support the last-iterate convergence claim of Theorem II.1; a plot of the gradient norm or a clear statement about how it is computed would strengthen the empirical section.
  5. [Eq. (19)] Eq. (19) contains an apparent typo: '(Im tensor u^T G(X_n))^T u^T G(X_n, xi_n)' should presumably be '(u^T G(Im tensor u^T X_n))^T u^T G(X_n, xi_n)'; the notation should be made consistent throughout the proof.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central theorem is an extension of a centralized momentum-SGD analysis to distributed averaging, and the self-citations used are generic lemmas and assumptions, not the conclusion.

full rationale

No significant circularity. The paper's central result, Theorem II.1, is a mathematical extension of centralized momentum-SGD analysis to distributed averaging through the consensus-error estimate in Eq. (17). Although the proof reuses lemmas from the authors' prior works [30] and [37] (Lemmas A.2, A.3, A.6) and attributes Assumption II.1(3) to [21], which shares an author with the current paper, these are standalone sequence/optimization facts with stated assumptions that do not include the target last-iterate convergence; they are not fitted values and are not conclusions imported by definition. The descent-inequality chain in Eqs. (18) through (26) is written out in the appendix, and the summability of ϵ_s E||∇g(x_s)||^2 is derived from the descent inequality rather than assumed. The apparent issue in Eqs. (21) and (22), where the factor u^T G(X_s) is replaced by ∇g(x_s) without separately displaying the consensus-error cross term, is a proof-completeness gap rather than a circular reduction: the missing term is not already the theorem's conclusion, and no equation in the paper defines the gradient-at-the-average in terms of the average gradient by construction. No fitted parameter is relabeled as a prediction, and no uniqueness theorem from prior work is invoked to force the argument. Therefore the derivation is not circular.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The central proof imports Lemmas A.3 and A.6 from the authors' prior works [30, 37]; these are not formalized here. No parameters are fitted to data; the step sizes and momentum coefficients are chosen standardly.

assumptions (8)
  • domain assumption Global Lipschitz and bounded gradients (Assumption II.1.3): ||G(X)-G(Y)|| <= L||X-Y|| and ||G(X)|| <= M.
    Used throughout the proof, e.g., to bound E||v_n||^2 = O(epsilon_n^2) in Eq. (10) and to relate local to averaged gradients in Eq. (22).
  • domain assumption Unbiased stochastic gradients and bounded variance (Assumption II.1.1 and II.1.4).
    Standard in stochastic approximation; used to handle noise in Eq. (22)-(25).
  • domain assumption Mixing matrix W_n is symmetric doubly stochastic with a single eigenvalue equal to one and other eigenvalues of magnitude less than one (Assumption II.1.2).
    Ensures the consensus error decays geometrically between communication rounds (Eq. 12-17).
  • domain assumption Robbins-Monro step-size condition (Assumption II.2): epsilon_n positive, decreasing to zero, sum epsilon_n = infinity, sum epsilon_n^2 < infinity.
    Needed to balance noise and progress; used to show sum epsilon_n diverges and sum epsilon_n^2 converges.
  • domain assumption Non-negative loss function (Assumption II.1).
    Used to apply Lemma A.2; the paper notes it can be replaced by a finite lower bound.
  • domain assumption Convexity and unique optimum for the rate result (Assumption II.3).
    Required for Theorem II.2, which uses convexity to lower bound (z_t-z)^T grad g(z_t).
  • ad hoc to paper Stability assumption for the rate result (Assumption II.4): ||u^T X_n|| < G almost surely.
    A boundedness condition on the iterates that is not implied by the other assumptions and is introduced only for Theorem II.2.
  • ad hoc to paper Bounded loss function for Theorem II.3.
    Stated only in Theorem II.3, not in Assumption II.1; needed to bound E[I_n g(x_n)] below by a positive constant times E[I_n].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum." pith.science (2026). https://pith.science/paper/R6Z6YUXP

@misc{pith2026250510889,
  author       = {Pith},
  title        = {Pith review of: Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6Z6YUXP}},
  note         = {Machine review of arXiv:2505.10889}
}
abstract

Distributed stochastic gradient methods are widely used to preserve data privacy and ensure scalability in large-scale learning tasks. While existing theory on distributed momentum Stochastic Gradient Descent (mSGD) mainly focuses on time-averaged convergence, the more practical last-iterate convergence remains underexplored. In this work, we analyze the last-iterate convergence behavior of distributed mSGD in non-convex settings under the classical Robbins-Monro step-size schedule. We prove both almost sure convergence and $L_2$ convergence of the last iterate, and derive convergence rates. We further show that momentum can accelerate early-stage convergence, and provide experiments to support our theory.

Figures

Figures reproduced from arXiv: 2505.10889 by the authors.

Figure 1
Figure 1. Training and prediction performance on CIFAR-10 and CIFAR-100 with 1,3,10,20 sub-datasets (workers). (a)-(d): The training loss with [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 30 canonical work pages

  1. [1]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,” Annals of Mathematical Statistics , vol. 22, no. 3, pp. 400–407, 1951

    1951

  2. [2]

    Some methods of speeding up the convergence of iteration methods,

    B. T. Polyak, “Some methods of speeding up the convergence of iteration methods,” USSR Computational Mathematics & Mathematical Physics, vol. 4, no. 5, pp. 1–17, 1964

    work page 1964

  3. [3]

    Adaptive deep feature learning network with Nesterov momentum and its application to rotating machinery fault diagnosis,

    S. Tang, C. Shen, D. Wang, S. Li, W. Huang, and Z. Zhu, “Adaptive deep feature learning network with Nesterov momentum and its application to rotating machinery fault diagnosis,” Neurocomputing, vol. 305, pp. 1–14, 2018

    work page 2018

  4. [4]

    Combining ordered subsets and momentum for accelerated X-ray CT image reconstruction,

    D. Kim, S. Ramani, and J. A. Fessler, “Combining ordered subsets and momentum for accelerated X-ray CT image reconstruction,” IEEE Transactions on Medical Imaging , vol. 34, no. 1, pp. 167– 178, 2014

    work page 2014

  5. [5]

    Speech recognition with deep recurrent neural networks,

    A. Graves, A.-r. Mohamed, and G. E. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , 2013, pp. 6645–6649

    work page 2013

  6. [6]

    SGD and Hogwild! convergence without the bounded gradients assumption,

    L. Nguyen, P. H. Nguyen, M. Dijk, P. Richtárik, K. Scheinberg, and M. Takác, “SGD and Hogwild! convergence without the bounded gradients assumption,” in International Conference on Machine Learning, 2018, pp. 3750–3758

    work page 2018

  7. [7]

    Reducing the dimensionality of data with neural networks,

    G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006

    2006

  8. [8]

    Distributed training strategies for the structured perceptron,

    R. McDonald, K. Hall, and G. Mann, “Distributed training strategies for the structured perceptron,” in Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics , 2010, pp. 456–464

    work page 2010

Show all 38 references
  1. [9]

    Deep learning with elastic averaging sgd,

    S. Zhang, A. E. Choromanska, and Y . LeCun, “Deep learning with elastic averaging sgd,” Advances in Neural Information Processing Systems, vol. 28, 2015

    2015

  2. [10]

    Network topology and communication-computation tradeoffs in decentralized optimization,

    A. Nedi ´c, A. Olshevsky, and M. G. Rabbat, “Network topology and communication-computation tradeoffs in decentralized optimization,” Proceedings of the IEEE , vol. 106, no. 5, pp. 953– 976, 2018

    2018

  3. [11]

    Local sgd converges fast and communicates little,

    S. U. Stich, “Local sgd converges fast and communicates little,” arXiv preprint arXiv:1805.09767 , 2018

    2018 arXiv

  4. [12]

    Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning,

    H. Yu, S. Yang, and S. Zhu, “Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning,” in Proceedings of the AAAI conference on artificial intelligence , vol. 33, no. 01, 2019, pp. 5693–5700

    2019

  5. [13]

    A linear speedup analysis of distributed deep learning with sparse and quantized communication,

    P. Jiang and G. Agrawal, “A linear speedup analysis of distributed deep learning with sparse and quantized communication,” Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  6. [14]

    Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,

    X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,” Advances in neural information processing systems , vol. 30, 2017

    2017

  7. [15]

    Collaborative deep learning in fixed topology networks,

    Z. Jiang, A. Balu, C. Hegde, and S. Sarkar, “Collaborative deep learning in fixed topology networks,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  8. [16]

    On nonconvex decentralized gradient descent,

    J. Zeng and W. Yin, “On nonconvex decentralized gradient descent,” IEEE Transactions on signal processing , vol. 66, no. 11, pp. 2834–2848, 2018

    2018

  9. [17]

    Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms,

    J. Wang and G. Joshi, “Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms,” Journal of Machine Learning Research , vol. 22, 2021

    2021

  10. [18]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105, 2012

    2012

  11. [19]

    A unified analysis of stochastic momentum methods for deep learning,

    Y . Yan, T. Yang, Z. Li, Q. Lin, and Y . Yang, “A unified analysis of stochastic momentum methods for deep learning,” arXiv preprint arXiv:1808.10396, 2018

    2018 arXiv

  12. [20]

    On the importance of initialization and momentum in deep learning,

    I. Sutskever, J. Martens, G. Dahl, and G. E. Hinton, “On the importance of initialization and momentum in deep learning,” in International Conference on Machine Learning , 2013, pp. 1139– 1147

    2013

  13. [21]

    On the linear speedup analysis of communication efficient momentum sgd for distributed non- convex optimization,

    H. Yu, R. Jin, and S. Yang, “On the linear speedup analysis of communication efficient momentum sgd for distributed non- convex optimization,” in International Conference on Machine Learning. PMLR, 2019, pp. 7184–7193

    2019

  14. [22]

    On consensus- optimality trade-offs in collaborative deep learning,

    Z. Jiang, A. Balu, C. Hegde, and S. Sarkar, “On consensus- optimality trade-offs in collaborative deep learning,” Frontiers in artificial intelligence, vol. 4, p. 573731, 2021

    2021

  15. [23]

    Deep gradient compression: Reducing the communication bandwidth for distributed training,

    Y . Lin, S. Han, H. Mao, Y . Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” ICLR, 2018

    2018

  16. [24]

    Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,

    X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  17. [25]

    Decentlam: Decentralized momentum sgd for large-batch deep training,

    K. Yuan, Y . Chen, X. Huang, Y . Zhang, P. Pan, Y . Xu, and W. Yin, “Decentlam: Decentralized momentum sgd for large-batch deep training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 3029–3039

    2021

  18. [26]

    2020SQuARM: Communication-efficient momentum SGD for decentralized optimization,

    N. Singh, D. Data, J. George, and S. Diggavi, “2020SQuARM: Communication-efficient momentum SGD for decentralized optimization,” IEEE Journal on Selected Areas in Information Theory, vol. 2, no. 3, pp. 954–969, 2021

    2021

  19. [27]

    Periodic stochastic gradient descent with momentum for decentralized training,

    H. Gao and H. Huang, “Periodic stochastic gradient descent with momentum for decentralized training,” arXiv preprint arXiv:2008.10435, 2020

    2008 arXiv

  20. [28]

    Decentralized deep learning using momentum-accelerated consensus,

    A. Balu, Z. Jiang, S. Y . Tan, C. Hedge, and S. Sarkar, “Decentralized deep learning using momentum-accelerated consensus,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021

    2021

  21. [29]

    Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning,

    H. Yu, S. Yang, and S. Zhu, “Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, pp. 5693–5700, 2019

    2019

  22. [30]

    On the convergence of mSGD and AdaGrad for stochastic optimization,

    R. Jin, Y . Xing, and X. He, “On the convergence of mSGD and AdaGrad for stochastic optimization,” in International Conference on Learning Representations , 2022

    2022

  23. [31]

    Don’t decay the learning rate, increase the batch size,

    S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V . Le, “Don’t decay the learning rate, increase the batch size,” arXiv preprint arXiv:1711.00489, 2017

    2017 arXiv

  24. [32]

    Bayesian learning via stochastic gradient langevin dynamics,

    M. Welling and Y . W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in International Conference on International Conference on Machine Learning , 2011

    2011

  25. [33]

    Convergence of proximal-gradient stochastic variational inference under non-decreasing step-size sequence,

    M. E. Khan, R. Babanezhad, W. Lin, M. Schmidt, and M. Sugiyama, “Convergence of proximal-gradient stochastic variational inference under non-decreasing step-size sequence,” Journal of Comparative Neurology , vol. 319, no. 3, pp. 359–86, 2015

    2015

  26. [34]

    Understanding the role of momentum in stochastic gradient methods,

    I. Gitman, H. Lang, P. Zhang, and L. Xiao, “Understanding the role of momentum in stochastic gradient methods,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  27. [35]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” IEEE, 2016

    2016

  28. [36]

    Nesterov,Introductory Lectures on Convex Optimization: A Basic Course

    Y . Nesterov,Introductory Lectures on Convex Optimization: A Basic Course. Introductory Lectures on Convex Optimization: A Basic Course, 2004

    2004

  29. [37]

    Revisit last- iterate convergence of msgd under milder requirement on step size

    R. Jin, X. He, L. Chen, D. Cheng, and V . Gupta, “Revisit last- iterate convergence of msgd under milder requirement on step size.” in NeurIPS, 2022. 6

    2022

  30. [38]

    On almost sure convergence for sums of stochastic sequence,

    Z.-z. Wang, Y . Dong, and F. Ding, “On almost sure convergence for sums of stochastic sequence,” Communications in Statistics- Theory and Methods , vol. 48, no. 14, pp. 3609–3621, 2019. V. B IOGRAPHY SECTION Difei Cheng (M’22)received B.S. degree in mathematics and applied mat...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.