Pith. sign in

REVIEW 5 major objections 3 minor 1 cited by

Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization

T0 review · 5 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DSGT provably converges to a stationary point for non-convex decentralized ERM with rates matching centralized SGD.

desk verdict Solid theory paper with a couple of load-bearing 'left to the reader' steps; worth refereeing if the authors close those gaps. read the letter →

arxiv 1909.02712 v4 pith:QU22TTYL submitted 2019-09-06 cs.LG cs.DCcs.MAcs.SYeess.SYstat.ML

Jiaqi Zhang , Keyou You This is my paper · ORCID
classification cs.LGcs.DCcs.MAcs.SYeess.SYstat.ML MSC 90C1590C3590C06
keywords decentralizedoptimizationgradienttrackingnon-convexempiricalriskminimizationstochasticconsensusconvergenceratelinearspeedupalgebraicconnectivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that the decentralized stochastic gradient tracking algorithm (DSGT) converges for non-convex empirical risk minimization, not just for convex problems as previously shown. It provides a non-asymptotic convergence rate for both constant and diminishing stepsizes, with explicit dependence on the graph's algebraic connectivity, the variance of local stochastic gradients, and the mini-batch proportion. If the result is correct, DSGT can match the iteration complexity of centralized mini-batch SGD, can be independent of the network topology up to constants, and can achieve linear speedup in the number of nodes under stated conditions. The design that makes this work is a mini-batch whose size is proportional to each local dataset size, so the tracked gradient estimates the true global objective rather than a dataset-size-weighted surrogate.

What carries the argument

The load-bearing object is the stochastic gradient tracking variable $y_{i,k}$: each node mixes neighbors' $y$ values and adds the difference between its fresh mini-batch gradient and its previous one, so $y_{i,k}$ tracks the average stochastic gradient of the global objective instead of a dataset-size-weighted surrogate. The secondary mechanism is the choice of mini-batch size $\eta N_i$ proportional to the local dataset size. The proof then forms two interlacing linear inequalities, one for the cumulative consensus error and one for $\sum \gamma_t^2 E[\|\bar{y}_t\|^2]$, and solves a two-variable linear program to bound the optimality gap; the whole rate follows from the contraction coefficient $\rho = \|W - \frac{1}{n}\mathbf{1}\mathbf{1}^T\|$.

What would settle it

Take a smooth non-convex one-node ERM with a polynomial loss so that gradients are unbounded, run DSGT with the prescribed stepsizes, and check whether the measured quantity $R(K)$ fails to fall at the promised rate; a more direct check is to sample gradients along the trajectory and verify that the left-hand side of Assumption 1(c) exceeds $\sigma_i^2 + \lambda^2\|\eta \nabla f(x)\|^2$, which would place the problem outside the theorem's domain.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 3.1: when local losses are Lipschitz smooth, the graph is connected with doubly stochastic weights, and local stochastic gradient variances satisfy the bound in Assumption 1(c), any non-increasing stepsize below the stated threshold yields $R(K) \le \frac{1}{\sum_{t=1}^K \gamma_t}\left(\frac{9L\sigma_s^2}{\eta}\sum \gamma_t^2 + \frac{96\rho^2(1+\sqrt{\rho})^2 n L \tilde{L} \sigma_s^2}{(1-\rho)^3}\sum \gamma_t^3 + 12 n L \gamma_1 C + \frac{12 D^2}{\eta L}\right)$, where $R(K)$ mixes the gradient norm at the averaged iterate with the consensus error. From this bound the authors derive an $O(1/\sqrt{K})$ stationary-point rate for tuned constant stepsizes, $O(1/k^{1-p})$ rates for stepsizes $\gamma_k = O(1/k^p)$, a function-value rate under convexity, and conditions under which the network topology affects only constants.

Load-bearing premise

The proof depends on Assumption 1(c): the variance of each local stochastic gradient is bounded by $\sigma_i^2 + \lambda^2 \|\eta \nabla f(x)\|^2$, which the paper notes is safe when gradients are bounded, such as logistic regression or sigmoid networks, but is not guaranteed for arbitrary smooth losses.

Editorial extensions

If this is right

  • With a constant stepsize, DSGT reaches a stationary point at $O(1/\sqrt{K})$ plus a term $O(\rho^2/(1-\rho)^3 K)$, so the dominant iteration complexity matches centralized SGD.
  • When $\frac{n \tilde{L} \rho^2}{(1-\rho)^3}\sum \gamma_t^3 = O(\sum \gamma_t^2)$, the convergence rate is independent of the network, meaning topology only changes constant factors.
  • Under equal-sized local datasets and Lipschitz constants proportional to dataset size, DSGT achieves linear speedup in the number of nodes.
  • For convex losses, the stronger function-value measure $R_c(k)$ converges to the optimum at the same order, extending the result beyond stationary points.
  • In some empirical risk minimization settings, DSGT can be faster than centralized mini-batch SGD because locally duplicated samples can eliminate stochastic variance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to run DSGT on time-varying or directed topologies; the proof only needs a contraction coefficient $\rho<1$, so the same LP argument should transfer if such a coefficient can be defined.
  • A practical diagnostic suggested by Assumption 1(c) is to measure, for a given loss and sampling scheme, how the worst-case SG variance scales with $\|\nabla f\|^2$; if the slope exceeds the assumed $\lambda^2$, the theorem's rate should degrade in a predictable way.
  • Because the tracking variable already aggregates historical gradient information, inserting momentum or variance reduction at the $y_{i,k}$ update is a natural acceleration that the current analysis does not cover.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The manuscript studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization over a peer-to-peer network. Each node uses a mini-batch whose size is proportional to the size of its local dataset, updates a local model variable and a gradient-tracking variable, and communicates with neighbors through a doubly stochastic weight matrix. The main result, Theorem 3.1, states a non-asymptotic bound on a combined optimality-gap and consensus-error quantity R(K) under Lipschitz smoothness, a variance assumption, and a stepsize upper bound that depends on the network connectivity ρ and the variance-growth parameter λ. From this theorem the paper derives constant-stepsize and diminishing-stepsize corollaries, discusses a network-independence property, compares DSGT with centralized SGD, analyzes speedup, and presents experiments on CIFAR-10 for logistic regression and LeNet.

Significance. If the proof is completed, the result is a useful contribution: it extends gradient-tracking analysis from convex to non-convex ERM, gives explicit dependence on algebraic connectivity, mini-batch variance, and stepsize, and provides a transparent LP-based proof technique for coupling consensus and optimality-gap estimates. The paper also gives a concrete divergence example for D2, a fair comparison with D-PSGD, and a careful discussion of when linear speedup is or is not achievable. The proof structure is largely self-contained and no quantity is fitted to data. However, the central theorem is not fully verified as written: two load-bearing polynomial inequalities are left to the reader, and one displayed identity in the eigen-decomposition argument is false. These issues are repairable but require a substantive revision of the proof.

major comments (5)
  1. [Section 4.1, Lemma 4.3] The claim θ ≤ (√ρ + ρ)/(1 + ρ) < 1 is asserted after the sentence 'substituting the upper-bound into θ yields a polynomial fraction in ρ' and is justified only by 'finding the minimum value or simply plotting the curve. We omit the details here to save space.' This bound is load-bearing: it makes the consensus recursion contractive, and it is used in (4.22) to control 1/(1−θ), in the estimate a1a2 ≤ 1/4, and in Lemma 4.4. A plotted curve is not a proof, and a wrong sign or missed ρ-region in this polynomial would invalidate Lemma 4.3, Lemma 4.4, and hence Theorem 3.1. Please provide a complete algebraic verification, preferably in an appendix, or cite a machine-checkable certificate.
  2. [Section 4.1, Eq. (4.8)] The proof of Lemma 4.3 contains the displayed identity θ_t^k − θ_t^k = (θ_t − θ_t) Σ_{l=0}^{k−1} θ_t^l θ_t^{k−1−l} = Ψ_t k θ_t^{k−1}. The second equality is false in general: for k = 2 the sum equals θ_t + θ_t, not 2θ_t. The subsequent bound (4.8), which is used to obtain C0(k) and then Lemma 4.4, relies on this step. The intended conclusion may be obtainable as an inequality with absolute values, and the sign of the middle term in (4.7) may be harmless because P11 ≤ P22, but the text as written contains a false equality at a load-bearing point. Please replace it with a correct inequality and verify the sign handling.
  3. [Section 4.2, after Eq. (4.23)] The final absorption step requires the polynomial inequality 72ρ²(1−ρ)(1+√ρ)² + 5(1+ρ)²(1−ρ)² < 6(1+ρ)⁴? The paper states that the displayed fraction is < 1/4 and says 'the last inequality can be readily verified since it is equivalent to checking the positiveness of a polynomial on ρ∈(0,1), and we omit the details.' This inequality is load-bearing: it is exactly what permits subtracting (1/3)Σγ_t E‖∇f(bar x_t)‖² from both sides to obtain the final R(K) bound without reintroducing the gradient term. Without this verification, the proof of Theorem 3.1 is incomplete at its final step. Please provide the explicit polynomial and its verification.
  4. [Corollary 3.3] The corollary states its result for p ∈ [0.5, 1] and gives the bound R(k) ≤ (1−p)/(a(k^{1−p}−1))(...). At p = 1 the denominator is zero, and the displayed formula is undefined. The proof also treats p = 1 separately via ∫_1^k t^{−1}dt = ln k, but the final expression does not handle this case. Please restrict the statement to p ∈ [0.5, 1) or provide a separate, correctly simplified bound for p = 1.
  5. [Section 3, Assumption 1(c)] Assumption 1(c) bounds the local stochastic-gradient variance as E‖∂f_i(x;ξ) − η∇f_i(x)‖² ≤ σ_i² + λ²‖η∇f(x)‖². The paper itself notes that this is safe only when gradients are bounded, e.g., logistic regression or sigmoid networks. Every convergence rate in Theorem 3.1 and both corollaries depends on σ_s² and λ through the stepsize bound and through ilde L, so the title's claim of convergence for general non-convex ERM is stronger than what the assumptions support. Please state this bounded-gradient restriction prominently in the abstract or introduction, or weaken the scope claim accordingly.
minor comments (3)
  1. [Section 6, Figures 2–4] The experiments appear to report single runs with no error bars, repeated trials, or stated random seeds. Since the stepsize is tuned per algorithm and per task, adding multiple-seed results with variance estimates would strengthen the empirical comparison.
  2. [Lemma 4.3, definition of C0(k)] The term γ_{tk}² in C0(k) appears to be a typo; the subsequent lemmas use γ_1². Please correct the notation.
  3. [Section 5.2, Eq. (5.4)] The comparison with centralized SGD relies on the additional Assumptions 2 and 3 after Theorem 3.1. The text should state explicitly that the 'comparable to centralized SGD' conclusion holds only under those additional assumptions, not under Assumption 1 alone.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DSGT convergence proof is a self-contained derivation from stated assumptions; omitted polynomial checks are a rigor gap, not a circular step.

full rationale

The paper's central claim, Theorem 3.1, is a self-contained derivation from Assumption 1. The proof bounds cumulative consensus error and optimality gap by two linear inequalities (Lemmas 4.4 and 4.6) and solves a two-variable LP; all quantities in the final bound (sigma_s^2, lambda, rho, L, eta, stepsizes) are parameters appearing in the assumptions or algorithm, and no target quantity is used as an input. The comparison with centralized SGD in Section 5.2 invokes Assumptions 2 and 3, which are stated conditions, not fitted values, and the speedup discussion explicitly labels them as assumptions. Self-citations ([29,30] for introduction of DSGT, [52,53] for asynchronous variants, [31] for the term 'network independence') are background or terminological and do not carry the proof. The only notable gaps are that two elementary polynomial inequalities in Section 4.1 and after (4.23) are asserted to be checkable by plotting and are omitted 'to save space'; these are unverified algebraic claims and a rigor concern, but they are not circular because the inequalities are concrete statements independent of the theorem being proved. No quantity is fitted to data, no known result is renamed, and no load-bearing argument reduces to a self-citation. Score 0.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central theorem relies on Assumption 1(a-c), the stepsize bound, and the standard convexity addition for Theorem 3.4. Assumptions 2 and 3 are used only in the network-independence and speedup discussion, not in Theorem 3.1. No parameters were fitted to data, and no new physical or mathematical entities are introduced.

assumptions (7)
  • domain assumption Assumption 1(a): each local f_i is L_i-Lipschitz smooth and global f is bounded below.
    Standard smoothness; enters every descent and consensus bound, e.g. Lipschitz descent in Lemma 4.6 and Theorem 3.1.
  • domain assumption Assumption 1(b): the graph is connected and W is doubly stochastic.
    Needed for consensus contraction with rho = ||W - 11^T/n|| < 1, used in Lemmas 4.1 and 4.3.
  • domain assumption Assumption 1(c): E[||partial f_i(x;xi) - eta grad f_i(x)||^2] <= sigma_i^2 + lambda^2 ||eta grad f(x)||^2.
    Controls stochastic gradient variance, including possible quadratic growth in the global gradient; carries sigma_s^2 and lambda into every rate.
  • domain assumption Stepsize bound: gamma_k <= (1-rho)^2 / (eta L (1+rho)^2 max{sqrt(1+n^2 lambda^2), 24 n^2 lambda^2}) and non-increasing.
    Algorithmic condition required by Theorem 3.1 to keep the LP coefficients a1*a2 < 1.
  • domain assumption Assumption 2: n tilde{L} rho^2 / (1-rho)^3 * sum_{t=1}^k gamma_t^3 = O(sum gamma_t^2).
    Used only in Section 5 to derive the network independence property and compare with centralized SGD.
  • domain assumption Assumption 3: L_i = c_l N_i, L_c = c_l N, and N_i = N_j for all i,j.
    Used in Sections 5.2 and 5.3 to equate local and global Lipschitz constants and to establish linear speedup.
  • domain assumption Theorem 3.4 additional: all f_i are convex and f has a minimizer.
    Used only for the convex convergence result in Theorem 3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization." pith.science (2026). https://pith.science/paper/QU22TTYL

@misc{pith2026190902712,
  author       = {Pith},
  title        = {Pith review of: Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QU22TTYL}},
  note         = {Machine review of arXiv:1909.02712}
}
read the original abstract

This paper studies a decentralized stochastic gradient tracking (DSGT) algorithm for non-convex empirical risk minimization problems over a peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT only for convex problems. To ensure exact convergence and handle the variance among decentralized datasets, each node performs a stochastic gradient (SG) tracking step by using a mini-batch of samples, where the batch size is designed to be proportional to the size of the local dataset. We explicitly evaluate the convergence rate of DSGT with respect to the number of iterations in terms of algebraic connectivity of the network, mini-batch size, gradient variance, etc. Under certain conditions, we further show that DSGT has a network independence property in the sense that the network topology only affects the convergence rate up to a constant factor. Hence, the convergence rate of DSGT can be comparable to the centralized SGD method. Moreover, a linear speedup of DSGT with respect to the number of nodes is achievable for some scenarios. Numerical experiments for neural networks and logistic regression problems on CIFAR-10 finally illustrate the advantages of DSGT.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A momentum variant of the Exact-Diffusion decentralized algorithm is shown to converge sublinearly for nonconvex objectives and linearly under the PL condition, with convergence error independent of data heterogeneity.

Reference graph

Works this paper leans on

54 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [20]

    S. Lu, X. Zhang, H. Sun, and M. Hong, Gnsd: a gradient-tracking based nonconvex stochastic algorithm for decentralized optimization , in 2019 IEEE Data Science Workshop (DSW), 2019, pp. 315–321

    work page 2019

  2. [45]

    R. Xin, U. A. Khan, and S. Kar , An improved convergence analysis for decentralized online stochastic non-convex optimization, arXiv preprint arXiv:2008.04195, (2020)

    work page Pith review arXiv 2020

  3. [1]

    Agarwal, M

    A. Agarwal, M. J. Wainwright, and J. C. Duchi , Distributed dual averaging in networks , in Advances in Neural Information Processing Systems, 2010, pp. 550–558

    work page 2010

  4. [2]

    Aragues, G

    R. Aragues, G. Shi, D. V. Dimarogonas, C. Sag ¨u´es, K. H. Johansson, and Y. Mezouar , Distributed algebraic connectivity estimation for undirected graphs with upper and lower bounds, Automatica, 50 (2014), pp. 3253–3259

    work page 2014

  5. [3]

    Assran, N

    M. Assran, N. Loizou, N. Ballas, and M. Rabbat , Stochastic gradient push for distributed deep learning, in Proceedings of the 36th International Conference on Machine Learning, vol. 97, 2019, pp. 344–353

    work page 2019

  6. [4]

    Assran and M

    M. Assran and M. Rabbat , Asynchronous gradient-push, IEEE Transactions on Automatic Control, (2020), pp. 1–1

    work page 2020

  7. [5]

    Bianchi, G

    P. Bianchi, G. Fort, and W. Hachem, Performance of a distributed stochastic approximation algorithm, IEEE Transactions on Information Theory, 59 (2013), pp. 7405–7418

    work page 2013

  8. [6]

    Bottou, F

    L. Bottou, F. E. Curtis, and J. Nocedal , Optimization methods for large-scale machine learning, Siam Review, 60 (2018), pp. 223–311

    work page 2018

Show all 54 references
  1. [7]

    Chang, M

    T.-H. Chang, M. Hong, H.-T. Wai, X. Zhang, and S. Lu , Distributed learning in the non- convex world: From batch to streaming data, and beyond , IEEE Signal Processing Maga- zine, 37 (2020), pp. 26–38, https://doi.org/10.1109/MSP.2020.2970170

    2020

  2. [8]

    F. R. Chung and F. C. Graham , Spectral graph theory, no. 92, American Mathematical Soc., 1997. 26 JIAQI ZHANG AND KEYOU YOU

    1997

  3. [9]

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. , Large scale distributed deep networks , in Advances in neural information processing systems, 2012, pp. 1223–1231

    2012

  4. [10]

    Defazio, F

    A. Defazio, F. Bach, and S. Lacoste-Julien , SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives , in Advances in neural informa- tion processing systems, 2014, pp. 1646–1654

    2014

  5. [11]

    Hendrikx, F

    H. Hendrikx, F. Bach, and L. Massouli´e, Asynchronous accelerated proximal stochastic gra- dient for strongly convex distributed finite sums , arXiv preprint arXiv:1901.09865, (2019)

    2019 arXiv

  6. [12]

    Johnson and T

    R. Johnson and T. Zhang , Accelerating stochastic gradient descent using predictive variance reduction, in Advances in neural information processing systems, 2013, pp. 315–323

    2013

  7. [13]

    Koloskova, S

    A. Koloskova, S. Stich, and M. Jaggi , Decentralized stochastic optimization and gossip algorithms with compressed communication, in Proceedings of the 36th International Con- ference on Machine Learning, PMLR, 2019, pp. 3478–3487

    2019

  8. [14]

    Krizhevsky and G

    A. Krizhevsky and G. Hinton , Learning multiple layers of features from tiny images , tech. report, Citeseer, 2009

    2009

  9. [15]

    Lan , First-order and Stochastic Optimization Methods for Machine Learning , Springer Series in the Data Sciences, Springer International Publishing, Cham, 2020

    G. Lan , First-order and Stochastic Optimization Methods for Machine Learning , Springer Series in the Data Sciences, Springer International Publishing, Cham, 2020

    2020

  10. [16]

    G. Lan, S. Lee, and Y. Zhou , Communication-efficient algorithms for decentralized and sto- chastic optimization, Mathematical Programming, (2017), pp. 1–48

    2017

  11. [17]

    LeCun, L

    Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. , Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86 (1998), pp. 2278–2324

    1998

  12. [18]

    X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu , Can decentralized algo- rithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent, in Advances in Neural Information Processing Systems, 2017, pp. 5330– 5340

    2017

  13. [19]

    X. Lian, W. Zhang, C. Zhang, and J. Liu , Asynchronous decentralized parallel stochastic gradient descent, in Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 3049–3058

    2018

  14. [21]

    Lu and C

    Y. Lu and C. De Sa , Moniqua: Modulo quantized communication in decentralized sgd , in Proceedings of Machine Learning and Systems 2020, 2020, pp. 1293–1303

    2020

  15. [22]

    McMahan, E

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas , Communication- efficient learning of deep networks from decentralized data , in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol. 54, Fort Lauderdale, FL, USA, 2017, PM...

    2017

  16. [23]

    Mokhtari and A

    A. Mokhtari and A. Ribeiro, DSA: Decentralized double stochastic averaging gradient algo- rithm, The Journal of Machine Learning Research, 17 (2016), pp. 2165–2199

    2016

  17. [24]

    Nedi´c and A

    A. Nedi´c and A. Olshevsky, Stochastic gradient-push for strongly convex functions on time- varying directed graphs, IEEE Transactions on Automatic Control, 61 (2016), pp. 3936– 3947

    2016

  18. [25]

    Nedi ´c, A

    A. Nedi ´c, A. Olshevsky, and M. G. Rabbat , Network topology and communication- computation tradeoffs in decentralized optimization, Proceedings of the IEEE, 106 (2018), pp. 953–976, https://doi.org/10.1109/JPROC.2018.2817461

    2018

  19. [26]

    Nedi´c, A

    A. Nedi´c, A. Olshevsky, and W. Shi , Achieving geometric convergence for distributed opti- mization over time-varying graphs , SIAM Journal on Optimization, 27 (2017), pp. 2597– 2633

    2017

  20. [27]

    Nedi ´c and A

    A. Nedi ´c and A. Ozdaglar , Distributed subgradient methods for multi-agent optimization , IEEE Transactions on Automatic Control, 54 (2009), pp. 48–61

    2009

  21. [28]

    Neglia, C

    G. Neglia, C. Xu, D. Towsley, and G. Calbi , Decentralized gradient methods: does topol- ogy matter? , in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020, pp. 2348–2358

    2020

  22. [29]

    Pu and A

    S. Pu and A. Nedi ´c, A distributed stochastic gradient tracking method , in 2018 IEEE Confer- ence on Decision and Control (CDC), IEEE, 2018, pp. 963–968

    2018

  23. [30]

    Pu and A

    S. Pu and A. Nedi´c, Distributed stochastic gradient tracking methods, Mathematical Program- ming, (2020), https://doi.org/10.1007/s10107-020-01487-0

    2020 doi

  24. [31]

    S. Pu, A. Olshevsky, and I. C. Paschalidis , Asymptotic network independence in distrib- uted stochastic optimization for machine learning: Examining distributed and centralized stochastic gradient descent, IEEE Signal Processing Magazine, 37 (2020), pp. 114–122

    2020

  25. [32]

    Qu and N

    G. Qu and N. Li , Harnessing smoothness to accelerate distributed optimization , IEEE Trans- actions on Control of Network Systems, 5 (2018), pp. 1245–1260. DSGT FOR NON-CONVEX ERM 27

    2018

  26. [33]

    Qu and N

    G. Qu and N. Li , Accelerated distributed nesterov gradient descent , IEEE Transactions on Automatic Control, 65 (2020), pp. 2566–2581

    2020

  27. [34]

    N. L. Roux, M. Schmidt, and F. R. Bach , A stochastic gradient method with an exponential convergence rate for finite training sets , in Advances in neural information processing systems, 2012, pp. 2663–2671

    2012

  28. [35]

    Scaman, F

    K. Scaman, F. Bach, S. Bubeck, Y. T. Lee, and L. Massouli ´e, Optimal algorithms for smooth and strongly convex distributed optimization in networks , in International Confer- ence on Machine Learning, 2017, pp. 3027–3036

    2017

  29. [36]

    Scaman, F

    K. Scaman, F. Bach, S. Bubeck, L. Massouli´e, and Y. T. Lee, Optimal algorithms for non- smooth distributed optimization in networks, in Advances in Neural Information Processing Systems, 2018, pp. 2745–2754

    2018

  30. [37]

    Z. Shen, A. Mokhtari, T. Zhou, P. Zhao, and H. Qian , Towards more efficient stochastic decentralized learning: Faster convergence and sparse communication , in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 4624–4633

    2018

  31. [38]

    W. Shi, Q. Ling, G. Wu, and W. Yin , EXTRA: An exact first-order algorithm for decentral- ized consensus optimization, SIAM Journal on Optimization, 25 (2015), pp. 944–966

    2015

  32. [39]

    Sirb and X

    B. Sirb and X. Ye , Decentralized consensus algorithm with delayed and stochastic gradients , SIAM Journal on Optimization, 28 (2018), pp. 1232–1254

    2018

  33. [40]

    H. Tang, S. Gan, C. Zhang, T. Zhang, and J. Liu , Communication compression for decen- tralized training, in Advances in Neural Information Processing Systems, 2018, pp. 7652– 7662

    2018

  34. [41]

    H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu,D2: Decentralized training over decentral- ized data, in Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 4848–4856

    2018

  35. [42]

    Y. Tian, Y. Sun, and G. Scutari , Achieving linear convergence in distributed asynchronous multi-agent optimization, arXiv preprint arXiv:1803.10359, (2018)

    2018 arXiv

  36. [43]

    Xiao and S

    L. Xiao and S. Boyd , Fast linear iterations for distributed averaging , Systems & Control Letters, 53 (2004), pp. 65–78

    2004

  37. [44]

    P. Xie, K. You, R. Tempo, S. Song, and C. Wu, Distributed convex optimization with inequal- ity constraints over time-varying unbalanced digraphs , IEEE Transactions on Automatic Control, 63 (2018), pp. 4331–4337

    2018

  38. [46]

    R. Xin, A. K. Sahu, U. A. Khan, and S. Kar, Distributed stochastic optimization with gradi- ent tracking over strongly-connected networks , in 2019 IEEE 58th Conference on Decision and Control (CDC), IEEE, 2019, pp. 8353–8358

    2019

  39. [47]

    J. Xu, S. Zhu, Y. C. Soh, and L. Xie, Augmented distributed gradient methods for multi-agent optimization under uncoordinated constant stepsizes, in Decision and Control (CDC), 2015 IEEE 54th Annual Conference on, IEEE, 2015, pp. 2055–2060

    2015

  40. [48]

    C. Yu, H. Tang, C. Renggli, S. Kassing, A. Singla, D. Alistarh, C. Zhang, and J. Liu , Distributed learning over unreliable networks , in International Conference on Machine Learning, 2019, pp. 7202–7212

    2019

  41. [49]

    H. Yu, R. Jin, and S. Yang, On the linear speedup analysis of communication efficient momen- tum SGD for distributed non-convex optimization, in Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 7184–7193

    2019

  42. [50]

    K. Yuan, S. A. Alghunaim, B. Ying, and A. H. Sayed, On the performance of exact diffusion over adaptive networks , (2019), pp. 4898–4903

    2019

  43. [51]

    K. Yuan, B. Ying, J. Liu, and A. H. Sayed , Variance-reduced stochastic learning by net- worked agents under random reshuffling , IEEE Transactions on Signal Processing, 67 (2018), pp. 351–366

    2018

  44. [52]

    Zhang and K

    J. Zhang and K. You , Asynchronous decentralized optimization in directed networks , arXiv preprint arXiv:1901.08215, (2019)

    2019 arXiv

  45. [53]

    Zhang and K

    J. Zhang and K. You , AsySPA: An exact asynchronous algorithm for convex optimization over digraphs, IEEE Transactions on Automatic Control, 65 (2020), pp. 2494–2509

    2020

  46. [54]

    Zhang, K

    J. Zhang, K. You, and K. Cai , Distributed dual gradient tracking for resource allocation in unbalanced networks, IEEE Transactions on Signal Processing, 68 (2020), pp. 2186–2198

    2020

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.